Avamar not accessing Data Domain

Avamar is receiving data integrity alerts.

Avamar Alert
Event Code: 22426
Category: SYSTEM
Severity: SYSTEM_FATAL
Summary: A data integrity issue has been detected.

Avamar is also failing garbage collection with error:

failed garbage collection with error MSG_ERR_DDR_ERROR

Looking into the Avamar GUI shows that the Data Domain is in red, which is a sign that there is a connection issue with the Data Domain. Logging into the Data Domain with the Sysadmin account reveals that everything looks normal with no alerts or errors, but when attempting to login with the ddboost account it fails login. To solve this login issue I follow the steps on post “Data Domain failed login with DDBoost account“

After following the linked steps and the ddboost account login is now accessible, I notice that nothing has changed as to Avamar’s connection to the Data Domain. I use the “Server” tab and then the sub-tab “Server Management” to update the ddboost login connection settings to validate they are set correctly. To do this I first select the the Data Domain and then click the “Edit Data Domain System” button.

Then I updated the password and click OK

This fails to update with an error that it can not validate host name. I validate DNS settings using “nslookup” in CLI and all IP to name resolution are fine. This includes repeating the above Data Domain edit using both IP and name in the “Data Domain System Name” field.

I login to the Data Domain through CLI and the ddboost account shows enabled and not in a “locked” state. At this point knowing that the issue seems to not be with the Data Domain I login to the Avamar CLI and attempt to ssh into the Data Domain.

admin@avamar:~/>: ssh ddboost@data_domain-1
Data Domain OS
Account locked due to 36 failed logins
Password: 
Connection to data_domain-1 closed by remote host.

Here it shows the account is “locked due to 36 failed logins”! So now I wait about three minutes and reattempt.

Account locked due to 38 failed logins

This validates that the failed logins are not just when I attempt to “Edit Data Domain System” as shown above. From here I believe that Avamar is repeatedly attempting to connect with the wrong credentials. I decided I have no other option but to bounce (reboot) the Data Domain. I run the reboot command at the CLI of the Data Domain.

sysadmin@data_domain-1# reboot

When it is accessible again I go to the Avamar CLI and reattempt ssh.

admin@avamar:~/>: ssh ddboost@data_domain-1
Data Domain OS
Password: 
Last login: Wed Mar 23 13:25:08 CDT 2022 from 10.10.10.20 on pts/0


Welcome to Data Domain OS 6.0.2.20-587212
-----------------------------------------
**
** NOTICE: There is 1 outstanding alert. Run "alerts show current"
**         to display outstanding alert(s).
**
ddboost@data_domain-1# 

Going back to the Avamar GUI shows that the Data Domain is green and connections are working. I also check to see if I can “Edit Data Domain System” and I can. Everything is working! You will need to run the Avamar Clear Data Integrity Alert.