How to fix Avamar not coming up after restart
I was troubleshooting an issue and thought that if I restarted the Avamar that it would be fixed. After it finished it didn’t come up and was worse then before because I couldn’t open anything. I looked into the CLI using Putty and ran the dpnctl status command to check everything.
admin@avamar:~/>: dpnctl status
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: down.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]
admin@avamar:~/>:From the output we can see that “MCS status: down” which is the mcserver. We now issue the mcserver.sh –start command.
admin@avamar:~/>: mcserver.sh --start
connectemcctl: INFO: fix_bug_335524: running command /usr/sbin/usermod -a -G postgres admin
connectemcctl: INFO: fix_bug_335524: exit status = 0
connectemcctl: INFO: fix_bug_335524: running command cp -R /home/admin/.postgresql /home/tomcat/
connectemcctl: INFO: fix_bug_335524: exit status = 1
…………………………..
Upgrade MCDB: processing time = 325ms
Check node list: processing time = 1460ms
Prepare MC SSL Properties: processing time = 0ms
Administrator Server startup failed at Wed Jan 31 15:17:21 CST 2024: Closing all connections to database. Server already started on port: 7778
Starting REST API Service……
REST API Service startup failed.
Start MC Rest: processing time = 14 s.
Already started, PID: [12756]
admin@avamar:~/>:Here we see the “Startup Failed: Closing all connections to database. Server already started on port: 7778” error. This is caused by an old stalled MCS/Java PID that is still running on the server despite MCS down.
Now let’s login as superuser by issuing the su – command and entering the password.
admin@avamar:~/>: su -
Password:
root@avamar:~#>:Now enter the following netstat -tulpn |egrep “7778|Active|Proto” command verify the issue
root@avamar:~/#: netstat -tulpn |egrep "7778|Active|Proto"
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::7778 :::* LISTEN 17143/java
root@avamar:~/#:In my case port 7778 is listening on 17143 (your “17143” will be different) Let’s enter in the command based on the number you get.
root@avamar:~/#: ps -eaf |grep 17143 |grep -v "grep" |cut -c1-130
admin 17143 1 50 2023 ? 59-15:37:50 /usr/java/latest//bin/java -Xmx3G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpP
root@avamar:~/#:Now we need to manually kill the old MCS/Java PID
root@avamar:~/#: kill 17143
root@avamar:~/#:Let’s check again by reissuing the previous command
root@avamar:~/#: ps -eaf |egrep "17143|PID" |grep -v "grep" |cut -c1-130
UID PID PPID C STIME TTY TIME CMD
root@avamar:~/#We now see the old Java process has stopped. Lets go back into admin with superuser privileged by issue the su – admin command and entering the superuser password.
root@avamar:~/#: su - admin
Password:
Directory: /home/admin
Wed Jan 31 15:32:17 CST 2024
admin@avamar:~/>: Now enter the following ssh-agent bash and ssh-add ~/.ssh/admin_key commands
admin@avamar:~/>: ssh-agent bash
admin@avamar:~/>: ssh-add ~/.ssh/admin_key
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
admin@avamar:~/>:Now let’s start mcserver and the backup schedule by issuing the dpnctl start mcs,sched –verbose commands.
admin@avamar:~/>: dpnctl start mcs,sched --verbose
Using /usr/local/avamar/var/probe.xml
dpnctl: INFO: dpnctl_util::set_verbose 1
dpnctl: INFO: dpnctl_cleanup::set_verbose 1…………….
……..dpnctl: INFO: Backup scheduler resumed.
dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist.
rm -f /usr/local/avamar/var/dpnctl.lck
dpnctl: INFO: [user "admin"] program exit status = 0 (normal)
admin@avamar:~/>:Now we will issue the dpnctl status to check that everything is now running.
admin@avamar:~/>: dpnctl status
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: ConnectEMC status: up.
dpnctl: INFO: ddrmaint-service status: up.
admin@avamar:~/>:We now have confirmation that everything is running and if we attempt to login to the Avamar console we will see that we can now access the Avamar!
Please comment below with any question! Thanks and see you next time!
