Usually we would only do a manual filesystem cleanup when we are running out of storage. We have gone in and made the tough choices of deleting old backups to make room. We attempt to refresh the view to see how much space we now have and nothing has changed. This is because the filesystem cleanup will remove this marked for deletion data only after it has ran. This also includes backups that have expired out, we will see no change until this process runs.
First let me give a word of caution on doing this too often. A filesystem clean up is a defaulted scheduled task that takes place on Tuesdays every week, so if we can, we should wait for this schedule to come around. The reason is because of de-duplication and how it works. Every time this procedure is done, the data is changed, cleaned up and if done too often the data will look different than the same data backed up previously. The de-duplication will begin to think it is new data and store a copy on this data, again essentially doubling the storage size of the believed new data. This can cause a serious adverse affect on your storage capacity. This is the tight rope we begin to walk, balancing between running to create space and not running it too often to cause de-duplication to stop working as effectively as it should.
Dell’s recommended maximum filesystem cleanup is no more that twice a week.
As we get deeper into using shell commands in the Avamar and Data Domain, we should always keep in mind that everything we can do in the GUI can be done in a shell. Here we are going to look at how we can do a filesystem clean up on our Data Domain. We will first start a Putty session into Avamar and we will run the “ddrmaint read-ddr-info –format=full” command to get information about our Data Domain.
admin@avamar:/#: ddrmaint read-ddr-info --format=full
====================== Read-DDR-Info ======================
System name : DD3300.domain.local
System ID : 0123456789
DDBoost user : ddboost
System index : 2
Replication : True
CP Backup : True
Model number : DD3300
Serialno : APM00123456789
DDOS version : 6.2.0.20-624005
System attached : 2022-05-05 19:55:08 (1588708508)
System max streams : 50
admin@wcm-avm-main:/#:Now we know the name of our DDBoost account and the System Name which we will use to access the Data Domain. Let’s load bash and access our Data Domain. First let’s run the “ssh-agent bash” and then the “ssh-add ~/.ssh/ddr_key”.
.
admin@wcm-avm-main:~/>: ssh-agent bash
admin@wcm-avm-main:~/#: ssh-add ~/.ssh/ddr_key
Identity added: /home/admin/.ssh/ddr_key (/home/admin/.ssh/ddr_key)
admin@wcm-avm-main:~/#:Now using the ddboost account name followed be the @ and then followed by the System ID we will enter “ssh ddboost@dd3300.domain.local” command.
admin@avamar:~/#: ssh ddboost@dd3300.domain.local
Data Domain OS
Last login: Wed Aug 5 09:01:46 CDT 2022 from 10.20.30.41 on pts/0
Welcome to Data Domain OS 6.2.0.20-624005
-----------------------------------------
ddboost@DD3300#Going the extra mile, we can start by putting in “snapshot list all” command to validate our snapshots.
ddboost@DD3300# snapshot list all
Snapshots Summary:
-------------------
Mtree Name Total Not Expired Min Expire Max Expire
Expired (day:hh:mm) (day:hh:mm)
---------------------------- ----- ------- ------- ----------- -----------
/data/col1/avamar-1234703033 145 1 144 - -
/data/col1/backup 0 0 0 - -
---------------------------- ----- ------- ------- ----------- -----------
ddboost@DD3300#Wow, look at all of the expired snapshots! 144 is a lot of space that was leftover from a past issue that was unknown to us. Let’s take from the above output and run the “snapshot list mtree /data/col1/avamar-1234703033”
ddboost@DD3300# snapshot list mtree /data/col1/avamar-1234703033
Snapshot Information for MTree: /data/col1/avamar-1234703033
----------------------------------------------
Name Pre-Comp (GiB) Create Date Retain Until Status
----------------- -------------- ----------------- ----------------- -------
cp.20220506140313 0.0 May 6 2022 08:57 Aug 4 2022 15:41 expired
cp.20220506141510 0.0 May 6 2022 09:09 Aug 4 2022 15:41 expired
cp.20220507140324 0.0 May 7 2022 08:57 Aug 4 2022 15:41 expired
cp.20220507150526 342.3 May 7 2022 10:00 Aug 4 2022 15:41 expired
cp.20220508140320 1074.2 May 8 2022 08:57 Aug 4 2022 15:41 expired
----------------- -------------- ----------------- ----------------- -------
Snapshot Summary
-------------------
Total: 145
Not expired: 1
Expired: 144
ddboost@DD3300#Let’s run the “filesys clean start” command to start the filesystem clean up.
ddboost@DD3300# filesys clean start
Cleaning started. Use 'filesys clean watch' to monitor progress.
ddboost@DD3300#As it suggested lets watch as this will take awhile.
ddboost@DD3300# filesys clean watch
Beginning 'filesys clean' monitoring. Use Control-C to stop monitoring.
Cleaning: phase 1 of 12 (pre-merge)
100.0% complete, 2246 GiB free; time: phase 0:00:46, total 0:00:46
Cleaning: phase 2 of 12 (pre-analysis)
100.0% complete, 2246 GiB free; time: phase 0:15:50, total 0:16:37
Cleaning: phase 3 of 12 (pre-enumeration)
0.0% complete, 2246 GiB free; time: phase 0:00:10, total 0:16:47^C
ddboost@DD3300#WOW, that’s was boring! I hit the Ctr + C on the old keyboard to get out of there as we all have better things to do. It will keep on running in the background until it completes. After it finishes we should see a big improvement. Now your system may not have the snapshots on them like mine did, but if you have removed old backup to increase storage, maybe while a Data Domain storage expansion is on the slow boat to getting installed to keep you running, then this can buy you some time.
Good luck to you and as always, let me know what you think about this tutorial.
