Avamar Bare Metal Recovery

If you have ever had to do a BMR or Bare Metal Recovery with Avamar you may find it difficult to have much success. This is not Avamar’s fault, but instead it is the endless amount of variables that are unique to each environment. These need to be tweaked and tuned to have a successful restore of your server.

What I am going to do here is to get you past as many hurtles as I know of, in hopes that it will hopefully build confidence in this process and allow you to have success. So lets get started!

Why Use Bare Metal Recovery

In my opinion, and please write to me and tell me yours, BMR should only be used to recover a physical server, to another physical server or VM (Virtual Machine), from a previous backup, and for the purpose of making it a bootable OS. That was a mouth full! Let me explain my thinking.

Most of my current company’s physical servers, that require backups, are almost always DB (Databases). The reason they are physical servers rather than a VM is because they require a lot of resources in CPU and Memory (RAM). So much so, that they would use all the resources of a VMware ESXi host server. So why not take out the VM part and give it the whole server. You get the point.

Now not all DB are like this, but there are a few that are and we found that it has been easier to just backup the file system and not the VSS (Volume Shadow Copy Service). The reason for this is the DBAs (Database Administrators) have already backed up the DB regularly to an X:\Drive on the same server. The DBA only require that the X:\ drive be backed up, because they are able to reinstall Windows with SQL and restore the DB in under an hour. This also allows for OS (Operating System) upgrades, so bonus.

The outlier is the DB that have so much vendor software and configurations that it is better to go a head and restore the whole thing to a bootable OS (and let the vendor figure out how to upgrade the OS later when it is EOL (End Of Life)).

On the other side of this is that Avamar backups of VMware are just restored VM directly to the VMware environment so no need to use BMR. So as you can tell there is a rare chance I need to use Avamar’s BMR.

Basics Of Setting Up To Backup

Physical server are the only backups that require a client to be installed on the server. We do not need to do this to a VMware VM server. Avamar can access the vCenter environment and backup and restore directly in vCenter.

The client that can be downloaded from our Avamar webpage https://YourAvamarNameOr-IP-Here/dtlt/home.html#downloads. This client is installed on the physical server itself and activated to the Avamar.

In the Avamar console we will create a Dataset to have the “filesystems” and the “VSS” plugin at a minimum.

After a backup has ran, in the Avamar console we can see in the backups manage tab that we have two backups of the VSS and the Filesystem.

Restoring With Bare Metal Recovery

When we need to do a restore of a physical server, it is not practical to have a spare server laying around to restore to, unless your really lucky. So even though we have a resource hungry physical server, we will most likely restore it to a VM and the vendor or admin can just deal will less resources until a physical replacement can be acquired.

ISO To Restore to a Physical Server

If you do have a physical server available then the process is as simple as downloading the BMR ISO from the Avamar download page mentioned earlier. Then you will use something like Rufus to not only put the ISO on a USB drive but more importantly, make the drive bootable, which is what Rufus does. You can get it here. Once Rufus has done its magic then plug it in a USB port of the server and boot to that USB.

ISO To Restore to a Virtual Machine

Now if we aren’t so lucky as to have an extra server laying around, then we will restore it to a VM. In vCenter we will just upload the ISO to the vCenter ISO storage. We will then create a VM with nothing on it. It doesn’t even have to have hard drives when it is created. Just give it the minimum RAM and CPU. Put the NIC on the same network as your Avamar to eliminate issues. We can change it to something else later. That’s it!

Let’s Talk Hard Drives

The hard drive size and order of the drives are critical to a successful restore. We have to make sure that our server has hard drives space that is the exact same as the original or larger than the original. If we are talking about restoring to a VM then we definitely need to give it some extra space to assure success. We also need to make sure that the disk sizes are in the same order as the original server. As an example, if Disk 0 is 100GB and Disk 1 is 300GB on the original backed up server, then we need to make sure that the disks in BMR are in the same order.

Depending on our situation, we can use two different methods to find out what size drives we need exactly. If we are doing a test restore of server and the original server is still operational then we can login to the server and use Disk Management. If the server is hard down, (not running) then their is a script we can run against the backups. Let’s explore this options.

Method 1: Disk Management

We should always perform test restores to validate that we have the ability when it really counts. Let’s RDP into the original production server and open Disk Management. To do this we right click the start button and select “Disk Management”.

We can see that Disk 0 is a total of 279.36GB, while Disk 1 is a total of 1676.64GB. So we know that our restore target server needs, at a minimum, a slight bit larger than those amounts. How much large you get make no difference as the restore needs to hit its minimum size value.

Method 2: Run A Script Against The Backup

Here is a true test of our ability to restore, if the production server is hard down. If we were at the same location as the physical server, we may not be able to see the RAID configuration and see what space amounts goes to which Disk. And if we were not at the same physical location, but restoring remotely, we would not be about to see the physical drives or the RAID configuration. Here we need a script.

First we need to access a computer that either already has or can install the Avamar Client, that is used to backup physical servers. This computer also has to have an ability to reach the Avamar server, that holds the backup, with a ping test to make sure we have network routes in place.

To get the client, we can go to the “https://MyAvamarNameOr-IP-Here/dtlt/home.html#downloads” to download it and install it on your route-able computer. In this example, as the production server is up and running, I will use it, but your desktop will work too, if it is route-able to the Avamar.

Once logged in to the client installed computer, I will check the file location of the Avamar Client. This is usually in the “C:\Program Files\avs” location.

Once located, we will open a Command Prompt and Change Directory to the AVS directory with the command “cd C:\Program Files\avs”

Now we will run the following command while editing the parts that suite your environments information.

avtar.exe -x --server=IP_of_Avamar_Server --id=Username --ap=Password --path=/clients/servername_FQDN --labelnum=label_number_of_the_backup_to_be_restored --internal --target=.\tmp\ .system_info

Explanation of how to fill out the script:

–server=The FQDN name or IP of the Avamar server.

–id=MCUser is the best account to use for this

–ap=ThisIsMyPassword!!ToAvamar!!

–path=/clients/ProductionServer.domain.local

****Capitalization is important and must match.

****When I setup my Avamar, instead of using the default location for client based backups, I created a new Avamar Domain called “Physical_Servers” to store may Physical Servers Backups. Therefore my path is /Physical_Servers/”FullyQualifiedDomainNameOfTheServer”. Yours maybe set to the default of /clients/FQDN-ofServerHere.

–labelnum=263

****This is the label number of the VSS backup circled in the above graphic. This needs to be the VSS.

–target=.\tmp\

*****This is a folder that will be created in the avs directory where you command prompt cd to earlier.

Here is the command filled out:

avtar.exe -x --server=10.20.30.40 --id=MCUser --ap=Password! --path=/Physical_Servers/ProductionServer.domain.local --labelnum=263 --internal --target=.\tmp\ .system_info

Once the script has ran, based on it being filled out correctly, we can go to the tmp folder we just created. In here we are looking for two folders that can help us, the “CriticalVolumesMapping.xml” and “partitiontables.xml” files.

First if we look at the CriticalVolumesMapping.xml we can see that the backup contains serveral “Volume DiskNumbers”. First we have a Disk 0 that contains a C:\, then a Disk 1 that contains a D:\, and finally a Disk 0 again with \\? which represents the two other partitions, of three total, that do not have a letter assigned to them.

Now let’s look at the partitiontables.xml file.

As we can see where I placed the arrows in the graphic below the DiskNumber 0 or Disk 0 is 279GB and the Disk 1 is 1676GB. This is the same information we received from the Disk Management in the production server.

Also keep in mind that this data is not being pulled from the production server, but from the Avamar and its VSS backup of the production server.

Here we will create a VM, in VMware’s vCenter, to restore the backup

I will name it “Test-Restore”

Then select a host, storage, compatibility with host version, and VM Operating System. Now, on the “Customize hardware” page we will need to make changes from the default settings. The CPU and Memory (RAM) does not need to match the original production server that was backed up, but the hard disk (hard drives) do. This is where the above mentioned process of finding the original production server’s hard drives was so important. When creating the Hard Disks on the VM, they should be slightly larger then the original server. They can be even a great deal larger if you wanted.

We will make the first drive 300GB (original was 279GB) and the second 2TB (original was 1676GB). This will give us plenty of room for the restore.

Now we will put this VM on the same network as the Avamar, so we have guaranteed access to our backups, by changing the “New Network” to the same as the Avamar network. We will also click on the drop down arrow for “New CD/DVD Drive” and select “Datastore ISO File”.

We will navigate to the ISO of the Bare Metal Recover and select it.

Once selected we need to check the box to “Connect At Power On” so that when it boots up for the first time it will attempt to boot to the ISO.

Now lets go to the top of this page and select the “VM Options” tab. Here we need to change the “Firmware” to “BIOS”

Everything else should be left at its defaults values that work for you particular environment.

Now we will power on the new VM and open the web console.

One the first page we can leave everything as default. The time, data, and time zone with not effect the restore. You can set this if you like and it will show up in logs if you fail and want to troubleshoot. I find that even then it is not needed.

On the next page you will see your network adapter you used when creating the VM. I had changed mine from E1000E to VMXNet3 as this is what we use in my environment.

On this page we need to configure the information that will be used by the network to restore the VM. This is for the connection that will be used between the BMR VM and the Avamar server. The “Hostname” is not important as the restored server will retain its name and IP information from the original production server. We need to add the “DNS domain”, which is the Active Directory domain of the server. If we are not using DHCP then we need to statically assign the IP address, Subnet mask, Default gateway, and up to two DNS servers.

If the information you just put in is incorrect, it should give an error and force you to correct the information.

Once successful it will progress to the next screen where you will see the disks we created at the VM creation. The key thing to look for here is that Disk 0 and Disk 1 are in the correct order. If we looked back to the command where we viewed the size of the backup’s hard disks sizes, we saw that Disk 0 was 279GB and Disk 1 was 1676GB. We can see that Disk 0 is set to 300GB and Disk 1 is set to 2048GB. So we know that they are larger than the original backup and they are in the correct order. I have seen them switched on occasion, even while the VM in vCenter shows them in the correct order. I had to go back and create them in reverse order and they displayed correctly on this page.

On the next page we will fill out the information about the Avamar server and where it needs to look in the Avamar for the backup files.

Here we will fill out “Avamar Server” with the DNS name of the Avamar server. The name is not fully qualified and this is why we put in DNS server earlier. “Avamar User Name” needs to be the MCUser account and its password in the next field labeled “Avamar Password”. On the “Avamar Domain” we put a forward slash / only. In the “Avamar Account” we put the location of where the backup files are located in the Avamar.

The above will validate the information you provided. If it fails check that the capitalization of the Avamar is the same as it is inside the Avamar itself. Validate the MCUser password by attempting to login to Avamar with the MCUser account and password so that you know the credentials you are using are correct. Make sure that the path to the backup is also correct and also check that the it is capitalized in the same way. Remember, Avamar is a Linux OS and capitalization is important in Linux.

On the next page you need to select the backup date to be restored.

Remember that the backups all have a “Label” that corresponds to every specific backup. When we look at the Avamar itself we see the labels are called “Number”, but that the are one in the same. We can also see that all the labels listed are VSS only. That is because that is what is needed for BMR to proceed.

As I have Label 967 selected by default we will proceed with that one, but your label will be different with the same results.

Here we see the backup contents of the VSS backup and this matches all of our previous information regarding disks and there sizes. “Perform a quick format of disks” is already selected and on to the next page.

Here we are at a summary page and you can click “Restore”

Check the box a click OK

If it formats it then the storage size and order was satisfied. It we get a fail here then that is the issue.

When we get to this point it has begun restoring and we are on our way to success!

This was a long one! It was two days of work to not only write this but to go through it with you can copy and format this adventure. In the end, my goal is to save you time and heartache trying to solve this beast of BMR. As you can see there is nothing wrong with the Avamar BMR process itself, but it is all of the unknown variables that make up each individual environment, that the Avamar has to back up, that makes each one unique and prone to fail.

I truly hope you success in your recovery efforts and good luck to you.