The advantage of virtualisation is that complete servers including all applications and even the operating system are kept as files on the hypervisor (the host running the virtualised servers). These files containing the complete virtual server can be backed-up. On a VMware ESX(i) server the virtual hosts can be backed-up while they keep running using standard tooling. This blog describes a solution (with the help of a 3rd party script) to backup a virtual host without downtime.
A good backup is a must. Hardware can fail and software can get corrupted. Making a backup must be easy. I wanted a single command to backup a complete server including all data. All my servers are virtualized on a ESXi server. So there must be an easy way to just copy a virtual host to a storage device.
I already use a Western Digital Live NAS device to backup my desktop, laptop and 'production' NAS. So this must be my backup medium where I want to store my backups.
A virtual host can be copied to another hypervisor (the server running the virtual host) and started on the new hardware. I already use this to install new machines or maintain the hypervisors hardware.
I do not want to shutdown a host to be able to perform a backup. VMWare supports the creation of so called snap-shots. A snap-shot is like a photo of the host. With a snap-shot the current state of the machine is frozen, and all new changes are writen to another file. Making a snap-shot is very quick, like a few seconds. The snap-shot contains the complete host and can be copied and started again on a different hypervisor.
I want to start a backup by running just one command, or even be able to automatically run a specific backup on a specific time (scheduled backups).
I want to store my backups not on my VMWare ESX(i) server. The server contains a single disk. So if this disk crashes, I will loose my servers and the backups all at once. I already use a WD-Live NAS device as general purpose backup storage. The NAS is located in my garage, so even after a fire the NAS will probably survive. (the garage is located 20 meters away from the house).
The WD-Live supports NFS. So I set-up a NFS link from the ESXi server to the WD-Live device. The NFS link is created during booting of the ESXi server and is always available. The NFS link will automatically recover when the connection was lost or the WD-Live was rebooted.
The NFS location is mounted as a datastore on the ESXi server using the vSphere Client.
- Open the vSphere Client and connect to the ESXi host.
- Go to the 'Configuration' tab and select 'Storage' in the right hardware menu.
- Select 'Add Storage...'
- Select Storage Type 'Network File System'
- Enter the name of the NFS server and the name of the NFS export folder.
The WD-Live exports the root of the disk as /nfs. All the WD-Live shares are located under the /nfs share.
- Enter a new name for the storage e.g. 'nfs_backup'.
The NFS location will now be mounted as a datastore on the ESXi host. In our case /vmfs/volumes/nfs_backup. This location will contain the root of the WD-Live NAS. I've created a share vmxbackup, so the complete path to my backup location will be /vmfs/volumes/nfs_backup/vmxbackup.
The VMWare ESX(i) server contains some standard commands to manipulate virtual servers. There is already a nice set of scripts that use this standard commands to create a easy to use backup solution. GhettoVCB is a open-source project that contains a set of scripts for backup and restore ESX(i) virtual servers using the standard VMWare commands.
The scripts are easy to install. I copied the GhettoVCB configuration file to /etc. The configuration file must be changed to meet the current configuration of the system. My configuration file looks as followed:
VM_BACKUP_VOLUME=/vmfs/volumes/nfs_backup/vmxbackup DISK_BACKUP_FORMAT=thin VM_BACKUP_ROTATION_COUNT=3 POWER_VM_DOWN_BEFORE_BACKUP=0 ENABLE_HARD_POWER_OFF=0 ITER_TO_WAIT_SHUTDOWN=3 POWER_DOWN_TIMEOUT=5 ENABLE_COMPRESSION=0 VM_SNAPSHOT_MEMORY=0 VM_SNAPSHOT_QUIESCE=0 ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP=0 ENABLE_NON_PERSISTENT_NFS=0 UNMOUNT_NFS=0 NFS_SERVER=<ip-address of WD-Live NAS> NFS_VERSION= NFS_MOUNT= NFS_LOCAL_NAME= NFS_VM_BACKUP_DIR= SNAPSHOT_TIMEOUT=15 EMAIL_LOG=1 EMAIL_SERVER=<ip-addres of mail server> EMAIL_SERVER_PORT=25 EMAIL_DELAY_INTERVAL=1 EMAIL_TO=<mail address for receiving alerts> EMAIL_FROM=ghettoVCB@schaake.nu WORKDIR_DEBUG=0 VM_SHUTDOWN_ORDER= VM_STARTUP_ORDER=
The VM_BACKUP_VOLUME parameter contains the complete path to the backup store (seen from the ESXi host). I do not power down any servers for the backup. I will do a on-line backup using the automatically created snap-shot. If a snap-shot exists which is not created by GhettoVCB, it will not be backed-up. You could choose to purge all old snap-shots before backup, but I use the snap-shot when I want to play with a virtual machine. I do not want any automated process destroying my snap-shots.
I do not use the NFS function of GhettoVCB. This option is still experimental, and the current NFS mount is working fine for me.
After the backup, I want to receive a status mail. Backups will take a couple of hours to finish, so an email when the backup is done with the exact result is very helpfull.
The first thing we need to do is check if everything is configured correctly. A backup of a virtual server can take a couple of hours (in my case even more than 11 hours). It is very frustrating to wait for a backup to finish, only to find out it wend wrong. GhettoVCB has a dryrun option. In dryrun mode all steps are executed, except for the actual backup. The script will check if the virtual server can be backed-up. It will check if the backup location exists and is accessible and has enough storage. Any error will be displayed. To execute the first dryrun, enter the following command on the ESXi host:
/bin/ghettoVCB/ghettoVCB.sh -m "virtual host" -g /etc/ghettoVCB.conf -d dryrun
The "virtual host" is the name of the host from the inventory of the vSphere Client. The next step would be executing a actual backup. Use the debug mode (replace dryrun with debug on the command line) to get the full details of the executed steps. You now made your first backup of a complete virtual server.
Backups are executed from the command line. You don't want to have your client pc (with the ssh terminal) open during the whole backup. So we need a way to start an unattended backup from the command line. This is very simple. We can use the nohup command to keep the backup running and catch output in a logfile. Use the following command to start a single unattended backup:
nohup /bin/ghettoVCB/ghettoVCB.sh -m "server" -g /etc/ghettoVCB.conf -d info > /vmfs/volumes/nfs_backup/vmxbackup/server.log 2>>&1 &
The last &-sign will start the process in background. The nohup will keep the command running when you logoff. The 2>>&1 will redirect all errors to standard output. The standard output is redirected to the log file.
If you want to see the progress of the backup, you can alway look in the logfile. The file contains the progress of the backup in percentages. There are 2 reasons to store the log file on the backup location. First the local filesystem of the ESXi host is very limited. The second reason is that I now can access the log file from any client that can connect to the NAS.
The last step is a fully automated backup of all virtual servers. To be continued...