Recently, a bug was discovered that affects changed block tracking on VMware hosts running ESXi 6. This is a critical flaw as changed block tracking (CBT) is the mechanism that many backup products (including Veeam) uses to decide what data has changed and needs to be backed as part of an incremental backup job. Unfortunately, the result of the bug is that there is no guarantee a successful backup job will actually allow you to restore good data.
The options for a workaround to the problem were not practical for most of our customers and included rolling back the host to a previous version of ESXi or shutting down the virtual machine during the backup process.
VMware has released a patch to resolve the issue. Network Center, Inc. is recommending that all customers running ESXi 6 hosts apply this patch as soon as possible to insure integrity of their backup data.
Unfortunately simply applying the patch does not resolve the issue. Resetting the changed block tracking map is also required to insure consistency. Veeam has created a script that will reset CBT on virtual machines. The script requires that Powershell and VMware CLI are installed on the management station where it is executed.
A couple of quick warnings:
- Resetting CBT may impact the amount of time a backup job will run as each block of data needs to be evaluated during the next backup cycle. Please take this into consideration if your backup window is constrained.
- Also, the script creates a VMware snapshot, resets CBT then removes the snapshot. This process may “stun” a virtual machine. Therefore it is recommended that the script is run during off-peak hours.
Again, this issue only impacts servers running ESXi version 6. All previous versions of VMware are unaffected. If you have questions, concerns or would like assistance applying the patches please reach out to anyone at Network Center, Inc. and we will be happy to help!
Please review the supporting documentation and find a link to the CBT reset script that Veeam has provided.
Veeam KB article: https://www.veeam.com/kb2075
Veeam CBT reset script: https://www.veeam.com/kb1113
VMware CLI download: https://my.vmware.com/web/vmware/details?downloadGroup=VCLI600&productId=491