Backing up a VM with a PCIe device attached to it with Veeam

by

In a previous post I talked about installing a Quadro P620 into my ESXi host so I could attach it to my Plex VM. This worked out great except my Veeam backups started failing.

There is a limitation in VMware vSphere where you can’t take a Snapshot of a VM with a PCIe device passed through to it.

One option is to install the Veeam Agent for the OS you’re running and use it to take guest based backups. This isn’t ideal though in my opinion. I would much rather keep my host based backups of the VM. Fortunately this is a easy solution to this problem.

Shut off the VM before taking the Veeam backup and then power it back on after the backup is complete.

To get this working you need to install the VMware PowerShell Module on your Veeam server. To do this perform the following steps:

  1. Right click on the PowerShell shortcut and choose ‘Run as Administrator’
  2. Run the following commands: 
    Find-Module -Name VMware.PowerCLI
Install-Module -Name VMware.PowerCLI -Scope AllUsers
Get-Command -Module *VMWare*
Set-PowerCLIConfiguration -Scope AllUsers -ParticipateInCeip $false -InvalidCertificateAction Ignore
  3. You should see a large list of VMware PowerShell commands output which means you’ve successfully installed the module

Next up you need to make sure your Veeam Services are running under a Service Account with the appropriate permissions in vCenter. I believe this is normally a best practice and chances are you’ve all already done this. In my case I’d installed Veeam as a local service. Don’t know why but to fix it I just flipped over the following Windows Services to run as my backup operator account which had Domain Admin, Backup Operator, Local Admin on the Veeam Server and Administrator on vSphere permissions already.

The services were:

  • Veeam Backup Enterprise Manager
  • Veeam Backup Service
  • Veeam Broker Service
  • Veeam Cloud Connect Service
  • Veeam Guest Catalog Service
  • Veeam RESTful API Service

I then rebooted my Veeam server.

I already have my vCenter service joined to my domain but I did run into an issue where single sign-on wasn’t working properly. If I attempted to connect to my vCenter server via PowerShell using “Connect-VIServer <VCENTER SERVER FQDN>” I would be prompted for credentials which shouldn’t be happening since the account I’m logged in as is an Administrator in vCenter.

Turned out I needed to add my AD Group that gives users Administrative access to the vCenter Global Permissions list:

  1. Login to vCenter as an administrator
  2. Click ‘Menu’ and ‘Administrator’
  3. Click ‘Global Permissions’
  4. Click ‘Add’
  5. Change the ‘User’ field to your domain, search for the user or security group (I recommend security groups) and select it, make sure the role is ‘Administrator’ and check ‘Propagate to children’ and click ‘Ok’

After doing this I could run “Connect-VIServer <VCENTER SERVER FQDN>” and not be prompted for credentials.

Now that all the prep-work is done we can re-configure our backup job in Veeam.

First we’re going to need two scripts, one to shutdown the VM and one to boot it back up. I’ve saved these scripts on my Veeam server in “C:\Scripts\<VM FQDN>\”

The shutdown script is “shutdown.bat”, be sure to search and replace “VCENTER FQDN” and “VM FQDN” with your values:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -ExecutionPolicy Bypass -Command Connect-VIServer -Server "VCENTER FQDN"; Shutdown-VMGuest -VM "VM FQDN" -Server VCENTER FQDN -Confirm:0; do{$vm=Get-VM -Name "VM FQDN"}while ($vm.PowerState -eq \"PoweredOn\"); Disconnect-VIServer -Server "VCENTER FQDN" -Confirm:0

The startup script is “startup.bat”, be sure to search and replace “VCENTER FQDN” and “VM FQDN” with your values:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -ExecutionPolicy Bypass -Command Connect-VIServer -Server "VCENTER FQDN"; Start-VM -VM "VM FQDN" -Server "VCENTER FQDN"; Start-Sleep -s 90; Disconnect-VIServer -Server "VCENTER FQDN" -Confirm:0

Once you’ve created these fire up the Veeam console and re-configure the VMs job:

  1. Launch Veeam
  2. Find the backup job for your VM, right click on it and choose ‘Edit’
  3. Go to ‘Storage’
  4. Click ‘Advanced’
  5. Go to ‘Scripts’
  6. Checkmark ‘Run the following script before the job:’ and select your “shutdown.bat” script
  7. Checkmark ‘Run the following script after the job:’ and select your “startup.bat” script
  8. Click ‘Ok’
  9. Click ‘Finish’
  10. Perform a test run of the job, you can monitor the start-up/shutdown in vCenter

That’s it. Minor inconvenience but it works. Hopefully vSphere 7 will allow for snapshots on VMs with pass-through devices configured.

References

10 thoughts on “Backing up a VM with a PCIe device attached to it with Veeam”

  1. Hi,

    I’m trying to use your guide, “Backing up a VM with a PCIe device attached to it with Veeam” to shutdown a vm that has pci passthrough enabled, then once snapshot is taken/backup is complete to power it back on. I followed your guide but its not working and I’m unsure where I messed up. Would you be able to assist me please?

    I ensured all Veeam services are running under a domain admin. I have my vcsa joined to the domain.

    Here’s my shutdown script:
    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -executionpolicy bypass -Command Connect-VIServer -Server vcsa.rivard.local; Shutdown-VMGuest -VM RIV-PLEX.rivard.local -Server VCENTER FQDN -Confirm:0; do{$vm=Get-VM -Name RIV-PLEX.rivard.local”}while ($vm.PowerState -eq \”PoweredOn\”)

    And my power on script:
    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -executionpolicy bypass -Command Connect-VIServer -Server vcsa.rivard.local; Start-VM -VM riv-plex.rivard.local -Server vcsa.rivard.local; Start-Sleep -s 90

    Reply

    • You are missing “; Disconnect-VIServer -Server “vcsa.rivard.local” -Confirm:0″ at the end of both commands.

      You want to disconnect from vCenter when done or the connections will pile up consuming RAM and slowing things down.

      If you run these by hand in a PowerShell window do they work? If not what is the error output you’re getting?

      Reply

      • Thank you for your prompt response. Sorry for the delayed reply.

        I was able to get it working by adding a static dns entry in my DC vcsa.rivard.local. However a couple days ago with no changes made, it stopped. I’m getting prompted to enter credentials when running “Connect-VIServer and not sure why.

        Reply

  2. I am having a problem with your script. I am running Windows 2019 Server, ESXi 6.7, and Veeam 11.

    After putting installing your modules, here is your shutdown script with my details for shutdown (below) I get the follow error (below that):

    Shutdown Script:
    C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -executionpolicy bypass -Command Connect-VIServer -Server “sv-vcenter01.smoothrunnings.ca”; Shutdown-VMGuest -VM “sv-plex01.smoothrunnings.ca” -Server sv-vcenter01.smoothrunnings.ca -Confirm:0; do{$vm=Get-VM -Name “sv-plex01.smoothrunnings.ca”}while ($vm.PowerState -eq \”PoweredOn\”)

    Error:
    At line:1 char:336
    + … -Name “sv-plex01.smoothrunnings.ca”}while ($vm.PowerState -eq \”Power …
    + ~
    You must provide a value expression following the ‘-eq’ operator.
    At line:1 char:337
    + … sv-plex01.smoothrunnings.ca”}while ($vm.PowerState -eq \”PoweredOn\”)
    + ~~~~~~~~~~~~~
    Unexpected token ‘\”PoweredOn\”‘ in expression or statement.
    At line:1 char:336
    + … -Name “sv-plex01.smoothrunnings.ca”}while ($vm.PowerState -eq \”Power …
    + ~
    Missing closing ‘)’ after expression in ‘while’ statement.
    At line:1 char:350
    + … sv-plex01.smoothrunnings.ca”}while ($vm.PowerState -eq \”PoweredOn\”)
    + ~
    Unexpected token ‘)’ in expression or statement.
    + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : ExpectedValueExpression

    I am also using the Administrator account which also works on sv-vcenter01

    Reply

    • This could just be how things got posted to WordPress when you commented but if I compare your command to mine the only problem I see is your quotes look like “smart quotes” that something like Word loves to use to make things look pretty but basically breaks anywhere else.

      Just a wild stab in the dark since I have no idea what text editors you are using.

      Try copy/pasting your command into notepad.exe and then replace every single quote and try running it again.

      Reply

  3. Hi Eric,

    I reconfigured my scripts to not include the quotes around the server names and included the disconnect command at the end of them which if you do a search in your page the only time you mention “disconnect” is when you tell Rob he needed to added them, it doesn’t say anything in your notes that we should be adding it. LOL

    And I ran the new bat files I created to verify they work, shuting down the guest server in vcenter and starting it again.

    The problem I am at now is even though I log into my Veeam (mgt01) server as domain\Admin and log into veeam as domain\admin when I add the scripts to backup job and run it the scripts time out and I cannot figure out why. I get it might be a permissions thing but what permissions would it be using, or how do I tell?

    The jobs are setup to use domain\admin so it just add another level of complexity.

    Reply

    • Good catch. I will update my post to include the Disconnect-VIServer to the end of things.

      I suspect the issue you’re having now is that the script is running under the account Veeam runs under which by default is Local System (I think). I run mind under a dedicated service account and it has access to my vCenter.

      Reply

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.