Windows 2008 R2 virtual machine spawns Event Id 58 whenever backup of the virtual machine is performed

I don't like warnings, and I like errors and critical events even less, so when any of them show up in my event logs, no matter HOW benign they appear to be, I take it personally.

One of my application servers was running Windows Server 2008 R2 as a clustered virtual machine role on a Windows 2012 R2 Hyper-V cluster. I noticed this warning message appearing in my event log every night at 3:30 AM

Which is exactly the time the backup runs on this server. Other posts have noticed the same thing, and it doesn't seem to matter what backup software you're using. Some are using Veeam, some are using Windows Server backup. I'm using HP Data Protector 8.1.

Now, this machine was originally a VMware machine running on ESXi 5.1 with the disks as vmdk files. At some point it got converted over to Hyper-V (vhdx) and everything appeared to be fine.

I BELIEVE some time after the upgrade to Hyper-V 2012 R2 this error first started showing up. To be honest, I don't know.

A few things I got in my google results suggested I look and possibly even change the disk ID for the disk in question. I'm sure if you're here you've probably already seen those sites.

What I did was slightly different. Since I only have one disk in this virtual machine to begin with (which I verified by running disk part), I began to wonder if there might be some kind of hidden disk, or hidden device

So I opened up command prompt and typed

set devmgr_show_nonpresent_devices=1

and then fired up the Device Manager with

start devmgmt.msc

Expanding the disk section, something immediately caught my eye

Remember I said this virtual machine used to run on a VMware ESXi implementation? A quick check of some of my other virtual machines that weren't getting the error, including ones that were built on hyper-v directly revealed this mystery disk was not present.

So I did what any reasonable person would do. I made a checkpoint and I deleted it.

Rebooted the server and then forced an immediate backup using the same job that normally runs at 3:30 AM. The error? Still there.

Damnit.

The more I looked, the more I found that phantom SCSI disk device on other machines that had been converted. And THEY weren't getting the error. What could it be?

So, I went back to my Sesame street days.

You know, one of these things is not like the others... (go ahead, sing it, you know you want to)

And what I found was two possibilities.

The first was that the server with the error had a setting / component that the other servers didn't have

VMBus SCSI Controller

Hmm, that shouldn't be there. So I powered down the machine (again) and took it out. Thank god I work in a non 24/7 environment.

But unfortunately a quick backup attempt shows THATS not it either.

So the one thing I got left? It actually makes a little sense. It seems that machine has a checkpoint associated with it. So I deleted all of them and let the drives merge out.

But its not that either. *sigh*

The only last thing I notice is that this server has a fixed disk, while the other server is dynamic.

Nope. That's not it either.

Hmmm, maybe its the clustering.

Nope.

I still haven't solved this problem, but at least if you're here reading this I may have saved you a few hours of testing things that (probably) aren't why you're seeing this error.