Crayon Logo
Authors Posts by Jan Egil Ring

Jan Egil Ring

Jan Egil Ring works as a Lead Architect on the Infrastructure Team at Crayon, Norway. He mainly works with Microsoft server-products and has a strong passion for Windows PowerShell. In addition to being a consultant, he is a Microsoft Certified Trainer. He has obtained several certifications such as MCSE: Server Infrastructure and MCSE: Private Cloud. He is also a multiple-year recipient of the Microsoft Most Valuable Professional Award for his contributions in the Windows PowerShell technical community.

Recently we were troubleshooting an issue in System Center Virtual Machine Manager 2012 R2 where running a compliance scan against Hyper-V hosts was failing with the following error:

VMM Error 2931

Error (2931) VMM is unable to complete the request. The connection to the VMM agent on the virtualization server (server01.domain.local) was lost. Unknown error (0x80338029)

The issue was present on all Hyper-V hosts, most of them running Windows Server 2012 R2. The latest Windows updates was installed on the hosts, and Virtual Machine Manager was running the latest Update Rollup available.

During troubleshooting we uninstalled System Center Endpoint Protection from the Hyper-V hosts, and noticed that compliance scan worked immediately.

After opening a support case with Microsoft, we were advised to decline all superseded updates for System Center Endpoint Protection from the WSUS server.

After doing that, compliance scan worked as expected on hosts with System Center Endpoint Protection installed.

The issue is explained in more detail in this article, where there is also a Windows PowerShell script available for automating the cleanup process. It can be accomplished by using the WSUS clean-up wizard, but this process should be automated and run on a scheduled basis for example once a month to prevent the issue from happening again.

The referenced article states the following:

A hotfix for the Windows Update Agent is currently in development. The update will change how the update metadata is loaded into memory. Pending the results of final testing, this fix should be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.

From a VMM perspective, the error message is quite misleading when compared to what the underlying root cause is. A suggestion to improve the error message has been posted to Microsoft’s VMM User Voice.

The 4th edition of the Nordic Infrastructure Conference is happening in Oslo, February 11-13 2015.

Many exciting things are going on in the world of PowerShell, with a new version coming up later this year and many improvements around Desired State Configuration.

If you are into PowerShell and automation there are many interesting sessions scheduled at the conference you don`t want to miss:


PowerShell Workflows: Prepare yourself for SMA and Azure Automation

Aleksandar Nikolic

Thursday 10:00 – 11:00


System Center – Service Management Automation – Introduction

Jakob Gottlieb Svendsen

Thursday 13:20 – 14:20


System Center – Service Management Automation – Automating System Center 2012 R2

Jakob Gottlieb Svendsen

Thursday 16:00 – 17:00


The State of PowerShell Desired State Configuration

Jan Egil Ring

Friday 10:20 – 11:20


PowerShell Desired State Configuration – Make it so

David O’Brien

Friday 12:20 – 13:20


Managing Microsoft Azure IaaS with Windows PowerShell DSC

Aleksandar Nikolic

Friday 15:00 – 16:00


You can build your own agenda on this page.

Hope to see you at the conference!

In this article we will look at an issue related to Storage Live Migration in Windows Server 2012 R2, where the source VHD/VHDX-files is not deleted when the migration completes.

A while ago we started noticing at several customers and our own lab-environment that source VHD/VHDX-files was not deleted when the migration completes. In the Hyper-V Virtual Machine Management Service (VMMS) event log on the Hyper-V host where the virtual machine is running, event 20878 was logged with the error Failed to delete file ‘<Path to VHD/VHDX-file>’: ‘General access denied error'(‘0x80070005’):


To reproduce the problem, create a new virtual machine and perform a Storage Live Migration. A strange behavior we noticed is that the issue is not present on subsequent Storage Live Migrations of the same virtual machine.

After troubleshooting the issue for a while we opened a support case with Microsoft, who eventually advised us to uninstall the following hotfix:

KB2966407 – Backing up virtual machines fails when using the CSV writer after installation of update 2919355 in Windows

The issue seems to be resolved after uninstalling the hotfix. The current status is reported back to the Hyper-V development team, and we would expect an upcoming hotfix to resolve the issue without needing to uninstall KB2966407.

This article will be updated when new information is available.

In this article we will look at how we can ease the process of doing housekeeping on Cluster Shared Volumes used for storing virtual machines in Hyper-V.

Hyper-V Virtual Disk Files (VHD or VHDX) might be inactive for a number of reasons:

  • The Virtual Machine was deleted from Hyper-V Manager, this will not delete the Virtual Disk Files
  • The Virtual Machine was deleted from System Center Virtual Machine Manager without the option to delete the virtual hard disk files
  • Old files and folders present from previous migrations

In addition, I am also working on  troubleshooting a bug together with Microsoft support where the source Virtual Disk Files does not get deleted when performing a Storage Live Migration in Windows Server 2012 R2, thus this might also be a possible reason for inactive Virtual Disk Files.

In this specific case I was tasked with providing a report of all inactive disk files in an environment with several hundred virtual machines stored on  a relatively large number of Cluster Shared Volumes. This would be a very cumbersome task to do manually, so I wrote a small PowerShell script which I`ve published here.

The script can be run from any computer with PowerShell 3.0 or later and the Virtual Machine Manager module installed. The following steps is performed by the script:

  • Retrieve all active Virtual Disk Files in use in the specified cluster (also including checkpoints/snapshots)
  • Retrieve all Cluster Shared Volumes from the specified cluster
  • Retrive all Virtual Disk Files present on the Cluster Shared Volumes
  • Compare the list of active files with the list of present files
  • Return the size of all inactive files

Note that the script is provided “as-is” in order to show how PowerShell can be leveraged to quickly build useful reports which can save us time in day to day operations. If the script was to be run in an automated fashion, additional error handling, logging and such should be added. Feel free to further extend the script with support for other storage scenarios like local disks, Scale-Out File Server shares and so on.

I could also have included an option to delete inactive files automatically, but as a precaution I would suggest that the returned list of inactive files is reviewed manually. The reason is that some files should be archived instead of deleted or if there is a bug in the script. I also had one occurance of a virtual machine present in the cluster which was not yet discovered by SC VMM, this would lead to the Virtual Disk Files for the virtual machine being reported as inactive.

At the environment I ran this script to generate a report of inactive Virtual Disk Files, we were able to reclaim close to 2 TB of data.

Bonus tip – ISO files may also use a lot of disk space if the “Share file instead of copying it” option is not leveraged. Here is a few one-liners to work with mounted ISO-files:


In this article we will look at a connection problem for migrated virtual machines in Windows Server 2012/2012 R2 VDI.

The scenario is a new Windows Server 2012 R2 VDI deployment, where a a new collection is created for personal unmanaged Windows 7 virtual machines. These machines was migrated from an existing Windows Server 2008 R2 VDI deployment.

The problem occurred when launching the Remote Desktop shortcut for the VDI collection Remote Desktop Web Access. The progress bar stays on the “Loading the virtual machine” status for 1-2 minutes, and will eventually time out with the message “Your computer can’t connect to the remote computer because an error occurred on the remote computer that you want to connect to”. On the server side the message “Remote Desktop Services has taken too long to complete the client connection” is logged in the “Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin” event log:


A different VDI collection for new Windows 8.1 virtual machines was also setup, and they did not have this problem.

On the Hyper-V servers hosting the VDI virtual machines, the Remote Desktop Virtualization Host server role is installed in order to orchestrate and integrate with the Remote Desktop Connection broker. When installed the “Remote Desktop Virtualization Host Agent” service is present:


This service logs activity in the Microsoft-Windows-TerminalServices-TSV-VmHostAgent/Operational event log, and we could see the following message when the Windows 8.1 virtual machines was logged on to:


However, for the Windows 7 virtual machines we did not see any activity in this event log.

The Hyper-V Integration Services was upgraded to the Windows Server 2012 R2 version on the problematic virtual machines, and we could not see any problems in the Device Manager:



At this point, we found the following KB-article which led us to the solution:

Connection hangs after you import virtual machines into Windows Server 2012 Remote Desktop Services Unmanaged Pool


The article states:

The problem occurs because Windows Server 2012 VDI virtual machines add a Remote Desktop Virtualization (RDV) device that does not exist in virtual machines that are not created by using Windows Server 2012. Without this device, the VDI RDP client cannot connect to the virtual machines, and the connections hangs.
To work around this problem, re-create the virtual machines in Windows Server 2012 and copy over the Virtual Hard Disk (VHD) instead of exporting and importing the virtual machines.


After re-creating one virtual machine for testing purposes, the following new device was found in the Device Manager:



The remote desktop connection to the virtual machine also worked without issues.

In summary: When migrating virtual machines from pre-Windows Server 2012 VDI environments, you should not import the virtual machines in the new environment. They should be re-created and attached to the virtual hard disks copied from the old environment. The Hyper-V Integration Services should also be upgraded to the latest version, if not the migrated virtual machines won`t have drivers for the new virtual devices such as the “Microsoft Hyper-V Remote Desktop Virtualization” device needed for VDI:




At the end, I would also like to mention two other articles which provides information about issues you might run into when configuring Windows Server 2012/2012 R2 VDI environments:

A great feature for learning the underlying PowerShell commands when performing an administrative action in Exchange Server 2007/2010, was the Exchange Management Console which showed the PowerShell commands. In Exchange Server 2013 the MMC-based Exchange Management Console was replaced by the web-based Exchange Management Console, which unfortunately did not show PowerShell commands. With the release of Service Pack 1 for Exchange Server 2013, the Exchange team brought the PowerShell Command Logging feature for providing similar capabilities.

In Exchange Server 2010, we could see the PowerShell commands at the end of the wizard when creating a new object:


We also had the “Show Exchange Management Shell command” button available when performing changes to an existing object:


Pressing the button would show us the PowerShell commands:


In Exchange Server 2013 SP1, the “Show Command Logging” option is available in the help menu in the upper right corner in the Exchange Admin Center:


This will open a new window where all commands from actions made in the Exchange Admin Center will be logged:


As an example, we are creating a new mailbox:



After pressing the Save-button, the PowerShell command for creating the mailbox is shown in the Command Logging window:



As we can see from the above screenshot, the Get- cmdlets is also logged when navigating around in the user interface.

In summary, the new Show Command Logging feature in Exchange Server 2013 Service Pack 1 provides a great way for Exchange administrators to learn how to perform an administrative task in PowerShell by first doing it in the graphical web based GUI, and then looking in the Command Logging window.

Recently I was troubleshooting an issue related to WMI Performance Counters in System Center Virtual Machine Manager 2012 R2.

Two Hyper-V clusters was added to SC VMM, one running Windows Server 2008 R2 and one running Windows Server 2012 R2. All cluster nodes in the 2008 R2 cluster was healthy, but there was problems with the 2012 R2 cluster nodes.
When adding the 2012 R2 cluster nodes to SC VMM, it failed on all of them with the following error:

Error (2912)

An internal error has occurred trying to contact the hyperv5.domain.local server:

WinRM: URL: [http://hyperv5.domain.local:5985], Verb: [ENUMERATE], Resource: [], Filter: []

Unknown error (0xc0000bbb)

Recommended Action

Check that WS-Management service is installed and running on server hyperv5.domain.local. For more information use the command “winrm helpmsg hresult”. If hyperv5.domain.local is a host/library/update server or a PXE server role then ensure that VMM agent is installed and running. Refer to for more details.

The VMM agent was installed, but still had a “Pending” state in SC VMM.
We then removed and re-added the cluster in SC VMM, but the same thing happened. After some time the agents suddenly started responding and VMs was being discovered.
But there still was some problems, when refreshing a host it completed with this warning:

Error (20513)

The VMM management server cannot retrieve performance data for the computer hyperv5.domain.local. This issue may occur if the performance counter provider in the Virtual Machine Manager agent is corrupted.

Recommended Action

Restart the System Center Virtual Machine Manager Agent service on the computer hyperv5.domain.local. This automatically restarts the performance provider. If the error persists, reinstall the VMM agent on the computer hyperv5.domain.local.

The same error was shown in the host status:


After some time the agents goes into a Not responding state, and we can see the following in the event logs on the hosts:

Faulting application name: vmmAgent.exe, version: 3.2.7634.0, time stamp: 0x532a5433

Faulting module name: vmmAgent.exe, version: 3.2.7634.0, time stamp: 0x532a5433

Exception code: 0xc0000005

Fault offset: 0x00000000003485da

Faulting process id: 0x1b3c

Faulting application start time: 0x01cf7014a2164f18

Faulting application path: C:Program FilesMicrosoft System Center 2012 R2Virtual Machine ManagerbinvmmAgent.exe

Faulting module path: C:Program FilesMicrosoft System Center 2012 R2Virtual Machine ManagerbinvmmAgent.exe

Along with the following event about 10 minutes later:

Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: HandleCount  Value: 16729 Maximum value: 4096 WMIPRVSE PID: 4200 Providers hosted in this process: C:WindowsSystem32wbemWmiPerfClass.dll, %systemroot%system32wbemwmiprov.dll, %systemroot%system32wbemwmiprov.dll

Initially we tried to run a consistency check on the WMI repository:

winmgmt  /salvagerepository

The above command performs a consistency check on the WMI repository, and if an inconsistency is detected, rebuilds the repository. The content of the inconsistent repository is merged into the rebuilt repository, if it can be read.

No errors was found, we then tried to rebuild the performance counters:

lodctr /R

That did not make a difference, the problem was still present. After running a number of other basic health checks like testing WinRM connectivity, verifying cluster health and so on we opened a support case with Microsoft Partner Support, which provided a solution:

The issue might be related to HP DSM. Also you have mentioned, all of your servers are HP servers. So could you try to rename HPPerfProv.dll and check if it can resolve the issue?

After renaming the DLL-file the issue was resolved and all hosts went into a normal condition in SC VMM.

The customer was advised to contact the vendor (HP) in order to get an updated DSM or ask whether they should uninstall the DSM software and use the built-in MPIO feature in Windows.