Troubleshooting WMI Performance Counter in SC VMM

Troubleshooting WMI Performance Counter in SC VMM

Recently I was troubleshooting an issue related to WMI Performance Counters in System Center Virtual Machine Manager 2012 R2.

Two Hyper-V clusters was added to SC VMM, one running Windows Server 2008 R2 and one running Windows Server 2012 R2. All cluster nodes in the 2008 R2 cluster was healthy, but there was problems with the 2012 R2 cluster nodes.
When adding the 2012 R2 cluster nodes to SC VMM, it failed on all of them with the following error:

Error (2912)

An internal error has occurred trying to contact the hyperv5.domain.local server:

WinRM: URL: [http://hyperv5.domain.local:5985], Verb: [ENUMERATE], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_PerfFormattedData_Tcpip_NetworkInterface], Filter: []

Unknown error (0xc0000bbb)

Recommended Action

Check that WS-Management service is installed and running on server hyperv5.domain.local. For more information use the command “winrm helpmsg hresult”. If hyperv5.domain.local is a host/library/update server or a PXE server role then ensure that VMM agent is installed and running. Refer to http://support.microsoft.com/kb/2742275 for more details.

The VMM agent was installed, but still had a “Pending” state in SC VMM.
We then removed and re-added the cluster in SC VMM, but the same thing happened. After some time the agents suddenly started responding and VMs was being discovered.
But there still was some problems, when refreshing a host it completed with this warning:

Error (20513)

The VMM management server cannot retrieve performance data for the computer hyperv5.domain.local. This issue may occur if the performance counter provider in the Virtual Machine Manager agent is corrupted.

Recommended Action

Restart the System Center Virtual Machine Manager Agent service on the computer hyperv5.domain.local. This automatically restarts the performance provider. If the error persists, reinstall the VMM agent on the computer hyperv5.domain.local.

The same error was shown in the host status:

Image

After some time the agents goes into a Not responding state, and we can see the following in the event logs on the hosts:

Faulting application name: vmmAgent.exe, version: 3.2.7634.0, time stamp: 0x532a5433

Faulting module name: vmmAgent.exe, version: 3.2.7634.0, time stamp: 0x532a5433

Exception code: 0xc0000005

Fault offset: 0x00000000003485da

Faulting process id: 0x1b3c

Faulting application start time: 0x01cf7014a2164f18

Faulting application path: C:Program FilesMicrosoft System Center 2012 R2Virtual Machine ManagerbinvmmAgent.exe

Faulting module path: C:Program FilesMicrosoft System Center 2012 R2Virtual Machine ManagerbinvmmAgent.exe

Along with the following event about 10 minutes later:

Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: HandleCount  Value: 16729 Maximum value: 4096 WMIPRVSE PID: 4200 Providers hosted in this process: C:WindowsSystem32wbemWmiPerfClass.dll, %systemroot%system32wbemwmiprov.dll, %systemroot%system32wbemwmiprov.dll

Initially we tried to run a consistency check on the WMI repository:

winmgmt  /salvagerepository

The above command performs a consistency check on the WMI repository, and if an inconsistency is detected, rebuilds the repository. The content of the inconsistent repository is merged into the rebuilt repository, if it can be read.

No errors was found, we then tried to rebuild the performance counters:

lodctr /R

That did not make a difference, the problem was still present. After running a number of other basic health checks like testing WinRM connectivity, verifying cluster health and so on we opened a support case with Microsoft Partner Support, which provided a solution:

The issue might be related to HP DSM. Also you have mentioned, all of your servers are HP servers. So could you try to rename HPPerfProv.dll and check if it can resolve the issue?

After renaming the DLL-file the issue was resolved and all hosts went into a normal condition in SC VMM.

The customer was advised to contact the vendor (HP) in order to get an updated DSM or ask whether they should uninstall the DSM software and use the built-in MPIO feature in Windows.

Jan Egil Ring works as a Lead Architect on the Infrastructure Team at Crayon, Norway. He mainly works with Microsoft server-products and has a strong passion for Windows PowerShell. In addition to being a consultant, he is a Microsoft Certified Trainer. He has obtained several certifications such as MCSE: Server Infrastructure and MCSE: Private Cloud. He is also a multiple-year recipient of the Microsoft Most Valuable Professional Award for his contributions in the Windows PowerShell technical community.

2 COMMENTS

Leave a Reply