Crayon Logo
Authors Posts by Jan Egil Ring

Jan Egil Ring

9 POSTS 1 COMMENTS
Jan Egil Ring works as a Lead Architect on the Infrastructure Team at Crayon, Norway. He mainly works with Microsoft server-products and has a strong passion for Windows PowerShell. In addition to being a consultant, he is a Microsoft Certified Trainer. He has obtained several certifications such as MCSE: Server Infrastructure and MCSE: Private Cloud. He is also a multiple-year recipient of the Microsoft Most Valuable Professional Award for his contributions in the Windows PowerShell technical community.

Recently we were troubleshooting an issue in System Center Virtual Machine Manager 2012 R2 where running a compliance scan against Hyper-V hosts was failing with the following error:

VMM Error 2931

Error (2931) VMM is unable to complete the request. The connection to the VMM agent on the virtualization server (server01.domain.local) was lost. Unknown error (0x80338029)

The issue was present on all Hyper-V hosts, most of them running Windows Server 2012 R2. The latest Windows updates was installed on the hosts, and Virtual Machine Manager was running the latest Update Rollup available.

During troubleshooting we uninstalled System Center Endpoint Protection from the Hyper-V hosts, and noticed that compliance scan worked immediately.

After opening a support case with Microsoft, we were advised to decline all superseded updates for System Center Endpoint Protection from the WSUS server.

After doing that, compliance scan worked as expected on hosts with System Center Endpoint Protection installed.

The issue is explained in more detail in this article, where there is also a Windows PowerShell script available for automating the cleanup process. It can be accomplished by using the WSUS clean-up wizard, but this process should be automated and run on a scheduled basis for example once a month to prevent the issue from happening again.

The referenced article states the following:

A hotfix for the Windows Update Agent is currently in development. The update will change how the update metadata is loaded into memory. Pending the results of final testing, this fix should be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.

From a VMM perspective, the error message is quite misleading when compared to what the underlying root cause is. A suggestion to improve the error message has been posted to Microsoft’s VMM User Voice.

The 4th edition of the Nordic Infrastructure Conference is happening in Oslo, February 11-13 2015.

Many exciting things are going on in the world of PowerShell, with a new version coming up later this year and many improvements around Desired State Configuration.

If you are into PowerShell and automation there are many interesting sessions scheduled at the conference you don`t want to miss:

 

PowerShell Workflows: Prepare yourself for SMA and Azure Automation

Aleksandar Nikolic

Thursday 10:00 – 11:00

 

System Center – Service Management Automation – Introduction

Jakob Gottlieb Svendsen

Thursday 13:20 – 14:20

 

System Center – Service Management Automation – Automating System Center 2012 R2

Jakob Gottlieb Svendsen

Thursday 16:00 – 17:00

 

The State of PowerShell Desired State Configuration

Jan Egil Ring

Friday 10:20 – 11:20

 

PowerShell Desired State Configuration – Make it so

David O’Brien

Friday 12:20 – 13:20

 

Managing Microsoft Azure IaaS with Windows PowerShell DSC

Aleksandar Nikolic

Friday 15:00 – 16:00

 

You can build your own agenda on this page.

Hope to see you at the conference!

In this article we will look at how to resolve the issue of missing Agent, Traps and Security tabs for the SNMP Service  in the Services MMC-snapin.

After installing the SNMP Service, you might experience that only the following tabs is available:

image_thumb

In order for the SNMP management-tabs to show, the Remote Server and Administration Tools feature SNMP Tools must also be installed.

Two possible ways to install the SNMP Tools is to leverage either Server Manager or the ServerManager PowerShell module.

Here is an example on how to perform the task using the ServerManager PowerShell module.

After re-opening the Services MMC-snapin, the missing tabs should be visible.

Bonus tip: PowerShell MVP Fabien Dibot has created a PowerShell Desired State Configuration Resource for configuring the SNMP Service, available here.

Here is an example configuration showing how to leverage the Resource:

In this article we will look at an issue related to Storage Live Migration in Windows Server 2012 R2, where the source VHD/VHDX-files is not deleted when the migration completes.

A while ago we started noticing at several customers and our own lab-environment that source VHD/VHDX-files was not deleted when the migration completes. In the Hyper-V Virtual Machine Management Service (VMMS) event log on the Hyper-V host where the virtual machine is running, event 20878 was logged with the error Failed to delete file ‘<Path to VHD/VHDX-file>’: ‘General access denied error'(‘0x80070005’):

WS2012R2_Storage_Migration_event_20878

To reproduce the problem, create a new virtual machine and perform a Storage Live Migration. A strange behavior we noticed is that the issue is not present on subsequent Storage Live Migrations of the same virtual machine.

After troubleshooting the issue for a while we opened a support case with Microsoft, who eventually advised us to uninstall the following hotfix:

KB2966407 – Backing up virtual machines fails when using the CSV writer after installation of update 2919355 in Windows

The issue seems to be resolved after uninstalling the hotfix. The current status is reported back to the Hyper-V development team, and we would expect an upcoming hotfix to resolve the issue without needing to uninstall KB2966407.

This article will be updated when new information is available.

In this article we will look at how we can ease the process of doing housekeeping on Cluster Shared Volumes used for storing virtual machines in Hyper-V.

Hyper-V Virtual Disk Files (VHD or VHDX) might be inactive for a number of reasons:

  • The Virtual Machine was deleted from Hyper-V Manager, this will not delete the Virtual Disk Files
  • The Virtual Machine was deleted from System Center Virtual Machine Manager without the option to delete the virtual hard disk files
  • Old files and folders present from previous migrations

In addition, I am also working on  troubleshooting a bug together with Microsoft support where the source Virtual Disk Files does not get deleted when performing a Storage Live Migration in Windows Server 2012 R2, thus this might also be a possible reason for inactive Virtual Disk Files.

In this specific case I was tasked with providing a report of all inactive disk files in an environment with several hundred virtual machines stored on  a relatively large number of Cluster Shared Volumes. This would be a very cumbersome task to do manually, so I wrote a small PowerShell script which I`ve published here.

The script can be run from any computer with PowerShell 3.0 or later and the Virtual Machine Manager module installed. The following steps is performed by the script:

  • Retrieve all active Virtual Disk Files in use in the specified cluster (also including checkpoints/snapshots)
  • Retrieve all Cluster Shared Volumes from the specified cluster
  • Retrive all Virtual Disk Files present on the Cluster Shared Volumes
  • Compare the list of active files with the list of present files
  • Return the size of all inactive files

Note that the script is provided “as-is” in order to show how PowerShell can be leveraged to quickly build useful reports which can save us time in day to day operations. If the script was to be run in an automated fashion, additional error handling, logging and such should be added. Feel free to further extend the script with support for other storage scenarios like local disks, Scale-Out File Server shares and so on.

I could also have included an option to delete inactive files automatically, but as a precaution I would suggest that the returned list of inactive files is reviewed manually. The reason is that some files should be archived instead of deleted or if there is a bug in the script. I also had one occurance of a virtual machine present in the cluster which was not yet discovered by SC VMM, this would lead to the Virtual Disk Files for the virtual machine being reported as inactive.

At the environment I ran this script to generate a report of inactive Virtual Disk Files, we were able to reclaim close to 2 TB of data.

Bonus tip – ISO files may also use a lot of disk space if the “Share file instead of copying it” option is not leveraged. Here is a few one-liners to work with mounted ISO-files:

 

In this article we will look at a connection problem for migrated virtual machines in Windows Server 2012/2012 R2 VDI.

The scenario is a new Windows Server 2012 R2 VDI deployment, where a a new collection is created for personal unmanaged Windows 7 virtual machines. These machines was migrated from an existing Windows Server 2008 R2 VDI deployment.

The problem occurred when launching the Remote Desktop shortcut for the VDI collection Remote Desktop Web Access. The progress bar stays on the “Loading the virtual machine” status for 1-2 minutes, and will eventually time out with the message “Your computer can’t connect to the remote computer because an error occurred on the remote computer that you want to connect to”. On the server side the message “Remote Desktop Services has taken too long to complete the client connection” is logged in the “Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin” event log:

image

A different VDI collection for new Windows 8.1 virtual machines was also setup, and they did not have this problem.

On the Hyper-V servers hosting the VDI virtual machines, the Remote Desktop Virtualization Host server role is installed in order to orchestrate and integrate with the Remote Desktop Connection broker. When installed the “Remote Desktop Virtualization Host Agent” service is present:

image

This service logs activity in the Microsoft-Windows-TerminalServices-TSV-VmHostAgent/Operational event log, and we could see the following message when the Windows 8.1 virtual machines was logged on to:

image

However, for the Windows 7 virtual machines we did not see any activity in this event log.

The Hyper-V Integration Services was upgraded to the Windows Server 2012 R2 version on the problematic virtual machines, and we could not see any problems in the Device Manager:

RDV_2008R2

 

At this point, we found the following KB-article which led us to the solution:

Connection hangs after you import virtual machines into Windows Server 2012 Remote Desktop Services Unmanaged Pool

 

The article states:

The problem occurs because Windows Server 2012 VDI virtual machines add a Remote Desktop Virtualization (RDV) device that does not exist in virtual machines that are not created by using Windows Server 2012. Without this device, the VDI RDP client cannot connect to the virtual machines, and the connections hangs.
To work around this problem, re-create the virtual machines in Windows Server 2012 and copy over the Virtual Hard Disk (VHD) instead of exporting and importing the virtual machines.

 

After re-creating one virtual machine for testing purposes, the following new device was found in the Device Manager:

RDV_present

 

The remote desktop connection to the virtual machine also worked without issues.

In summary: When migrating virtual machines from pre-Windows Server 2012 VDI environments, you should not import the virtual machines in the new environment. They should be re-created and attached to the virtual hard disks copied from the old environment. The Hyper-V Integration Services should also be upgraded to the latest version, if not the migrated virtual machines won`t have drivers for the new virtual devices such as the “Microsoft Hyper-V Remote Desktop Virtualization” device needed for VDI:

 

RDV_2012R2_before_IC

 

At the end, I would also like to mention two other articles which provides information about issues you might run into when configuring Windows Server 2012/2012 R2 VDI environments:

A great feature for learning the underlying PowerShell commands when performing an administrative action in Exchange Server 2007/2010, was the Exchange Management Console which showed the PowerShell commands. In Exchange Server 2013 the MMC-based Exchange Management Console was replaced by the web-based Exchange Management Console, which unfortunately did not show PowerShell commands. With the release of Service Pack 1 for Exchange Server 2013, the Exchange team brought the PowerShell Command Logging feature for providing similar capabilities.

In Exchange Server 2010, we could see the PowerShell commands at the end of the wizard when creating a new object:

image

We also had the “Show Exchange Management Shell command” button available when performing changes to an existing object:

image

Pressing the button would show us the PowerShell commands:

image

In Exchange Server 2013 SP1, the “Show Command Logging” option is available in the help menu in the upper right corner in the Exchange Admin Center:

image

This will open a new window where all commands from actions made in the Exchange Admin Center will be logged:

image

As an example, we are creating a new mailbox:

image

 

After pressing the Save-button, the PowerShell command for creating the mailbox is shown in the Command Logging window:

image

 

As we can see from the above screenshot, the Get- cmdlets is also logged when navigating around in the user interface.

In summary, the new Show Command Logging feature in Exchange Server 2013 Service Pack 1 provides a great way for Exchange administrators to learn how to perform an administrative task in PowerShell by first doing it in the graphical web based GUI, and then looking in the Command Logging window.

Recently I was troubleshooting an issue related to WMI Performance Counters in System Center Virtual Machine Manager 2012 R2.

Two Hyper-V clusters was added to SC VMM, one running Windows Server 2008 R2 and one running Windows Server 2012 R2. All cluster nodes in the 2008 R2 cluster was healthy, but there was problems with the 2012 R2 cluster nodes.
When adding the 2012 R2 cluster nodes to SC VMM, it failed on all of them with the following error:

Error (2912)

An internal error has occurred trying to contact the hyperv5.domain.local server:

WinRM: URL: [http://hyperv5.domain.local:5985], Verb: [ENUMERATE], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_PerfFormattedData_Tcpip_NetworkInterface], Filter: []

Unknown error (0xc0000bbb)

Recommended Action

Check that WS-Management service is installed and running on server hyperv5.domain.local. For more information use the command “winrm helpmsg hresult”. If hyperv5.domain.local is a host/library/update server or a PXE server role then ensure that VMM agent is installed and running. Refer to http://support.microsoft.com/kb/2742275 for more details.

The VMM agent was installed, but still had a “Pending” state in SC VMM.
We then removed and re-added the cluster in SC VMM, but the same thing happened. After some time the agents suddenly started responding and VMs was being discovered.
But there still was some problems, when refreshing a host it completed with this warning:

Error (20513)

The VMM management server cannot retrieve performance data for the computer hyperv5.domain.local. This issue may occur if the performance counter provider in the Virtual Machine Manager agent is corrupted.

Recommended Action

Restart the System Center Virtual Machine Manager Agent service on the computer hyperv5.domain.local. This automatically restarts the performance provider. If the error persists, reinstall the VMM agent on the computer hyperv5.domain.local.

The same error was shown in the host status:

Image

After some time the agents goes into a Not responding state, and we can see the following in the event logs on the hosts:

Faulting application name: vmmAgent.exe, version: 3.2.7634.0, time stamp: 0x532a5433

Faulting module name: vmmAgent.exe, version: 3.2.7634.0, time stamp: 0x532a5433

Exception code: 0xc0000005

Fault offset: 0x00000000003485da

Faulting process id: 0x1b3c

Faulting application start time: 0x01cf7014a2164f18

Faulting application path: C:Program FilesMicrosoft System Center 2012 R2Virtual Machine ManagerbinvmmAgent.exe

Faulting module path: C:Program FilesMicrosoft System Center 2012 R2Virtual Machine ManagerbinvmmAgent.exe

Along with the following event about 10 minutes later:

Windows Management Instrumentation has stopped WMIPRVSE.EXE because a quota reached a warning value. Quota: HandleCount  Value: 16729 Maximum value: 4096 WMIPRVSE PID: 4200 Providers hosted in this process: C:WindowsSystem32wbemWmiPerfClass.dll, %systemroot%system32wbemwmiprov.dll, %systemroot%system32wbemwmiprov.dll

Initially we tried to run a consistency check on the WMI repository:

winmgmt  /salvagerepository

The above command performs a consistency check on the WMI repository, and if an inconsistency is detected, rebuilds the repository. The content of the inconsistent repository is merged into the rebuilt repository, if it can be read.

No errors was found, we then tried to rebuild the performance counters:

lodctr /R

That did not make a difference, the problem was still present. After running a number of other basic health checks like testing WinRM connectivity, verifying cluster health and so on we opened a support case with Microsoft Partner Support, which provided a solution:

The issue might be related to HP DSM. Also you have mentioned, all of your servers are HP servers. So could you try to rename HPPerfProv.dll and check if it can resolve the issue?

After renaming the DLL-file the issue was resolved and all hosts went into a normal condition in SC VMM.

The customer was advised to contact the vendor (HP) in order to get an updated DSM or ask whether they should uninstall the DSM software and use the built-in MPIO feature in Windows.

Service Management Automation (SMA) is a new component in System Center for process automation.

Before looking into the technical details, lets have a look at what practical use cases there are for an automation product like SMA:

  • Automate maintenance tasks such as orchestrating patching of Windows Failover clusters, stop and start services as well as starting and stopping virtual machines in a defined order when doing maintenance work
  • Automate business processes such as creating, changing and removing user accounts, password resets and automating file transfers
  • Automating tasks in a service catalog and change management such as creating virtual machines and automatic configuration of backups
  • Dynamic resource allocation based on load or calendar. A practical example might be extending a web farm with additional instances before the Christmas holidays
  • Respond to alarms from operations tools such as System Center Operations Manager
  • Integrations across systems, such as automatically creating an incident in an incident management system based on an event from an operations system

SMA is not a separate component in the System Center suite on the same level as the other components such as Configuration Manager and Virtual Machine Manager, the installation files is included in the installation media for Orchestrator:

clip_image002

As we can see based on the above screenshot, SMA has 3 components:

  • Runbook Worker – Executes runbook jobs. These might be scheduled or triggered manually.
  • PowerShell Module – Cmdlets for administering SMA.
  • Web Service – Web API (REST OData) used to several tasks such as communication with Windows Azure Pack, distribute runbook-jobs to runbook workers as well as delegating permissions.

In addition a SQL-database is needed to store data (needs to be specified during installation).

From an architecture level there are similarities between System Center Orchestrator (SCO) and Service Management Automation (SMA). The main difference in the user experience is that SCO has a graphical interface with drag and drop support, while SMA is 100% based on Windows PowerShell Workflow.

PowerShell Workflow, introduced in PowerShell 3.0, is built upon Windows Workflow Foundation. PowerShell Workflow makes it possible to orchestrate long-running activities and automate complex tasks, such as deploying a service containing several servers. PowerShell Workflow got several capabilities such as executing activities in parallell, resume execution after a failure (such as network outage) and continue after a reboot. Details and links to more information about PowerShell Workflow is available in the article «When Windows PowerShell Met Workflow» on the Windows PowerShell Team blog.

Portal

SMA itself does not have a built-in portal, the only way to define runbooks is using the PowerShell cmdlets or the web service. Windows Azure Pack for Windows Server is a free component for hosting providers and enterprises which provides a user experience consistent with Windows Azure:

clip_image004

One of the components in Windows Azure Pack is Automation, which is based on a connection to the SMA web service. This makes it possible to use the Automation component available in Windows Azure Pack as a web portal for SMA:

clip_image006

The first item we see when entering the Automation portal is Dashboard, which provides an overall view of the defined runbooks.

Runbooks provides a view of all runbooks defined, with options for sorting based on status, tags or search query:

clip_image008

Clicking on a runbook brings you to a dedicated dashboard for the selected runbook, where statistics for the runbook jobs is available:

clip_image010

Jobs provides history and output from each job instance, which has be executed for the runbook:

clip_image012

Author shows the published version of the runbook:

clip_image014

Draft provides the ability to edit and test the code defined in the runbook: clip_image016

All runbooks have to start with the keyword for PowerShell Workflow followed by the name of the runbook. Optionally a param block can be provided in order to define parameters for the runbook. The actual code is defined after the param block.

If parameters is defined, values for these may be provided when starting the runbook:

clip_image018

These values can also be provided if the runbook is sceduled.
Schedules may be defined in the next menu item:

clip_image020

clip_image022

The last menu item for a runbook is Configure, giving options such as providing a description, tags and enabling debugging and logging:

clip_image024

Debugging and logging will require much space in the database, and should only be enabled during troubleshooting.

The last item we will have a look at in the Automation portal in Windows Azure Pack is Assets:

clip_image026

An Asset may be PowerShell modules or one of the following:

clip_image028

Connections is connections to other systems, such as other System Center components like Data Protection Manager and Virtual Machine Manager. Credentials and variables can be defined globally in order to avoid hardcoding them in runbooks. The last type of asset is scheduled, which we have already looked at.

Architecture

When PowerShell Workflow is used outside of SMA, state («persistence») is stored locally on the machine executing the workflow. With SMA we get a highly available workflow, because persistence is stored in a SQL-database.

This makes it possible to build a highly available automation platform, by configuring a highly available SQL-service (using clustering or AlwaysOn) as well as 2 or more servers with the SMA Runbook Worker and Webservice installed:

clip_image030

Source of illustration: System Center Orchestrator Engineering Team Blog

At the Build conference in April 2014 Microsoft also announced Microsoft Azure Automation preview, making it possible to leverage SMA without an on-premise automation platform.

Summary

Since SMA is based on PowerShell Workflow its possible to automate everything which can be accomplished from PowerShell. Unlike Orchestrator which can be extended by Integration Packs, SMA can be extended using PowerShell modules.

The first version of the product (“V1”) lacks some functionality available in Orchestrator, such as the ability to configure runbook permissions. This is a feature an enterprise using SMA would find useful in order to delegate access to runbooks based on for example Active Directory security groups.

There are no official statements regarding the strategy around Orchestrator and SMA will be in the future, but its not unlikely that SMA will overtake Orchestrators role in the Microsoft ecosystem when more functionality comes in place.

Resources

Overview of Service Management Automation

Microsoft Azure Automation preview

Windows Azure Pack for Windows Server

Getting Started with Windows PowerShell Workflow

Intro to SMA

SMA Capabilities in Depth