Troubleshooting System Center 2012 Virtual Machine Manager Object Locking Issues

Welcome back to the System Center: Virtual Machine Manager Engineering Blog. Lately, a few customers have called wanting to know why some jobs are failing in System Center 2012 Virtual Machine Manager (VMM 2012). Those of us who have been working with the product for a while know this happens, and that it can happen for a variety of reasons, however one of the more perplexing job failures is associated with object locking as reported in this sample error message:

Error (2602)

"Unable to perform the job because one or more of the selected objects are locked by another job"

Recommended Action

To find out which job is locking the object, in the Jobs view, group by Status and then find the running or canceling job for the object. When the job is complete, try again.

Object locking errors can be reported as part of a large job that completes overall but still contains errors. An example of this would be a System-level (background job) VM Refresher job executed against a managed host supporting a large number of virtual machines. A job of this magnitude could report one or more failures but the overall job status can be 'Completed w/Info.' The error message in this case could be Error (23234) indicating a refresh of one or more virtual machines running on a host could not be completed due to object locking. An example is shown below.

Multiple error messages of this type may be listed in the job Summary tab:

In this article we’ll discuss a little about what is going on behind the scenes with the hope it provides some additional understanding.

System Center Virtual Machine Manager Jobs

Most activities executed in Virtual Machine Manager are tracked collectively as 'Jobs' in the Virtual Machine Manager database. In general, VMM jobs fall into two groupings: System jobs and user initiated jobs. System jobs include the default refreshers such as a Virtual Machine Refresh or a Host Cluster Refresh. Manual jobs are user-initiated as part of VMM defined role (e.g. Administrator, Fabric Administrator, Tenant Administrator, etc.) activity.

A job is a PowerShell script executed by the Virtual Machine Manager server using WinRM\WMI calls to a VMM agent running on a remote target to complete an operation or series of operations. For the most part, a VMM job is a combination of tasks, and potentially subtasks, executed asynchronously (i.e. in parallel).

Virtual Machine Manager jobs either verify existing information (e.g. Virtual Machine periodic refresher), update existing information (e.g. adding a virtual hard disk to an existing virtual machine), or add new information to the SCVMM database (Add a Hyper-V Host, establish a connection with System Center Operations Manager server, etc.).

Virtual Machine Manager Jobs may also require access to information that already exists in the VMM database. The pieces of information contained within the various tables that make up the database are loosely referred to as 'objects.' To gain access to these objects, VMM creates tasks and subtasks (parts of an executing job) which place locks on the object they need as part of task\subtask execution. In general, these locks are Read, Write, Delete, NoDelete or NoLock locks. Here is a table of lock compatibility within VMM:

Lock Type	Lock Compatibility
NoDelete	NoDelete, Read, Write
Read	NoDelete, Read
Write	NoDelete
Delete	No other lock type
NoLock	All other locks

Here is an example of a Task executed as part of a Refresh a Service Instance job releasing all locks on a specific Service Instance identified by its ObjectID in the database:

Line 122354: 121452,20:15:51.165 08-20-2014,0x1478,0x2BFC,4,CarmineObjectLock.cs,719,0x00000000,CarmineObjectLock - Release All; Task 8b7c6aa1-b269-4926-b4c6-4f53be5e2726(Refresher:Refresh a ServiceInstance) Releasing Write lock on ServiceInstance:f99308f7-2aef-4d92-a206-2f649b179432,{00000000-0000-0000-0000-000000000000},4,

Addressing Job Failures due to Object Locking

When you experience job failures due to object locking you usually just want to understand the failure. The recommendation in the failed job is to simply search the jobs list for any running jobs, but no running jobs are usually found that might account for or explain the error. I admit I would be confused too. Others, perhaps more familiar with Virtual Machine Manager, who have experienced job failures due object locking will typically just right-click on the ‘Failed’ job and restart it. Most times the job will then complete because the lock on the required object no longer exists (perhaps a competing job completed and released the lock).

To gain a better understanding of this behavior, one needs to understand that not all VMM Jobs are visible in the Jobs View. As previously mentioned, there are jobs executed in the background by the VMM service itself (System Jobs) on pre-determined schedules. These are often referred to as ‘Refreshers’ and here are some examples:

Refresher	Default Setting
Host Refresher	30 minutes
Full VM Refresher	24 hours
Cluster Refresher	30 minutes
Library Refresher	User Configurable (Default = 1 hour)
Storage Light Refresher	2 hours

Every job is an independent entity and executed asynchronously. If a system generated refresh job is executing in the background and a user manually executes a job, the potential for job failure due to object locking increases. If, however, a user manually executes a job and a system refresher job executes, the system job will fail silently if it is unable to lock the objects it needs and then try again at the next scheduled interval. User initiated jobs, on the other hand, follow the path of 'try and wait' like all user-initiated tasks. Eventually, that job times out or fails perhaps due to locked objects. A failed job is considered finished, and the result is recorded in the database.

While there are different reasons for a job to fail, I hope the above information was useful in understanding how a job failure due to object locking might be due to multiple activities attempting to access the same database objects. In most cases, waiting a brief period will allow the conflicting activities to complete and you can restart the failed job. If a job continues to fail due to object locking, more in-depth troubleshooting is required beginning with the collection of a VMM trace. See the knowledge base article below for more information.

2913445 - How to enable debug logging in Virtual Machine Manager (http://support.microsoft.com/kb/2913445)

Thanks, and come back again soon.

Chuck Timon | Senior Support Escalation Engineer | Microsoft Enterprise Platforms Support
Dewitt Hurst | Senior Support Escalation Engineer | Microsoft Enterprise Platforms Support

Get the latest System Center news onFacebookandTwitter: