VMware vCenter – Another Task Is Already In Progress

If you’ve ever tried to execute a task in vCenter only to be told that another task is already in progress but you can’t see it then this post may be of use to you.

In this example I tried to initiate a removal of all snapshots against a target virtual machine – vCenter then indicated that another task was in progress but when we look in the web interface to see running tasks nothing shows. This can be a common occurrence especially with long running tasks or where somebody reboots a vCenter.

To see what is happening we will connect via SSH to the host running the virtual machine and then leverage the vim-cmd commands to determine what is happening and how far the task has progressed.

The first command we will run is vim-cmd vmsvc/getallvms which will return a list of the running virtual machines on this host. This will allow us to identify the virtual machine ID which is required by our subsequent commands.

[root@BSA-ESXI04:~] vim-cmd vmsvc/getallvms

Vmid     Name                           File                                Guest OS          Version             Annotation         
122    BSA-NP   [BSA-NEW-3par01-Datastore04] BSA-NP/BSA-NP.vmx   windows8Server64Guest   vmx-15    [REMOVED]

[root@BSA-ESXI04:~]

In the above example our virtual machine ID is 122 – we can now feed this into our next command, vim-cmd vmsvc/get.tasklist

This command takes the VMID we determined previously and displays a list of all the tasks, as we see below there is already a snapshot removal task running.

[root@BSA-ESXI04:~] vim-cmd vmsvc/get.tasklist 122

(ManagedObjectReference) [
   'vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401'
]
[root@BSA-ESXI04:~]

Now we know what the running task is it would be useful to understand how far it has progressed and what our completion percentage is. We leverage vim-cmd vimsvc/task_info command which takes the task information from our previous command. We need to take the value for vim.Task: which in our case is haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401

[root@BSA-ESXI04:~] vim-cmd vimsvc/task_info haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401

(vim.TaskInfo) {

   key = "haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401",
   task = 'vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401',
   description = (vmodl.LocalizableMessage) null,
   name = "vim.VirtualMachine.removeAllSnapshots",
   descriptionId = "VirtualMachine.removeAllSnapshots",
   entity = 'vim.VirtualMachine:122',
   entityName = "BSA-NP",
   locked = <unset>,
   state = "running",
   cancelled = false,
   cancelable = true,
   error = (vmodl.MethodFault) null,
   result = <unset>,
   progress = 93,
   reason = (vim.TaskReasonUser) {
      userName = "vpxuser"
   },
   queueTime = "2019-07-05T08:12:03.51941Z",
   startTime = "2019-07-05T08:12:03.519649Z",
   completeTime = <unset>,
   eventChainId = 869472401,
   changeTag = <unset>,
   parentTaskKey = <unset>,
   rootTaskKey = <unset>,
   activationId = <unset>
}

[root@BSA-ESXI04:~] 

We get quite a bit of information returned but for me the most useful part is the progress value which in this case shows the task is 93% complete. We can run the command multiple times through the day to see how the task is doing and when it has completed.

Hopefully this information proves useful to you and helps identify those ‘hidden’ tasks which the web management interface may no longer display but which are still active.

9 thoughts on “VMware vCenter – Another Task Is Already In Progress”

  1. Estava com o mesmo problema executei os passos abaixo e resolveu,
    Do Console Local ou SSH:

    Faça login no SSH ou no console local como root.2.
    Execute esses comandos:/etc/init.d/hostd restart
    /etc/init.d/vpxa restart

    Reply
  2. vim-cmd vimsvc/task_cancel
    example: vim-cmd vimsvc/task_cancel haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401

    Reply
  3. Thank you for this quick detail and thank you to Tracy for the cancel command. I had a stalled vStorage vMotion that was stopping a VM from being powered on after a migration. This allowed me to find and cancel the offending registration command that was hung up.

    Reply
  4. I realize this is an old thread but am facing this same situation with queued snapshots that wont let me complete them. Whe I do the command to list the status of the tasks (or try to delete them) they show a status of (already delete or not completely created????):

    ([root@win-37ogfpmoa67:~] vim-cmd vimsvc/task_cancel haTask-122-vim.VirtualMachine.removeAllSnapshots-14278
    (vmodl.fault.ManagedObjectNotFound) {
    faultCause = (vmodl.MethodFault) null,
    faultMessage = ,
    obj = ‘vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-14278’
    msg = “Received SOAP response fault from [<<io_obj p:0x000000621f13ec48, h:5, , >, /sdk>]: cancel
    The object ‘vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-14278’ has already been deleted or has not been completely created”
    }
    )

    Is there another command to force these to go away or another way out of this state?

    Reply
    • I’m guessing the snapshot files still show against the VM/in the datastore? Out of curiosity, is this a VM on classic VMFS datastores? If the snap is large or there is high IO, it can result in a very long running task that I’ve seen throw this message. This wouldn’t be an issue on vVOLs but VMFS datastores do enjoy being troublesome with snapshots.

      I know some people have restarted the host management agents in situations like this, though that could be risky. Depending on the situation there is a chance of corruption but I have seen people regularly do this without issue. That doesn’t mean it can’t or won’t happen, but I have seen it play out OK.

      I have also come across people suggesting to remove the VM from your vCenter inventory and then add it back in again. I haven’t experience with going down this route so I can’t comment on the efficacy or safety.

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.