If you’ve ever tried to execute a task in vCenter only to be told that another task is already in progress but you can’t see it then this post may be of use to you.
In this example I tried to initiate a removal of all snapshots against a target virtual machine – vCenter then indicated that another task was in progress but when we look in the web interface to see running tasks nothing shows. This can be a common occurrence especially with long running tasks or where somebody reboots a vCenter.
To see what is happening we will connect via SSH to the host running the virtual machine and then leverage the vim-cmd
commands to determine what is happening and how far the task has progressed.
The first command we will run is vim-cmd vmsvc/getallvms
which will return a list of the running virtual machines on this host. This will allow us to identify the virtual machine ID which is required by our subsequent commands.
[root@BSA-ESXI04:~] vim-cmd vmsvc/getallvms
Vmid Name File Guest OS Version Annotation
122 BSA-NP [BSA-NEW-3par01-Datastore04] BSA-NP/BSA-NP.vmx windows8Server64Guest vmx-15 [REMOVED]
[root@BSA-ESXI04:~]
In the above example our virtual machine ID is 122 – we can now feed this into our next command, vim-cmd vmsvc/get.tasklist
This command takes the VMID we determined previously and displays a list of all the tasks, as we see below there is already a snapshot removal task running.
[root@BSA-ESXI04:~] vim-cmd vmsvc/get.tasklist 122
(ManagedObjectReference) [
'vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401'
]
[root@BSA-ESXI04:~]
Now we know what the running task is it would be useful to understand how far it has progressed and what our completion percentage is. We leverage vim-cmd vimsvc/task_info
command which takes the task information from our previous command. We need to take the value for vim.Task:
which in our case is haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401
[root@BSA-ESXI04:~] vim-cmd vimsvc/task_info haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401
(vim.TaskInfo) {
key = "haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401",
task = 'vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401',
description = (vmodl.LocalizableMessage) null,
name = "vim.VirtualMachine.removeAllSnapshots",
descriptionId = "VirtualMachine.removeAllSnapshots",
entity = 'vim.VirtualMachine:122',
entityName = "BSA-NP",
locked = <unset>,
state = "running",
cancelled = false,
cancelable = true,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 93,
reason = (vim.TaskReasonUser) {
userName = "vpxuser"
},
queueTime = "2019-07-05T08:12:03.51941Z",
startTime = "2019-07-05T08:12:03.519649Z",
completeTime = <unset>,
eventChainId = 869472401,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
activationId = <unset>
}
[root@BSA-ESXI04:~]
We get quite a bit of information returned but for me the most useful part is the progress value which in this case shows the task is 93% complete. We can run the command multiple times through the day to see how the task is doing and when it has completed.
Hopefully this information proves useful to you and helps identify those ‘hidden’ tasks which the web management interface may no longer display but which are still active.
Excelente aporte…
I like the info but it would be useful to go into how to abort the task if it’s a runaway task or something.
Estava com o mesmo problema executei os passos abaixo e resolveu,
Do Console Local ou SSH:
Faça login no SSH ou no console local como root.2.
Execute esses comandos:/etc/init.d/hostd restart
/etc/init.d/vpxa restart
I would like this info too.
vim-cmd vimsvc/task_cancel
example: vim-cmd vimsvc/task_cancel haTask-122-vim.VirtualMachine.removeAllSnapshots-869472401
Thank you for this quick detail and thank you to Tracy for the cancel command. I had a stalled vStorage vMotion that was stopping a VM from being powered on after a migration. This allowed me to find and cancel the offending registration command that was hung up.
Glad you found the post and comments useful, really cool knowing it’s benefiting people
I realize this is an old thread but am facing this same situation with queued snapshots that wont let me complete them. Whe I do the command to list the status of the tasks (or try to delete them) they show a status of (already delete or not completely created????):
([root@win-37ogfpmoa67:~] vim-cmd vimsvc/task_cancel haTask-122-vim.VirtualMachine.removeAllSnapshots-14278
(vmodl.fault.ManagedObjectNotFound) {
faultCause = (vmodl.MethodFault) null,
faultMessage = ,
obj = ‘vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-14278’
msg = “Received SOAP response fault from [<<io_obj p:0x000000621f13ec48, h:5, , >, /sdk>]: cancel
The object ‘vim.Task:haTask-122-vim.VirtualMachine.removeAllSnapshots-14278’ has already been deleted or has not been completely created”
}
)
Is there another command to force these to go away or another way out of this state?
I’m guessing the snapshot files still show against the VM/in the datastore? Out of curiosity, is this a VM on classic VMFS datastores? If the snap is large or there is high IO, it can result in a very long running task that I’ve seen throw this message. This wouldn’t be an issue on vVOLs but VMFS datastores do enjoy being troublesome with snapshots.
I know some people have restarted the host management agents in situations like this, though that could be risky. Depending on the situation there is a chance of corruption but I have seen people regularly do this without issue. That doesn’t mean it can’t or won’t happen, but I have seen it play out OK.
I have also come across people suggesting to remove the VM from your vCenter inventory and then add it back in again. I haven’t experience with going down this route so I can’t comment on the efficacy or safety.