vmtoolsd not stopping

tschaboo · 2012-11-03 06:33:51

Hello,

I'm playing around with Arch Linux in a VMware Virtual Machine, preparing to switch from Ubuntu which I was using the last 7 years. So far everything went pretty well, but now I have a problem which I can't solve by myself.

Since the last pacman -Syu vmtoolsd refuses to stop, when shutting down. After 90 seconds, systemd kills it and the shutdown continues.

...
Nov 03 07:04:22 vm-arch systemd[1]: Stopped Permit User Sessions.
Nov 03 07:04:22 vm-arch systemd[1]: Stopped dhcpcd on eth0.
Nov 03 07:04:22 vm-arch systemd[1]: Unit dhcpcd@eth0.service entered failed state
Nov 03 07:04:22 vm-arch systemd[1]: Stopping Remote File Systems.
Nov 03 07:04:22 vm-arch systemd[1]: Stopped target Remote File Systems.
Nov 03 07:05:52 vm-arch systemd[1]: vmtoolsd.service stopping timed out. Killing.
Nov 03 07:05:52 vm-arch systemd[1]: vmtoolsd.service: main process exited, code=killed, status=9/KILL
Nov 03 07:05:52 vm-arch systemd[1]: Stopped Open Virtual Machine Tools (VMware Tools).
Nov 03 07:05:52 vm-arch systemd[1]: Unit vmtoolsd.service entered failed state
Nov 03 07:05:52 vm-arch systemd[1]: Stopping Basic System.
Nov 03 07:05:52 vm-arch systemd[1]: Stopped target Basic System.
...

Does anyone else have this problem? How can I find out more about what's going wrong?

tschaboo · 2012-11-03 06:37:15

PS: After the upgrade i did

cat /proc/version > /etc/arch-release

so that's not the problem.

hungerfish · 2012-11-06 13:17:06

Hi,

I'm having the exact same problem. I was following this and found your post by googleling the recorded error message.

So far I haven't found solution, but I was wondering if by any chance you may have other problems I'm having( with Arch inside vmware ).
VMWare fusion(host is a mac) complains about the startup script not completing properly, only on first boot though. Subsequent reboots don't get this message.
I'm guessing this may be related.

I also cannot get shared folders to work. The hgfs module loads, but when I try and access the folder ( ls /mnt/hgfs/Shared), ls gets killed, and dmesg gives me a kernel OOPS. This has been going on ever since I've upgraded to vmware v5 (fusion or workstation), and for quite a few open-vm-tools\open-vm-module versions.
This also applies to multiple hosts, my mac host, but also a windows host running either player or workstation.

Last edited by hungerfish (2012-11-06 13:18:59)

p0x8 · 2012-11-08 13:16:14

I can confirm that vmtoolsd keeps running after being terminated, responding only to the kill (9) signal. I have no other VMWare linux guests so can't check if it is an Arch or upstream issue.

hungerfish wrote:

VMWare fusion(host is a mac) complains about the startup script not completing properly, only on first boot though.

Also had this issue, but at every boot, if I remember correctly. Solved it by removing a needless script that is installed by the open-vm-tools package:

# rm /etc/vmware-tools/scripts/vmware/network.orig

This should probably be reported as a bug in the package, as this file seems to be the original of the actual network script.

As for the shared folders issue, it's a feature I don't use and never really tested. I don't even have the kernel modules installed.

hungerfish · 2012-11-08 14:09:21

I have no other VMWare linux guests so can't check if it is an Arch or upstream issue.

This is not upstream, ubuntu, suse (both supported by vmware) work just fine without these issues. (ubuntu at least with also using open-vm-tools)

I will check later, if by removing said file the error message goes away, but if it does it means its not causing the hang during reboots.
EDIT:
Well I deleted the file you mentioned, but it didn't stop the error message from appearing after boot.
Just to make sure, I'm talking about this message.

@p0x8
would you mind maybe temporarily enabling shared folder, and just test with the stock arch packages?
I've been thinking of opening a bug report(because the problem applies to two separate hosts), but wouldn't mind additional feedback.
Maybe I am configuring something wrong...

Last edited by hungerfish (2012-11-08 16:59:06)

haohaolee · 2012-12-12 14:06:21

The same problem here, any progress?

alexoz · 2012-12-14 18:55:40

@OP: I have the same issue. It looks like vmtoolsd does not exit when a SIGTERM is received so systemd waits for 90 seconds before sending a SIGKILL. I solved this by editing /usr/lib/systemd/system/vmtoolsd.service and adding a KillSignal=SIGKILL line in the [Service] section. This way systemd sends a SIGKILL instead of a SIGTERM.

Another way to overcome this issue would be to set the timeout for this service to just a few seconds instead of the default 90, by adding e.g. TimeoutStopSec=2 (2 seconds) in the [Service] section.

p0x8 · 2012-12-15 16:17:44

alexoz wrote:

@OP: I have the same issue. It looks like vmtoolsd does not exit when a SIGTERM is received so systemd waits for 90 seconds before sending a SIGKILL. I solved this by editing /usr/lib/systemd/system/vmtoolsd.service and adding a KillSignal=SIGKILL line in the [Service] section. This way systemd sends a SIGKILL instead of a SIGTERM.
Another way to overcome this issue would be to set the timeout for this service to just a few seconds instead of the default 90, by adding e.g. TimeoutStopSec=2 (2 seconds) in the [Service] section.

Thanks for the tip. I really should take the time to learn systemd service syntax.

I did an strace on vmtoolsd, and it seems to get stuck waiting for some lock:

--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=2734, si_uid=1000} ---
write(13, "\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\256\n\0\0\350\3\0\0\200\373\2\1\0\0\0\0"..., 128) = 128
rt_sigreturn()                          = -1 EINTR (Interrupted system call)
poll([{fd=3, events=POLLIN}, {fd=12, events=POLLIN}, {fd=12, events=POLLIN}, {fd=12, events=POLLIN}, {fd=12, events=POLLIN}, {fd=12, events=POLLIN}], 6, 732) = 5 ([{fd=12, revents=POLLIN}, {fd=12, revents=POLLIN}, {fd=12, revents=POLLIN}, {fd=12, revents=POLLIN}, {fd=12, revents=POLLIN}])
read(12, "\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\256\n\0\0\350\3\0\0\200\373\2\1\0\0\0\0"..., 128) = 128
write(3, "\1\0\0\0\0\0\0\0", 8)         = 8
write(3, "\1\0\0\0\0\0\0\0", 8)         = 8
write(3, "\1\0\0\0\0\0\0\0", 8)         = 8
write(3, "\1\0\0\0\0\0\0\0", 8)         = 8
write(3, "\1\0\0\0\0\0\0\0", 8)         = 8
futex(0x100fe00, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x100fe00, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
+++ killed by SIGKILL +++
Killed

Someone reported it upstream but the developers didn't bother to check for the bug and just dismissed it with "we don't support systemd". And it doesn't bother me enough to try and fix it myself.

egates · 2013-01-11 02:41:21

hungerfish wrote:

Well I deleted the file you mentioned, but it didn't stop the error message from appearing after boot.
Just to make sure, I'm talking about this message.

If you're still having this problem, I deleted both /etc/vmware-tools/scripts/vmware/network{,.orig} leaving that directory empty. In short the network scripts were trying to use rc.d scripts and ifconfig, and failing because of it. Removing the scripts (or just commenting out the call to main at the bottom) stops them from running and fixes the problem.

nponeccop · 2013-01-22 07:56:04

I reported the problem upstream in another way as it has nothing to do with systemd integration but with executable failing to respond to signals.

Note that the bug is not new. Before systemd it worked because rc.d stop script didn't actually checked the exit code of kill:

stop)
    stat_busy "Stopping Open Virtual Machine Tools"
    [ ! -z "$PID" ]  && kill $PID &>/dev/null
#    if [ $? -gt 0 ]; then
#      stat_fail
#      exit 1
#    fi

hawaiicharles · 2013-01-23 18:12:36

Posting to report that I have the same issue, in a fresh installation (as of 2013-01-22). If I manually stop the vmtoolsd service by typing

systemctl stop vmtoolsd

it hangs for 60 seconds before finally stopping the service and returning me to a prompt. After that I can run "poweroff" and shutdown is almost instantaneous.

Last edited by hawaiicharles (2013-01-23 18:13:03)

ssl6 · 2013-02-23 16:19:48

I'm having the same problem as well. I'm not sure if it's related to my other issue or not. but if I try to run this archlinux vm on my esxi 5.1 server, i'm not sure if it causes the entire server to lock up, or if it just kills any network connectivity on the thing, but it's a problem

nponeccop · 2013-02-23 21:01:02

I have a rather idiotic setup of ESXi running in VmWare Workstration on Win7. Sometimes whole win7 locks up with exhausted non-paged kernel pool in event log. Not sure if it's related.

Last edited by nponeccop (2013-02-23 21:01:16)

PhCl · 2013-02-24 17:34:24

I have the same problem (BTW: also on VMWare running on Windows 7).

I solved it by reducing the time out in /usr/lib/systemd/system/vmtoolsd.service to 3 seconds:

[Service]
ExecStart=/usr/bin/vmtoolsd
TimeoutStopSec=3

It is only a workaround, but at least the shutdown is fast again.

nponeccop · 2013-02-25 10:43:41

The workaround is already mentioned in #7

BertiBoeller · 2013-03-14 15:14:23

Hello,

can anybody tell me what the vmtoolsd actually does?

When I start the vmtoolsd as systemd service the process shows as:

$ ps aux | grep vmt
root      1130  1.5  0.1  92748  4408 ?        Ss   16:12   0:00 /usr/bin/vmtoolsd

When I start vmware-user-suid-wrapper which in turn starts vmtoolsd the service is started like this:

$ ps aux | grep vmt                                                                                                                                         
gumstix    611  0.2  0.4 258724 18228 ?        S    17:05   0:01 /usr/bin/vmtoolsd -n vmusr

Does it make sense to start vmtoolsd like the service file does?

Arch Linux

#1 2012-11-03 06:33:51

vmtoolsd not stopping

#2 2012-11-03 06:37:15

Re: vmtoolsd not stopping

#3 2012-11-06 13:17:06

Re: vmtoolsd not stopping

#4 2012-11-08 13:16:14

Re: vmtoolsd not stopping

#5 2012-11-08 14:09:21

Re: vmtoolsd not stopping

#6 2012-12-12 14:06:21

Re: vmtoolsd not stopping

#7 2012-12-14 18:55:40

Re: vmtoolsd not stopping

#8 2012-12-15 16:17:44

Re: vmtoolsd not stopping

#9 2013-01-11 02:41:21

Re: vmtoolsd not stopping

#10 2013-01-22 07:56:04

Re: vmtoolsd not stopping

#11 2013-01-23 18:12:36

Re: vmtoolsd not stopping

#12 2013-02-23 16:19:48

Re: vmtoolsd not stopping

#13 2013-02-23 21:01:02

Re: vmtoolsd not stopping

#14 2013-02-24 17:34:24

Re: vmtoolsd not stopping

#15 2013-02-25 10:43:41

Re: vmtoolsd not stopping

#16 2013-03-14 15:14:23

Re: vmtoolsd not stopping

Board footer