[SOLVED] Failed unmount temporary directory on shutdown

mcloaked · 2017-07-11 17:42:16

On two of my laptops after recent systemd updates (current running is systemd 233.75-3 ) I have had a red Fail flag during the shutdown of the system, that had not occurred previously, saying that the unmounting of a temporary directory had failed. If I run sudo journalctl -b -1 after the next boot, and search for "Fail" I find in the output:

...
Jul 10 16:20:20 lenovo2 systemd[1]: Unmounting Temporary Directory...
Jul 10 16:20:20 lenovo2 systemd[1]: Unmounting /run/user/994...
Jul 10 16:20:20 lenovo2 systemd[1]: Unmounting /home...
Jul 10 16:20:20 lenovo2 systemd[1]: Unmounting /boot/efi...
Jul 10 16:20:20 lenovo2 systemd[1]: tmp.mount: Mount process exited, code=exited status=32
Jul 10 16:20:20 lenovo2 systemd[1]: Failed unmounting Temporary Directory.
Jul 10 16:20:20 lenovo2 systemd[1]: Stopped target Swap.
...

This happens on every shutdown, so it is consistent.

$ ls /tmp
akonadi-mike.QYDQ0o  checkup-db-mike  sddm-:0-DZZtLD  sddm-auth98525ba4-cf6c-40fb-bde3-0ac8a034738e  ssh-jn3oR5dPmtq2  xauth-1000-_0

So presumably it is /tmp that isn't properly unmounting - but I don't know how to proceed further to try to find out what the underlying cause is. I have googled for similar issues but haven't found anything current.

Last edited by mcloaked (2017-07-13 10:20:59)

tom.ty89 · 2017-07-11 20:27:09

Tried `systemctl stop tmp.mount` or even `umount /tmp` and see what happen?

mcloaked · 2017-07-12 07:20:29

tom.ty89 wrote:

Tried `systemctl stop tmp.mount` or even `umount /tmp` and see what happen?

I will try that this evening when I get back to one of the two machines (different hardware) - though I am presuming that since it only started happening recently that there is a buggy update package responsible.

Docbroke · 2017-07-12 10:18:10

Some process is using tmp for writing, and it is not responding to systemd. Have you installed any new programs recently. If so stop that before shutdown and see.

Trilby · 2017-07-12 10:30:19

FWIW, I just saw this last night on my work computer. It did not cause any problems - just the (almost unnoticable) red flag. I've not yet been able to investigate (I may this morning).

EDIT: it seems this shows up in the journal of my home computer too. I just checked earlier, and it seems it was only in the most recent shutdown on this system (4 days ago on this one). EDIT 2: this seems to have been a one off (or two-off, one on each system) error for me.

mcloaked · 2017-07-12 18:50:37

Docbroke wrote:

Some process is using tmp for writing, and it is not responding to systemd. Have you installed any new programs recently. If so stop that before shutdown and see.

No new programmes installed - just updates in recent times.

mcloaked · 2017-07-12 19:41:35

mcloaked wrote:

tom.ty89 wrote:
Tried `systemctl stop tmp.mount` or even `umount /tmp` and see what happen?
I will try that this evening when I get back to one of the two machines (different hardware) - though I am presuming that since it only started happening recently that there is a buggy update package responsible.

$ sudo systemctl stop tmp.mount
Job for tmp.mount failed.
See "systemctl  status tmp.mount" and "journalctl  -xe" for details.
$ systemctl  status tmp.mount
● tmp.mount - Temporary Directory
   Loaded: loaded (/usr/lib/systemd/system/tmp.mount; static; vendor preset: disabled)
   Active: active (mounted) (Result: exit-code) since Wed 2017-07-12 20:33:38 BST; 22s ago
    Where: /tmp
     What: tmpfs
     Docs: man:hier(7)
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 15162 ExecUnmount=/usr/bin/umount /tmp -c (code=exited, status=32)
  Process: 306 ExecMount=/usr/bin/mount tmpfs /tmp -t tmpfs -o mode=1777,strictatime,nosuid,nodev (code=exited, status=32)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/tmp.mount

Jul 12 20:32:17 lenovo1 systemd[1]: Mounting Temporary Directory...
Jul 12 20:32:17 lenovo1 systemd[1]: Mounted Temporary Directory.
Jul 12 20:33:38 lenovo1 systemd[1]: Unmounting Temporary Directory...
Jul 12 20:33:38 lenovo1 systemd[1]: tmp.mount: Mount process exited, code=exited status=32
Jul 12 20:33:38 lenovo1 systemd[1]: Failed unmounting Temporary Directory.

$ sudo journalctl  -xe
...
-- Unit tmp.mount has begun shutting down.
Jul 12 20:33:38 lenovo1 systemd[1]: tmp.mount: Mount process exited, code=exited status=32
Jul 12 20:33:38 lenovo1 systemd[1]: Failed unmounting Temporary Directory.
-- Subject: Unit tmp.mount has finished shutting down
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit tmp.mount has finished shutting down.
...

So I am not sure I am seeing anything useful here other than it failed to unmount?

I have filed an upstream bug report at https://github.com/systemd/systemd/issues/6350

Last edited by mcloaked (2017-07-12 20:24:57)

tom.ty89 · 2017-07-12 20:48:59

-x is mostly useless. -u tmp.mount (optionally with -b or -e) could be more helpful (at least it would not throw tons of garbage )

I am not sure if this is a systemd issue tbh. You should have at least tried umount and check /tmp with fuser before filing an upstream report...

mcloaked · 2017-07-12 21:39:24

OK I logged out from the plasma desktop and opened a VT to run commands as root only

# umount /tmp
umount: /tmp: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
# fuser -cu /tmp
/tmp:                  409(root)   957(mike)  4138(root)
# ps -a 409
  PID TTY      STAT   TIME COMMAND
  409 ?        Ssl    0:00 /usr/bin/sddm
 4605 pts/2    S      0:00 su -
 4606 pts/2    S+     0:00 -bash
 4612 pts/1    R+     0:00 ps -a 409
# ps -a 957
  PID TTY      STAT   TIME COMMAND
  957 ?        Ss     0:00 /usr/bin/ssh-agent
 4605 pts/2    S      0:00 su -
 4606 pts/2    S+     0:00 -bash
 4613 pts/1    R+     0:00 ps -a 957
# ps -a 4138
  PID TTY      STAT   TIME COMMAND
 4138 tty1     Ssl+   0:01 /usr/lib/xorg-server/Xorg -nolisten tcp -auth /var/run/sddm/{512cb5fb-8689-46b4-b2b0-76c
 4605 pts/2    S      0:00 su -
 4606 pts/2    S+     0:00 -bash
 4614 pts/1    R+     0:00 ps -a 4138

These processes would presumably normally be there on previous occasions when the shutdown process was completing without error. So I am not sure this gives any additional clues here?

Docbroke · 2017-07-13 01:09:52

Try to stop one of those process before doing shutdown. If you get the same error try with some other process next time. I will start with sddm or ssh-agent.

tom.ty89 · 2017-07-13 02:48:39

Yeah try these:
1. when you logout on plasma and open another VT, login as root instead of login as "mike" and `su -`
2. check with fuser to see if the ssh-agent process that grabs /tmp is gone
(2.1 you can try shutdown at this point as well to isolate the problem; if it happens, you may even want to compare if it makes a difference to shutdown in the tty and in the sddm login screen via its shutdown button)
3. `systemctl stop sddm`, fuser check if anything still grabs /tmp is gone
4. if not, shutdown

Btw, does this perhaps happen only on shutdown but not reboot (or vice versa)?

P.S. you don't really need -a for ps in this case...

Last edited by tom.ty89 (2017-07-13 03:08:34)

mcloaked · 2017-07-13 10:12:41

tom.ty89 wrote:

Yeah try these:
1. when you logout on plasma and open another VT, login as root instead of login as "mike" and `su -`
2. check with fuser to see if the ssh-agent process that grabs /tmp is gone
(2.1 you can try shutdown at this point as well to isolate the problem; if it happens, you may even want to compare if it makes a difference to shutdown in the tty and in the sddm login screen via its shutdown button)
3. `systemctl stop sddm`, fuser check if anything still grabs /tmp is gone
4. if not, shutdown
Btw, does this perhaps happen only on shutdown but not reboot (or vice versa)?
P.S. you don't really need -a for ps in this case...

Yes I will do a few more tests - though when I opened the VT I did log in as root and the one process belonging to mike was there despite not being logged in. To answer your question it happens for reboot as well as shutdown. But the fact remains that it never used to be a problem until a recent update (I update mostly daily). It certainly should not be necessary to stop sddm or X in order to shutdown or reboot without error! Stopping the ssh-agent could be the issue and I will test that - but I have made no changes to the way I setup the agent so perhaps that used to be closed before shutdown when it isn't now!

Edit: You were right - stopping the process ssh-agent allows a clean shutdown. Perhaps the most recent versions of systemd is now more fussy about such processes at the time it closes the system down before powering off. I will need to now look at my scripts for setting up ssh-agent and closing it down properly when I log out. In the meantime I will close this issue. Thank you for your help.

Last edited by mcloaked (2017-07-13 10:30:39)

Docbroke · 2017-07-13 12:31:01

you may tray adding "killall ssh-agent" in .bash_logout or /etc/bash.bash_logout

mcloaked · 2017-07-13 14:41:27

Docbroke wrote:

you may tray adding "killall ssh-agent" in .bash_logout or /etc/bash.bash_logout

Probably will plan on adding to a script file for the plasma shutdown rather than for bash since I usually want the agent running irrespective of opening and closing konsole windows during a plasma session, but thank you for the suggestion which is useful.

Docbroke · 2017-07-13 14:45:38

You don't logout when you close terminal running in x-server.

tom.ty89 · 2017-07-13 15:02:57

How is the ssh-agent started anyway? Doesn't sound right that it is not stopped when a user logout while it was run as the user.

Last edited by tom.ty89 (2017-07-13 15:03:24)

mcloaked · 2017-07-13 18:43:48

Turns out that handling ssh-agent isn't always straightforward! I run a script from a taskbar icon that checks to see if an agent is already running and then if not it starts a new one and asks for the passhrase - however starting a new konsole window usually gets a new agent PID so handling a shutdown needs some thought and careful coding to make it work (and to avoid duplicate agents and other problems!). However the scripts I use check for duplicate processes and kill any except the one originally started and share it. I will need to look at this carefully when I have a block of time.

Edit: I did a quick fiddle and found a simple solution that does seem to work:

$ cat ssh-agent-kill.sh
#!/bin/bash
#

# Place in .config/plasma-workspace/shutdown/ to terminate the agent at plasma shutdown

pkill ssh-agent

Last edited by mcloaked (2017-07-13 19:26:49)

Arch Linux

#1 2017-07-11 17:42:16

[SOLVED] Failed unmount temporary directory on shutdown

#2 2017-07-11 20:27:09

Re: [SOLVED] Failed unmount temporary directory on shutdown

#3 2017-07-12 07:20:29

Re: [SOLVED] Failed unmount temporary directory on shutdown

#4 2017-07-12 10:18:10

Re: [SOLVED] Failed unmount temporary directory on shutdown

#5 2017-07-12 10:30:19

Re: [SOLVED] Failed unmount temporary directory on shutdown

#6 2017-07-12 18:50:37

Re: [SOLVED] Failed unmount temporary directory on shutdown

#7 2017-07-12 19:41:35

Re: [SOLVED] Failed unmount temporary directory on shutdown

#8 2017-07-12 20:48:59

Re: [SOLVED] Failed unmount temporary directory on shutdown

#9 2017-07-12 21:39:24

Re: [SOLVED] Failed unmount temporary directory on shutdown

#10 2017-07-13 01:09:52

Re: [SOLVED] Failed unmount temporary directory on shutdown

#11 2017-07-13 02:48:39

Re: [SOLVED] Failed unmount temporary directory on shutdown

#12 2017-07-13 10:12:41

Re: [SOLVED] Failed unmount temporary directory on shutdown

#13 2017-07-13 12:31:01

Re: [SOLVED] Failed unmount temporary directory on shutdown

#14 2017-07-13 14:41:27

Re: [SOLVED] Failed unmount temporary directory on shutdown

#15 2017-07-13 14:45:38

Re: [SOLVED] Failed unmount temporary directory on shutdown

#16 2017-07-13 15:02:57

Re: [SOLVED] Failed unmount temporary directory on shutdown

#17 2017-07-13 18:43:48

Re: [SOLVED] Failed unmount temporary directory on shutdown

Board footer