You are not logged in.

#1 2009-08-17 09:44:03

JohannesE
Member
Registered: 2009-08-17
Posts: 22

Unmounting root fs fails on shutdown

Hey all,

since a few days my root filesystem refuses to umount on shutdown claiming it were busy. On every bootup a recovery is done which bothers me. The failing umount suggests using lsof to find the process causing the problem. I switched to runlevel 1 and did so, but the output is not very helpful for me...

COMMAND     PID USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
init          1 root  cwd       DIR        8,5     4096          2 /
init          1 root  rtd       DIR        8,5     4096          2 /
init          1 root  txt       REG        8,5    31352      37632 /sbin/init
init          1 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
init          1 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
init          1 root   10u     FIFO       0,12      0t0       1195 /dev/initctl
kthreadd      2 root  cwd       DIR        8,5     4096          2 /
kthreadd      2 root  rtd       DIR        8,5     4096          2 /
kthreadd      2 root  txt   unknown                                /proc/2/exe
migration     3 root  cwd       DIR        8,5     4096          2 /
migration     3 root  rtd       DIR        8,5     4096          2 /
migration     3 root  txt   unknown                                /proc/3/exe
ksoftirqd     4 root  cwd       DIR        8,5     4096          2 /
ksoftirqd     4 root  rtd       DIR        8,5     4096          2 /
ksoftirqd     4 root  txt   unknown                                /proc/4/exe
watchdog/     5 root  cwd       DIR        8,5     4096          2 /
watchdog/     5 root  rtd       DIR        8,5     4096          2 /
watchdog/     5 root  txt   unknown                                /proc/5/exe
migration     6 root  cwd       DIR        8,5     4096          2 /
migration     6 root  rtd       DIR        8,5     4096          2 /
migration     6 root  txt   unknown                                /proc/6/exe
ksoftirqd     7 root  cwd       DIR        8,5     4096          2 /
ksoftirqd     7 root  rtd       DIR        8,5     4096          2 /
ksoftirqd     7 root  txt   unknown                                /proc/7/exe
watchdog/     8 root  cwd       DIR        8,5     4096          2 /
watchdog/     8 root  rtd       DIR        8,5     4096          2 /
watchdog/     8 root  txt   unknown                                /proc/8/exe
events/0      9 root  cwd       DIR        8,5     4096          2 /
events/0      9 root  rtd       DIR        8,5     4096          2 /
events/0      9 root  txt   unknown                                /proc/9/exe
events/1     10 root  cwd       DIR        8,5     4096          2 /
events/1     10 root  rtd       DIR        8,5     4096          2 /
events/1     10 root  txt   unknown                                /proc/10/exe
khelper      11 root  cwd       DIR        8,5     4096          2 /
khelper      11 root  rtd       DIR        8,5     4096          2 /
khelper      11 root  txt   unknown                                /proc/11/exe
async/mgr    12 root  cwd       DIR        8,5     4096          2 /
async/mgr    12 root  rtd       DIR        8,5     4096          2 /
async/mgr    12 root  txt   unknown                                /proc/12/exe
kblockd/0    13 root  cwd       DIR        8,5     4096          2 /
kblockd/0    13 root  rtd       DIR        8,5     4096          2 /
kblockd/0    13 root  txt   unknown                                /proc/13/exe
kblockd/1    14 root  cwd       DIR        8,5     4096          2 /
kblockd/1    14 root  rtd       DIR        8,5     4096          2 /
kblockd/1    14 root  txt   unknown                                /proc/14/exe
kacpid       15 root  cwd       DIR        8,5     4096          2 /
kacpid       15 root  rtd       DIR        8,5     4096          2 /
kacpid       15 root  txt   unknown                                /proc/15/exe
kacpi_not    16 root  cwd       DIR        8,5     4096          2 /
kacpi_not    16 root  rtd       DIR        8,5     4096          2 /
kacpi_not    16 root  txt   unknown                                /proc/16/exe
kseriod      17 root  cwd       DIR        8,5     4096          2 /
kseriod      17 root  rtd       DIR        8,5     4096          2 /
kseriod      17 root  txt   unknown                                /proc/17/exe
khungtask    18 root  cwd       DIR        8,5     4096          2 /
khungtask    18 root  rtd       DIR        8,5     4096          2 /
khungtask    18 root  txt   unknown                                /proc/18/exe
pdflush      19 root  cwd       DIR        8,5     4096          2 /
pdflush      19 root  rtd       DIR        8,5     4096          2 /
pdflush      19 root  txt   unknown                                /proc/19/exe
kswapd0      21 root  cwd       DIR        8,5     4096          2 /
kswapd0      21 root  rtd       DIR        8,5     4096          2 /
kswapd0      21 root  txt   unknown                                /proc/21/exe
aio/0        22 root  cwd       DIR        8,5     4096          2 /
aio/0        22 root  rtd       DIR        8,5     4096          2 /
aio/0        22 root  txt   unknown                                /proc/22/exe
aio/1        23 root  cwd       DIR        8,5     4096          2 /
aio/1        23 root  rtd       DIR        8,5     4096          2 /
aio/1        23 root  txt   unknown                                /proc/23/exe
crypto/0     24 root  cwd       DIR        8,5     4096          2 /
crypto/0     24 root  rtd       DIR        8,5     4096          2 /
crypto/0     24 root  txt   unknown                                /proc/24/exe
crypto/1     25 root  cwd       DIR        8,5     4096          2 /
crypto/1     25 root  rtd       DIR        8,5     4096          2 /
crypto/1     25 root  txt   unknown                                /proc/25/exe
ata/0       450 root  cwd       DIR        8,5     4096          2 /
ata/0       450 root  rtd       DIR        8,5     4096          2 /
ata/0       450 root  txt   unknown                                /proc/450/exe
ata/1       452 root  cwd       DIR        8,5     4096          2 /
ata/1       452 root  rtd       DIR        8,5     4096          2 /
ata/1       452 root  txt   unknown                                /proc/452/exe
ata_aux     453 root  cwd       DIR        8,5     4096          2 /
ata_aux     453 root  rtd       DIR        8,5     4096          2 /
ata_aux     453 root  txt   unknown                                /proc/453/exe
scsi_eh_0   457 root  cwd       DIR        8,5     4096          2 /
scsi_eh_0   457 root  rtd       DIR        8,5     4096          2 /
scsi_eh_0   457 root  txt   unknown                                /proc/457/exe
scsi_eh_1   473 root  cwd       DIR        8,5     4096          2 /
scsi_eh_1   473 root  rtd       DIR        8,5     4096          2 /
scsi_eh_1   473 root  txt   unknown                                /proc/473/exe
scsi_eh_2   483 root  cwd       DIR        8,5     4096          2 /
scsi_eh_2   483 root  rtd       DIR        8,5     4096          2 /
scsi_eh_2   483 root  txt   unknown                                /proc/483/exe
scsi_eh_3   484 root  cwd       DIR        8,5     4096          2 /
scsi_eh_3   484 root  rtd       DIR        8,5     4096          2 /
scsi_eh_3   484 root  txt   unknown                                /proc/484/exe
kjournald   564 root  cwd       DIR        8,5     4096          2 /
kjournald   564 root  rtd       DIR        8,5     4096          2 /
kjournald   564 root  txt   unknown                                /proc/564/exe
kondemand   900 root  cwd       DIR        8,5     4096          2 /
kondemand   900 root  rtd       DIR        8,5     4096          2 /
kondemand   900 root  txt   unknown                                /proc/900/exe
kondemand   905 root  cwd       DIR        8,5     4096          2 /
kondemand   905 root  rtd       DIR        8,5     4096          2 /
kondemand   905 root  txt   unknown                                /proc/905/exe
ksuspend_  1409 root  cwd       DIR        8,5     4096          2 /
ksuspend_  1409 root  rtd       DIR        8,5     4096          2 /
ksuspend_  1409 root  txt   unknown                                /proc/1409/exe
khubd      1469 root  cwd       DIR        8,5     4096          2 /
khubd      1469 root  rtd       DIR        8,5     4096          2 /
khubd      1469 root  txt   unknown                                /proc/1469/exe
hd-audio0  1914 root  cwd       DIR        8,5     4096          2 /
hd-audio0  1914 root  rtd       DIR        8,5     4096          2 /
hd-audio0  1914 root  txt   unknown                                /proc/1914/exe
kpsmoused  2006 root  cwd       DIR        8,5     4096          2 /
kpsmoused  2006 root  rtd       DIR        8,5     4096          2 /
kpsmoused  2006 root  txt   unknown                                /proc/2006/exe
iwl3945    2007 root  cwd       DIR        8,5     4096          2 /
iwl3945    2007 root  rtd       DIR        8,5     4096          2 /
iwl3945    2007 root  txt   unknown                                /proc/2007/exe
phy0       2017 root  cwd       DIR        8,5     4096          2 /
phy0       2017 root  rtd       DIR        8,5     4096          2 /
phy0       2017 root  txt   unknown                                /proc/2017/exe
kmmcd      2081 root  cwd       DIR        8,5     4096          2 /
kmmcd      2081 root  rtd       DIR        8,5     4096          2 /
kmmcd      2081 root  txt   unknown                                /proc/2081/exe
khpsbpkt   2087 root  cwd       DIR        8,5     4096          2 /
khpsbpkt   2087 root  rtd       DIR        8,5     4096          2 /
khpsbpkt   2087 root  txt   unknown                                /proc/2087/exe
knodemgrd  2248 root  cwd       DIR        8,5     4096          2 /
knodemgrd  2248 root  rtd       DIR        8,5     4096          2 /
knodemgrd  2248 root  txt   unknown                                /proc/2248/exe
usbhid_re  3320 root  cwd       DIR        8,5     4096          2 /
usbhid_re  3320 root  rtd       DIR        8,5     4096          2 /
usbhid_re  3320 root  txt   unknown                                /proc/3320/exe
bluetooth  3446 root  cwd       DIR        8,5     4096          2 /
bluetooth  3446 root  rtd       DIR        8,5     4096          2 /
bluetooth  3446 root  txt   unknown                                /proc/3446/exe
kjournald  3463 root  cwd       DIR        8,5     4096          2 /
kjournald  3463 root  rtd       DIR        8,5     4096          2 /
kjournald  3463 root  txt   unknown                                /proc/3463/exe
krfcommd   4048 root  cwd       DIR        8,5     4096          2 /
krfcommd   4048 root  rtd       DIR        8,5     4096          2 /
krfcommd   4048 root  txt   unknown                                /proc/4048/exe
pdflush    6018 root  cwd       DIR        8,5     4096          2 /
pdflush    6018 root  rtd       DIR        8,5     4096          2 /
pdflush    6018 root  txt   unknown                                /proc/6018/exe
udevd     10990 root  cwd       DIR        8,5     4096          2 /
udevd     10990 root  rtd       DIR        8,5     4096          2 /
udevd     10990 root  txt       REG        8,5    99948       7631 /sbin/udevd
udevd     10990 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
udevd     10990 root  mem       REG        8,5    50746       1769 /lib/libnss_files-2.10.1.so
udevd     10990 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
udevd     10990 root    0u      CHR        1,3      0t0       1185 /dev/null
udevd     10990 root    1u      CHR        1,3      0t0       1185 /dev/null
udevd     10990 root    2u      CHR        1,3      0t0       1185 /dev/null
udevd     10990 root    3r      DIR        0,9        0          1 inotify
udevd     10990 root    4u     unix 0xf1959000      0t0      56175 socket
udevd     10990 root    5u     sock        0,4      0t0      56176 can't identify protocol
syslog-ng 13248 root  cwd       DIR        8,5     4096          2 /
syslog-ng 13248 root  rtd       DIR        8,5     4096          2 /
syslog-ng 13248 root  txt       REG        8,5   235740       3439 /usr/sbin/syslog-ng
syslog-ng 13248 root  mem       REG        8,5    17914       1987 /lib/libattr.so.1.1.0
syslog-ng 13248 root  mem       REG        8,5    98082       1786 /lib/libnsl-2.10.1.so
syslog-ng 13248 root  mem       REG        8,5    82987     172784 /usr/lib/libz.so.1.2.3.3
syslog-ng 13248 root  mem       REG        8,5    13771       1744 /lib/libdl-2.10.1.so
syslog-ng 13248 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
syslog-ng 13248 root  mem       REG        8,5   211279       2024 /lib/libpcre.so.0.0.1
syslog-ng 13248 root  mem       REG        8,5    15405       1991 /lib/libcap.so.2.16
syslog-ng 13248 root  mem       REG        8,5    34551       3638 /usr/lib/libwrap.so.0.7.6
syslog-ng 13248 root  mem       REG        8,5  1561754      55876 /usr/lib/libcrypto.so.0.9.8
syslog-ng 13248 root  mem       REG        8,5   317643      64401 /usr/lib/libssl.so.0.9.8
syslog-ng 13248 root  mem       REG        8,5    16703      35966 /usr/lib/libevtlog.so.0.0.0
syslog-ng 13248 root  mem       REG        8,5   875900       2512 /usr/lib/libglib-2.0.so.0.2000.4
syslog-ng 13248 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
syslog-ng 13248 root    0r      CHR        1,3      0t0       1185 /dev/null
syslog-ng 13248 root    1w      CHR        1,3      0t0       1185 /dev/null
syslog-ng 13248 root    2w      CHR        1,3      0t0       1185 /dev/null
syslog-ng 13249 root  cwd       DIR        8,5     4096     474297 /var/run
syslog-ng 13249 root  rtd       DIR        8,5     4096          2 /
syslog-ng 13249 root  txt       REG        8,5   235740       3439 /usr/sbin/syslog-ng
syslog-ng 13249 root  mem       REG        8,5    17914       1987 /lib/libattr.so.1.1.0
syslog-ng 13249 root  mem       REG        8,5    98082       1786 /lib/libnsl-2.10.1.so
syslog-ng 13249 root  mem       REG        8,5    82987     172784 /usr/lib/libz.so.1.2.3.3
syslog-ng 13249 root  mem       REG        8,5    13771       1744 /lib/libdl-2.10.1.so
syslog-ng 13249 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
syslog-ng 13249 root  mem       REG        8,5   211279       2024 /lib/libpcre.so.0.0.1
syslog-ng 13249 root  mem       REG        8,5    15405       1991 /lib/libcap.so.2.16
syslog-ng 13249 root  mem       REG        8,5    34551       3638 /usr/lib/libwrap.so.0.7.6
syslog-ng 13249 root  mem       REG        8,5  1561754      55876 /usr/lib/libcrypto.so.0.9.8
syslog-ng 13249 root  mem       REG        8,5   317643      64401 /usr/lib/libssl.so.0.9.8
syslog-ng 13249 root  mem       REG        8,5    16703      35966 /usr/lib/libevtlog.so.0.0.0
syslog-ng 13249 root  mem       REG        8,5   875900       2512 /usr/lib/libglib-2.0.so.0.2000.4
syslog-ng 13249 root  mem       REG        8,5    50746       1769 /lib/libnss_files-2.10.1.so
syslog-ng 13249 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
syslog-ng 13249 root    0r      CHR        1,3      0t0       1185 /dev/null
syslog-ng 13249 root    1w      CHR        1,3      0t0       1185 /dev/null
syslog-ng 13249 root    2w      CHR        1,3      0t0       1185 /dev/null
syslog-ng 13249 root    3u     unix 0xf1958400      0t0      58926 /dev/log
syslog-ng 13249 root    4w     FIFO        0,6      0t0      58922 pipe
syslog-ng 13249 root    5u     unix 0xf1959800      0t0      58929 /var/run/syslog-ng.ctl
syslog-ng 13249 root    6r      REG        0,3        0 4026531978 /proc/kmsg
syslog-ng 13249 root    7w      REG        8,5    40938     475550 /var/log/syslog.log
syslog-ng 13249 root    8w      REG        8,5 17572696     475551 /var/log/everything.log
syslog-ng 13249 root    9w      REG        8,5 11246832     475552 /var/log/kernel.log
syslog-ng 13249 root   10w      REG        8,5 14397607     475553 /var/log/messages.log
syslog-ng 13249 root   12w      REG        8,5  5666055     475561 /var/log/daemon.log
init      13252 root  cwd       DIR        8,5     4096          2 /
init      13252 root  rtd       DIR        8,5     4096          2 /
init      13252 root  txt       REG        8,5    31352      37632 /sbin/init
init      13252 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
init      13252 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
init      13252 root    0u      CHR        5,1      0t0       1177 /dev/console
init      13252 root    1u      CHR        5,1      0t0       1177 /dev/console
init      13252 root    2u      CHR        5,1      0t0       1177 /dev/console
bash      13253 root  cwd       DIR        8,5     4096         22 /root
bash      13253 root  rtd       DIR        8,5     4096          2 /
bash      13253 root  txt       REG        8,5   607644      64366 /bin/bash
bash      13253 root  mem       REG        8,5  1821760     130893 /usr/lib/locale/locale-archive
bash      13253 root  mem       REG        8,5    50746       1769 /lib/libnss_files-2.10.1.so
bash      13253 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
bash      13253 root  mem       REG        8,5    13771       1744 /lib/libdl-2.10.1.so
bash      13253 root  mem       REG        8,5   345186       2004 /lib/libncursesw.so.5.7
bash      13253 root  mem       REG        8,5    33753      10766 /lib/libhistory.so.6.0
bash      13253 root  mem       REG        8,5   245785      10765 /lib/libreadline.so.6.0
bash      13253 root  mem       REG        8,5    11125      62090 /usr/lib/gconv/ISO8859-1.so
bash      13253 root  mem       REG        8,5    35764     302973 /usr/share/locale/de/LC_MESSAGES/bash.mo
bash      13253 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
bash      13253 root    0u      CHR        5,1      0t0       1177 /dev/console
bash      13253 root    1u      CHR        5,1      0t0       1177 /dev/console
bash      13253 root    2u      CHR        5,1      0t0       1177 /dev/console
bash      13253 root  255u      CHR        5,1      0t0       1177 /dev/console
lsof      13304 root  cwd       DIR        8,5     4096         22 /root
lsof      13304 root  rtd       DIR        8,5     4096          2 /
lsof      13304 root  txt       REG        8,5   122304         57 /usr/sbin/lsof
lsof      13304 root  mem       REG        8,5  1821760     130893 /usr/lib/locale/locale-archive
lsof      13304 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
lsof      13304 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
lsof      13304 root    0u      CHR        5,1      0t0       1177 /dev/console
lsof      13304 root    1w      REG        8,5        0         75 /root/lsof_out
lsof      13304 root    2u      CHR        5,1      0t0       1177 /dev/console
lsof      13304 root    3r      DIR        0,3        0          1 /proc
lsof      13304 root    4r      DIR        0,3        0      59163 /proc/13304/fd
lsof      13304 root    5w     FIFO        0,6      0t0      59168 pipe
lsof      13304 root    6r     FIFO        0,6      0t0      59169 pipe
lsof      13305 root  cwd       DIR        8,5     4096         22 /root
lsof      13305 root  rtd       DIR        8,5     4096          2 /
lsof      13305 root  txt       REG        8,5   122304         57 /usr/sbin/lsof
lsof      13305 root  mem       REG        8,5  1821760     130893 /usr/lib/locale/locale-archive
lsof      13305 root  mem       REG        8,5  1584604       1986 /lib/libc-2.10.1.so
lsof      13305 root  mem       REG        8,5   133581       1984 /lib/ld-2.10.1.so
lsof      13305 root    4r     FIFO        0,6      0t0      59168 pipe
lsof      13305 root    7w     FIFO        0,6      0t0      59169 pipe

Does anyone have an Idea what to do? Thanks!

Johannes

Offline

#2 2009-08-17 13:16:12

Profjim
Member
From: NYC
Registered: 2008-03-24
Posts: 658

Re: Unmounting root fs fails on shutdown

I noticed one thing, but that's not to say there's not other problems too: your syslog-ng is still running. At that point in shutdown, I think all your daemons including it should be shut down. I don't know why it's still running. Do you just launch it by putting "syslog-ng" in your DAEMONS array in /etc/rc.conf? It's possible that you're not really getting an accurate picture here of the state of your system at shutdown. I debug these issues by hacking /etc/rc.shutdown. Before the "Unmounting Filesystems" block you could do a

lsof / # I think this is the way to display only open files on the root filesystem, but you should read man lsof to be sure
ls /var/run/daemons
read -p "continue? " reply # pause here
...rest of /etc/rc.shutdown

Offline

#3 2009-10-16 16:59:52

imrehg
Member
From: London, UK
Registered: 2008-04-10
Posts: 108
Website

Re: Unmounting root fs fails on shutdown

I haven't checked it to this much detail yet, but I think I have the same issue for while too...
Almost (but not quite) every shutdown I have a "/ is busy" message. After this the file system has to be checked at every boot... That's a lot of time just sitting around while e2fsck runs on the whole drive....

Any suggestions? Is there a race condition somewhere in the shutdown script? I have another computer that I manage with Arch on it, and that has no such problem, so still checking the differences between them in software/hardware...

Offline

Board footer

Powered by FluxBB