[SOLVED] /dev mounted twice

emphire · 2011-02-23 18:33:16

I was just wondering if it's unusual to have /dev mounted twice. I don't recall seeing it happen before. Is anyone else getting this behavior?

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              180G  7.8G  163G   5% /
udev                  2.0G  260K  2.0G   1% /dev
none                  2.0G  260K  2.0G   1% /dev
shm                   2.0G     0  2.0G   0% /dev/shm
/dev/md0              124M   21M   97M  18% /boot
tmpfs                  40G  156K   40G   1% /tmp
tmpfs                 5.0M   40K  5.0M   1% /var/run
tmpfs                 2.0M     0  2.0M   0% /var/lock
/dev/sdc1             1.8T  962G  779G  56% /mnt/tank

$ cat /etc/mtab
/dev/md2 / ext4 rw,noatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=506600,mode=755 0 0
none /dev tmpfs rw,relatime,mode=755 0 0
devpts /dev/pts devpts rw 0 0
shm /dev/shm tmpfs rw,nosuid,nodev 0 0
/dev/md0 /boot ext4 rw,noatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,mode=1777,size=40G,nr_inodes=10k 0 0
tmpfs /var/run tmpfs rw,noexec,nosuid,nodev,mode=1777,size=5M 0 0
tmpfs /var/lock tmpfs rw,noexec,nosuid,nodev,mode=1777,size=2M 0 0
/dev/sdc1 /mnt/tank ext4 rw,noatime 0 0

Last edited by emphire (2011-03-03 00:59:38)

falconindy · 2011-02-23 18:56:37

And what's in your /etc/fstab? In the end it's harmless, but it's not the norm.

emphire · 2011-02-23 19:21:48

falconindy wrote:

And what's in your /etc/fstab? In the end it's harmless, but it's not the norm.

Here's my /etc/fstab:

devpts                 /dev/pts      devpts    defaults            0 0
shm                    /dev/shm      tmpfs     nodev,nosuid        0 0

/dev/md1 swap  swap defaults  0 0
/dev/md2 /  ext4 defaults,noatime 0 0
/dev/md0 /boot  ext4 defaults,noatime  0 1

tmpfs /tmp  tmpfs defaults,nodev,nosuid,mode=1777,size=40G,nr_inodes=10k  0 0
tmpfs /var/run  tmpfs defaults,nodev,nosuid,noexec,mode=1777,size=5M  0 0
tmpfs /var/lock tmpfs defaults,nodev,nosuid,noexec,mode=1777,size=2M  0 0

LABEL=tank /mnt/tank ext4 defaults,noatime,noauto  0 0
UUID=59ebc849-1d89-42f6-810a-6f6fb2f9d241 /mnt/backup ext4 defaults,noatime,noauto  0 0

falconindy · 2011-02-23 20:06:38

Strange... not immediately clear where the tmpfs /dev is being mounted. the devtmpfs mount on /dev is the "correct" mount which is mounted by either initcpio or /etc/rc.sysinit (if you don't boot with an initcpio). Have you modified any of your initscripts? Put anything in rc.local? Using an initrd generator other than Arch's?

emphire · 2011-02-23 21:01:50

It's a pretty fresh install (about three days old) and I haven't changed much other than adding a few packages. I am using initcpio.

Some of the less common things I'm doing are:

using extlinux as my boot loader
using a raid1 md software raid for my boot, root, and swap partitions
all my drives are partitioned with gpt/gdisk
/boot is ext4

Here is my /etc/rc.local:

# Prevent green drives from parking head every 8 seconds
for DISK in `fdisk -l |grep [1,2]000.4 | cut -c13-13`
  do echo hdparm -S 242 /dev/sd$DISK
done

That's all I've modified for init scripts other than rc.conf. I don't think that would cause /dev to mount twice though. Perhaps it's related to the md raid or a combination of md raid, gpt and extlinux/syslinux?

falconindy · 2011-02-24 00:50:22

Well, the early boot process in a nutshell is this:

- the initcpio's init is started, which promptly mounts /proc, /sys and /dev. if devtmpfs is available, its used for /dev -- otherwise, its tmpfs. That seems like a possibility, but the mount options do not seem to line up.
- udevd is started and modules are loaded, followed by various initcpio hooks in an effort to mount your real root. let's assume nothing goes bad and root is assembled/decrypted/scanned/hornswaggled/mounted.
- udevd is killed, the previously mounted /{proc,sys,dev} are moved to the new root, and switch_root is called with your real /sbin/init and life goes on with /etc/rc.sysinit

there's unfortunately very little in the way of debugging that gets logged for mounted pseudo-filesystems. It's possible to break into an interactive shell during the initcpio (add break=y to your kernel cmdline). It may be worthwhile to find out if indeed /dev is being mounted there as a tmpfs and not properly moved off to the new /. You might also get some useful info out of investigating your environment in runlevel 1 -- add a '1' or 'single' to your kernel cmdline.

Last edited by falconindy (2011-02-24 00:51:22)

emphire · 2011-02-25 04:04:42

Thanks falconindy.

I tried booting with break=y in single user mode.

When it stops in the initcpio, here is what I find:

 $ df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                  10.0M  120.0K  9.9M   1% /dev

 $ cat /proc/mounts
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=506600,mode=755 0 0

 $exit

Now as single user mode starts, I notice a couple of error messages that I hadn't noticed before. I rebooted normally in (without the extra kernel parameters) and confirmed that I get the same errors then (they just fly by my screen too fast to notice).

 > Arch Linux
 
 > http://www.archlinux.org
 > Copyright 2002-2007 Judd Vinet
 > Copyright 2002-2007 Aaron Griffin

   ----------------------------------------------
mount: proc already mounted
mount: none already mounted or /sys busy
mount: according to mtab, sys is already mounted on /sys
:: Loading Modules           [DONE]
[...]
:: Activating RAID arrays  [FAIL]
[...]

Now in single user mode, two /dev's:

 $ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              180G  7.8G  163G   5% /
udev                  2.0G     0  2.0G   0% /dev
none                  2.0G     0  2.0G   0% /dev
shm                   2.0G     0  2.0G   0% /dev/shm
/dev/md0              124M   21M   97M  18% /boot
tmpfs                  40G     0   40G   0% /tmp
tmpfs                 5.0M   4.0K  5.0M   1% /var/run
tmpfs                 2.0M     0  2.0M   0% /var/lock

Strange.

falconindy · 2011-02-25 14:59:56

Hurray for bisecting the boot process. This is strange, and I'm strangely intrigued by this problem.

So, the break in the initcpio occurs after hooks have been run, but before the root has been mounted. I think we can rule out the initcpio has being the culprit here which basically leaves us with rc.sysinit. For some reason, it appears that the /bin/mountpoint calls are failing to properly return success and udev gets remounted.

Could you try the following from runlevel 1?

/bin/mountpoint -q /dev; echo $?
/bin/findmnt /dev >/dev/null; echo $?

Also, could you temporarily comment out lines 20-32 of /etc/rc.sysinit and boot normally? That should make the errors go away and perhaps pinpoint the problem if my above experiment doesn't yield any results.

emphire · 2011-02-26 02:29:55

The plot thickens...

I tried booting into single user mode and now my keyboard doesn't work in it. It works when I reset my machine and boot into multiuser mode. I could swear it worked when booting into single user mode before. Odd. Anyway, I dropped to runlevel 1 with telinit 1 from multiuser mode:

 $ telinit 1
[...]
INIT: Going single user
Give root password for maintenance
(or type Control-D to continue):

 $ /bin/mountpoint -q /dev; echo $?
0
 $ /bin/findmnt /dev >/dev/null; echo $?
0

When I try booting with lines 20-32 of /etc/rc.conf commented out, I still get the messages:

mount: proc already mounted
mount: none already mounted or /sys busy
mount: according to mtab, sys is already mounted on /sys
:: Loading Modules           [DONE]
[...]
:: Activating RAID arrays  [FAIL]
[...]

falconindy · 2011-02-26 02:36:52

not rc.conf! rc.sysinit!

usb keyboard? You may want to add the 'usbinput' hook to your /etc/mkinitcpio.conf. There's excitement abound here...

emphire · 2011-02-26 18:55:23

falconindy wrote:

not rc.conf! rc.sysinit!

Oops... I meant to say I had lines 20-32 commented out in rc.sysinit.

I do have usbinput in my mkinitcpio.conf (ran mkinicpio again to to make sure) but I still get no keyboard when booting into single user mode. I tried again with both break=y and single when I exit from the mkinitcpio break and continue into runlevel 1, my keyboard does work. Odd that it doesn't when I boot directly into runlevel 1 without doing the "break=y".

Anyway, with those lines commented out, booting into single user mode, I still get the errors. Here's what I get when I run those commands after booting into runlevel 1 (with lines 20-32 of rc.sysinit commented out):

mount: proc already mounted
mount: none already mounted or /sys busy
mount: according to mtab, sys is already mounted on /sys
:: Loading Modules           [DONE]
[...]
:: Activating RAID arrays  [FAIL]
[...]
Give root password for maintenance
(or type Control-D to continue):

 $ /bin/mountpoint -q /dev; echo $?
0
 $ /bin/findmnt /dev >/dev/null; echo $?
0

It seems it must be doing the mount somewhere else. Perhaps the md stuff is doing it?

falconindy · 2011-02-26 23:52:36

Hmm... I'm running out of ideas without a hands-on view of your system. There's not a whole lot that happens between the breakpoint in initcpio (known good) and the rc.sysinit invocation (known bad). I'm mildly suspicious of the mdadm setup, but I have no hard evidence to back it up.

emphire · 2011-03-03 00:59:03

Thanks for your suggestions falconindy. After the update in initscripts. I noticed that when qinit installed as part of core, it changed my inittab. Now that I have removed the qinit stuff, I am booting without the errors and only one dev entry.

Arch Linux

#1 2011-02-23 18:33:16

[SOLVED] /dev mounted twice

#2 2011-02-23 18:56:37

Re: [SOLVED] /dev mounted twice

#3 2011-02-23 19:21:48

Re: [SOLVED] /dev mounted twice

#4 2011-02-23 20:06:38

Re: [SOLVED] /dev mounted twice

#5 2011-02-23 21:01:50

Re: [SOLVED] /dev mounted twice

#6 2011-02-24 00:50:22

Re: [SOLVED] /dev mounted twice

#7 2011-02-25 04:04:42

Re: [SOLVED] /dev mounted twice

#8 2011-02-25 14:59:56

Re: [SOLVED] /dev mounted twice

#9 2011-02-26 02:29:55

Re: [SOLVED] /dev mounted twice

#10 2011-02-26 02:36:52

Re: [SOLVED] /dev mounted twice

#11 2011-02-26 18:55:23

Re: [SOLVED] /dev mounted twice

#12 2011-02-26 23:52:36

Re: [SOLVED] /dev mounted twice

#13 2011-03-03 00:59:03

Re: [SOLVED] /dev mounted twice

Board footer