udev breaking the booting process

silk · 2012-01-21 10:25:39

Hello,

I have a very frustrating problem.
I did install Archlinux... baaaad things happened.
I can't actually boot... eveything is installed fine... but

When I boot the filesystem check fails, saying /dev/sda6 is already mounted.
Says that

/dev/sda6 is mounted   [fail]

**********************FILESYSTEM CHECK FAILED ******************
*
* Please repair manually and reboot. Note that the root file system is currently 
* mounted read-only. To remount it read-write type: mount -n -o remount,rw /
* When you exit the maintenance shell the system will reboot automatically
*
***************************************************************

I just won't post my fstab since I can't copy anything, but I've checked it numerous times, and it's not the source of the problem.
I can tell how it's parttioned tho...

sda1 - /boot
sda2 - swap
sda5 - /
sda6 - /usr
sda7 - /usr/local
sda8 - /var
sda9 - /var/log
sda10 - /var/spool
sda11 - /tmp
sda12 - /home

I've tried doing a 'lsof' on maintenance console.
I found out libkmod.so was loaded from /usr by UDEVD... Why the hell UDEVD is loaded there, before fscks?!

Oh... by the way. It's a 100% fresh install.

Thanks,
silk

falconindy · 2012-01-21 14:38:55

Crossposting here from the bug report for the google archives:

http://mailman.archlinux.org/pipermail/ … 24124.html
https://wiki.archlinux.org/index.php/Mk … _partition

nomorewindows · 2012-01-21 21:42:09

Here's what I did earlier in another post: (You can copy the needed files to your root partition until it is corrected using a recovery CD or another installation) Then when it finally mounts the /usr partition it will cover up the root partition.

/sbin/modprobe is a soft-link to /usr/bin/kmod which isn't mounted yet, despite the early /usr mount option. When it comes to mounting /usr at that time, I don't think mine is.
Have mine fixed, just put the /usr/bin/kmod binary "undermounted" on my / partition (temporarily) along with /usr/lib/libkmod.so.1 and /usr/lib/libz.so.1 (copying the actual files and then linking to them). Currently, the real files are /usr/lib/libkmod.so.1.3.0 and /usr/lib/libz.so.1.2.5

Last edited by nomorewindows (2012-01-21 21:45:30)

falconindy · 2012-01-21 21:56:21

What an awful idea.

nomorewindows · 2012-01-21 22:05:05

falconindy wrote:

What an awful idea.

Or you could add it to the init ram fs via /etc/rc.conf

falconindy · 2012-01-21 23:45:16

nomorewindows wrote:

falconindy wrote:
What an awful idea.
Or you could add it to the init ram fs via /etc/rc.conf

That doesn't even make sense.

Last edited by falconindy (2012-01-21 23:45:31)

nomorewindows · 2012-01-22 00:08:33

falconindy wrote:

nomorewindows wrote:
falconindy wrote:
What an awful idea.
Or you could add it to the init ram fs via /etc/rc.conf
That doesn't even make sense.

Well, why wouldn't it? lsmod/insmod/rmmod/modprobe and modules related are in the init ram fs image.

Anyhow, I'm not sure why they are soft linking the kmod binary to insmod/lsmod/rmmod/modprobe in /usr/bin when I looked back at an installation that I haven't upgraded to the newest yet and insmod/lsmod/rmmod/modprobe are their own binaries. The only thing I can hope is that they put the kmod binary in /sbin and point to those binaries their in later releases. My guess is that the pointed linkage is in the init ram fs image itself, pointing to something that will vanish at boot time.

If you add the modules to /etc/rc.conf and recompile the kernel with mkinitcpio, it should work (if you can get to it).

Last edited by nomorewindows (2012-01-22 00:31:11)

falconindy · 2012-01-22 00:42:07

/sbin/modprobe is a soft-link to /usr/bin/kmod which isn't mounted yet, despite the early /usr mount option

In early userspace with mkinitcpio 0.8.2, this is absolutely false. /sbin/modprobe is a real file (it's actually /usr/bin/kmod installed as /sbin/modprobe). Other tools -- insmod, rmmod, modinfo, lsmod, are provided as symlinks to busybox. These are not taken from module-init-tools or kmod (yet). It's completely irrelevant that /usr isn't mounted, because the initramfs is (and must be) a self-contained, self-sufficient environment.

Well, why wouldn't it? lsmod/insmod/rmmod/modprobe and modules related are in the init ram fs image.

Yes, they already are. You're referring to /etc/mkinitcpio.conf, and the real question is -- why do you want to re-add them when they already exist?

The only thing I can hope is that they put the kmod binary in /sbin and point to those binaries their in later releases.

I'm currently the developer of mkinitcpio, and I'm the maintainer of kmod. Nothing will be changing dramatically, because nothing here is broken.

My guess is that the pointed linkage is in the init ram fs image itself, pointing to something that will vanish at boot time.

Again, self-sufficient environment. What happens in early userspace, stays in early userspace.

This all sounds to me like you haven't regenerated your initramfs since upgrading to mkinitcpio 0.8.2. Perhaps you should do that.

nomorewindows · 2012-01-22 00:55:27

This all sounds to me like you haven't regenerated your initramfs since upgrading to mkinitcpio 0.8.2. Perhaps you should do that.

For some reason I don't remember seeing mkinitcpio today. I just recompiled the lts kernel a couple days prior. I remember kmod replacing module-init-tools, which I didn't figure I needed to do anything with it. If I had seen mkinitcpio I would've possibly suspected something.

falconindy · 2012-01-22 01:06:34

If you have kmod, you have mkinitcpio 0.8.2. Humor me and do something rational like check /var/log/pacman.log or pacman -Qi mkinitcpio and look at the Install Date.

nomorewindows · 2012-01-22 01:43:55

falconindy wrote:

If you have kmod, you have mkinitcpio 0.8.2. Humor me and do something rational like check /var/log/pacman.log or pacman -Qi mkinitcpio and look at the Install Date.

Ok so now I'm having the same problem that this person is.

falconindy · 2012-01-22 01:45:19

Did you bother to read the links I responded to him, with? Your answer is there.

byte · 2012-01-22 04:34:11

Exactly, the answer is there.

I had a similar experience: https://bugs.archlinux.org/task/27862#comment87739
After upgrading to a newer mkinitcpio and adding the shutdown and fsck hooks[^] everything booted (and shut down) fine.
:-)

^ later I just added /sbin/fsck{.jfs}, for size reasons (no autodetect), diff here: http://paste.pocoo.org/show/538561/

bnolsen · 2012-01-23 18:00:11

nomorewindows wrote:

Here's what I did earlier in another post: (You can copy the needed files to your root partition until it is corrected using a recovery CD or another installation) Then when it finally mounts the /usr partition it will cover up the root partition.
/sbin/modprobe is a soft-link to /usr/bin/kmod which isn't mounted yet, despite the early /usr mount option. When it comes to mounting /usr at that time, I don't think mine is.
Have mine fixed, just put the /usr/bin/kmod binary "undermounted" on my / partition (temporarily) along with /usr/lib/libkmod.so.1 and /usr/lib/libz.so.1 (copying the actual files and then linking to them). Currently, the real files are /usr/lib/libkmod.so.1.3.0 and /usr/lib/libz.so.1.2.5

A better option is to use ABS to grab core/kmod

edit PKGBUILD and add --prefix=/ to the configure args, remove "usr" from the soft links.

Putting something as basic as module loading anywhere other than '/bin' is pure lunacy. I'm sorry but I consider kernel module loading a touch more critical for system bootstrapping than say libreoffice, kde or firefox (all of which *belong* somwhere other than '/'.

I've always built my systems to be minimally bootable (and repairable) from '/' only, then use that to bootstrap all the other partitions like /usr /opt, etc.

I had the unpleasant experience that mkinitcpio.conf somehow got reset (did kernel 3.2.1 cause this?) so that my /usr wasn't mountable and because kmod only resided on '/usr' i couldn't load the module necessary to mount '/usr' to start bootstrap.

and just more ranting, i do't give a rip about what freedesktop says. I don't care about any 'desktop' when booting, i'm more than happy sometimes working in a console.

Last edited by bnolsen (2012-01-23 18:05:10)

dejari · 2012-01-25 15:19:16

I have a related problem.

kmod 4-1
mkinitcpio 0.8.2-1
HOOKS="base udev ide scsi sata usbinput encrypt filesystems shutdown fsck"

I've used buildpae from AUR to build a PAE-enabled kernel.

Bootup works fine using the stock kernel, but with PAE I get errors about udevadm and udevd missing libkmod (and libz, IIRC) before there's a chance to mount my encrypted root device.

Both kernels had their initramfs rebuilt manually by me after booting with PAE didn't work, but this didn't change anything.

Any idea where to look in order to solve this? I'd really like to have that additional RAM back again.

dejari · 2012-01-26 13:01:19

Fixed. An old compatibility symlink /usr/X11R6 -> . made me end up with zero-truncated versions of the libs in the initramfs image's /usr/lib directory. I'm not sure why it only affected the PAE kernel in the beginning.

leberyo · 2012-01-26 19:49:03

I've got the same problem. I tried to modify the mkinitcpio.conf file but it said that the filesystem is read only.

nomorewindows · 2012-01-26 21:55:48

leberyo wrote:

I've got the same problem. I tried to modify the mkinitcpio.conf file but it said that the filesystem is read only.

You will probably need a live CD to get into it to edit the file, and still a way to schroot to actually create the kernel when you do that.

leberyo · 2012-01-27 00:57:19

I used a live cd and modified mkinitcpio.conf

I'm a noob, could you tell me how to "schroot to actually create the kernel when you do that".

Thx

nomorewindows · 2012-01-27 01:28:18

Depending on the live CD you have you may or may not be able to do that. More than likely, it will complain that the kernel on the live CD is too old to schroot your newer kernel. It's easier if you can at least boot your normal kernel in some way, but this is not always possible. You probably already have your partition mounted somewhere (i.e. /mnt). All you need to do then is preferrably from /, type in at the shell prompt 'schroot /mnt' or 'chroot /mnt' whatever the case may be. This will tell you whether it will work with your live CD. But your /boot partition has to also be mounted along with the rest of your partition under /mnt, if it's on a separate partition from your / is. So you will need to do that before you schroot with the hopes of being successful with your 'mkinitcpio -k /boot/your-kernel -c /etc/mkinitcpio.conf -g /boot/your-initramfs-image' command. Making your fallback kernel would be nice, (just adding -S autodetect to the previous line), but you want to see if it will boot. Then when it finishes. back out of your schroot, and unmount your partitions (to be safe) and reboot. I always advise going ahead and installing the lts kernel also, so that you can have the possibility of getting back into your machine if something goes wrong with your primary kernel.

Last edited by nomorewindows (2012-01-27 01:33:31)

leberyo · 2012-01-27 19:13:38

Thanks for the tip nomorewindows.
Any way you could show me in baby language?
I tried running chroot /mnt and it said chroot: failed to run command '/bin/bash' : No such file or directory.

Thx so much

nomorewindows · 2012-01-28 00:17:14

leberyo wrote:

Thanks for the tip nomorewindows.
Any way you could show me in baby language?
I tried running chroot /mnt and it said chroot: failed to run command '/bin/bash' : No such file or directory.
Thx so much

Did you mount your / partition first and of course your /boot? If you schrooted without mounting it won't find /bin/bash, because it's not there.

Last edited by nomorewindows (2012-01-28 00:18:33)

leberyo · 2012-01-28 00:25:52

Typed:
mount /
.... already mounted
mount /boot
chroot /mnt
chroot: failed to run command '/bin/bash': No such file or directory

nomorewindows · 2012-01-28 01:09:47

This is why it gets confusing...
If you did 'ls /mnt' it would show no files, because of the message 'already mounted'. You have to explicitly tell it that you want something else to be mounted other than what the live CD is already using. If you try to mount your normal partition over the live CDs / mount then you'll really have problems and have to start over.
Your partition on your harddrive needs to be mounted on /mnt as your /, so that would be mount /dev/xxxx /mnt. If you were able to actually get to your partiton to edit /etc/mkinitcpio.conf (which would've actually been under /mnt/etc/mkinitcpio.conf), then you could just view your /etc/fstab to see where your partitions lie (under /mnt/etc/fstab). Then mount boot on top of that, mount /dev/xxxxx /mnt/boot. So now you have your normal setup (more or less) mounted under /mnt instead of the normal place it would mount if it would mount by itself. This is where the chroot comes in to make what you have mounted on /mnt to be just like it had mounted properly. Once you've done that then you can make the kernel, but at this point we haven't found out whether your live CD is adequate yet. If you are editing just /etc/mkinitcpio.conf it's probably the one on the live CD. Since you booted from the live CD / is in reference to the live CD, *NOT* the system you are normally using.

leberyo · 2012-01-28 01:20:18

Elsewhere somebody said to do:

mkinitcpio -p linux

Now I can't even get into command prompt.

Checking my mtab file, the partitions seem to be all off now. I can only see /dev/sda3 /mnt/sda3 ext4 rx...]

&*^@#$

Arch Linux

#1 2012-01-21 10:25:39

udev breaking the booting process

#2 2012-01-21 14:38:55

Re: udev breaking the booting process

#3 2012-01-21 21:42:09

Re: udev breaking the booting process

#4 2012-01-21 21:56:21

Re: udev breaking the booting process

#5 2012-01-21 22:05:05

Re: udev breaking the booting process

#6 2012-01-21 23:45:16

Re: udev breaking the booting process

#7 2012-01-22 00:08:33

Re: udev breaking the booting process

#8 2012-01-22 00:42:07

Re: udev breaking the booting process

#9 2012-01-22 00:55:27

Re: udev breaking the booting process

#10 2012-01-22 01:06:34

Re: udev breaking the booting process

#11 2012-01-22 01:43:55

Re: udev breaking the booting process

#12 2012-01-22 01:45:19

Re: udev breaking the booting process

#13 2012-01-22 04:34:11

Re: udev breaking the booting process

#14 2012-01-23 18:00:11

Re: udev breaking the booting process

#15 2012-01-25 15:19:16

Re: udev breaking the booting process

#16 2012-01-26 13:01:19

Re: udev breaking the booting process

#17 2012-01-26 19:49:03

Re: udev breaking the booting process

#18 2012-01-26 21:55:48

Re: udev breaking the booting process

#19 2012-01-27 00:57:19

Re: udev breaking the booting process

#20 2012-01-27 01:28:18

Re: udev breaking the booting process

#21 2012-01-27 19:13:38

Re: udev breaking the booting process

#22 2012-01-28 00:17:14

Re: udev breaking the booting process

#23 2012-01-28 00:25:52

Re: udev breaking the booting process

#24 2012-01-28 01:09:47

Re: udev breaking the booting process

#25 2012-01-28 01:20:18

Re: udev breaking the booting process

Board footer