You are not logged in.
So I tried to update today, power got busted (happens like 5 times a day here and freaks everyone) and the kernel panic ('tried to kill init' one) message greeted me after an initial systemd run.
I loaded live image, chrooted (mount my partition, then arch-chroot into it, maybe something wrong here?), reinstalled like 50 packages (pacman was giving a huge list of "empty" libraries), reinstalled 'linux', generated an image with 'mkinitcpio -p linux', reboot.
Kernel panic (another one - see below).
No systemd messages even.
And that's where I stuck. Wasted about 4 or 5 hours trying to get everything back to normal.
I'm sorry I can't give exact configs/panic output/etc now, but I'll describe/get them from same cases or provide later.
Panic gives
kernel panic-not syncing: VFS: unable to mount root fs on unknown-block(0,0)And pretty much nothing more (no advices like 'specify root/init', no info, just this and some codes which mean pretty much nothing to me. I can provide them later though if they are needed).
I use syslinux for bootloader and stock kernel with no modifications. 'Root' is set by UUID in syslinux.cfg.
I tried generating new images with (with and without doing depmod previously) mkinitcpio, reinstalling/downgrading both mkinitcpio and linux packages, using stock mkinitcpio.conf (added fglrx to hooks though), using another linux package (linux-lts), using fallback images, using stock syslinux.cfg (without UUID), installing grub (and autoconfiguring it) to no avail. All this was done with chroot. Maybe I forgot to include something in this list, sorry about that.
The problem is not with HDD itself (chroot, fsck goes without problems, Windows on other partition loads okay, etc) nor with bootloader (grub and syslinux give same results). mkinitcpio gives no unusual errors/messages when generating images.
One single thing left that I can think about is to reinstall all essential packages (as in installation guide), because clearly one of them is failing things for me - I just don't know which. Or to get a definitely working image from my laptop and see if it will boot on PC maybe?
Please help me fix this.
!!Update: it has been found that mkinitcpio generates initramfs of 50 bytes and that causes trouble.
!!Update2: Fix was found!
If you ever get stuck with this problem, then
Get a live CD and chroot into your machine, finish interrupted update, fix pacman package manually if needed (see wiki)
pacman -S baseReinstall all packages which 'pacman -Qi mkinitcpio' shows in 'Depends On'
Optionally generete new initramfs with mkinitcpio, boot into your machine (fallback is better here)
Run
pacman -Qkk 2>&1 | grep archiveTo find out which packages are broken, reinstall them.
Reinstall something (like pacman -S vi) and see if there are 'empty' libraries (pacman will swear at the end of reinstall if there are), find out which packages hold them, reinstall those
Generete initramfs image and boot into your machine if you haven't yet/reboot if you have. It should be fixed now.
And reinstall here means not just 'pacman -S', but also deal with 'file exists' errors. There will be plenty of those.
Last edited by Matalonder (2014-07-20 10:00:08)
Offline
Perhaps run pacman -Qk (or -Qkk) in chroot.
Offline
Perhaps run pacman -Qk (or -Qkk) in chroot.
-Qk returned only errors in some perl packages, -Qkk gave a hefty amount of interesting messages.
Some of them are warnings 'modification time mismatch' or 'size mismatch' — these are mostly configs I edited or files I think should have got edited by OS at some point.
Some of them are errors though. '/var/lib/pacman/local/*/mtree: Unrecognized archive format'.
I suppose these are corrupted packages which pacman can't even read supposed file contents of?
There are no 'important' ones like these though. Nothing that could break system so badly at least...
error: error while reading file /var/lib/pacman/local/aspell-ru-0.99f7-6/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/autoconf-2.69-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/beep-1.3-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/bluez-utils-5.21-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/ca-certificates-java-20140324-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/ccache-3.1.9-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/chromium-36.0.1985.125-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/compositeproto-0.4.2-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/damageproto-1.2.1-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/dbus-1.8.6-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/ddd-3.3.12-5/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/diffutils-3.3-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/dnssec-anchors-20140629-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/dnsutils-9.9.2.P2-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/docbook-xml-4.5-6/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/dvd+rw-tools-7.1-5/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/extundelete-0.2.4-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/filesystem-2014.07-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/findutils-4.4.2-6/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/fixesproto-5.0-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/flashplugin-11.2.202.394-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/git-2.0.2-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gksu-2.0.2-5/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gnome-doc-utils-0.20.10-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gnome-mime-data-2.18.0-7/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gnupg-2.0.25-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gparted-0.19.1-1/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gstreamer0.10-python-0.10.22-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/gtk-aurora-engine-1.5.1-4/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/icon-naming-utils-0.8.90-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/intltool-0.50.2-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/ipcalc-0.41-5/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/java-rhino-1.7R4-2/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/jre7-openjdk-7.u65_2.5.1-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/jre7-openjdk-headless-7.u65_2.5.1-3/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/libcl-1.1-4/mtree: Unrecognized archive format
error: error while reading file /var/lib/pacman/local/mozilla-common-1.4-4/mtree: Unrecognized archive formatOffline
The error in your first post makes me think of initcpio problems, so triple-check you have the proper initrd line in syslinux/grub and that freshly created initramfs-linux.img with a decent size (3.8 MB on this PC) sits in /boot. Also make sure the root= parameter points to your root partition.
Offline
The error in your first post makes me think of initcpio problems, so triple-check you have the proper initrd line in syslinux/grub and that freshly created initramfs-linux.img with a decent size (3.8 MB on this PC) sits in /boot. Also make sure the root= parameter points to your root partition.
Actually image is 50 bytes...
And also I noticed that mkinitcpio gives
modprobe: ERROR: missing parameters. See -hafter hook autodetect. Removing it doesn't change anything though.
Root and initrd are okay.
Offline
Well, at least you know you have to get your initcpio image in order.
-edit-
Can't check things right now, but perhaps you're not mounting /dev, /proc and whatnot in chroot, and that's why you get an almost empty image. Or there is something wrong with the modules in /usr/lib/ or /lib or wherever.
Last edited by lucke (2014-07-19 15:41:42)
Offline
Can't check things right now, but perhaps you're not mounting /dev, /proc and whatnot in chroot, and that's why you get an almost empty image.
No, arch-chroot does that automatically. Though I tried manual chroot just now and it didn't work.
Or there is something wrong with the modules in /usr/lib/ or /lib or wherever.
Huh. There are modules, but how can I check if there is something wrong with them?
Offline
Try reinstalling syslinux and update the boot record. Run
# syslinux-install_update -i -a -mfrom chroot. See wiki for more information.
What is your partition table? Do you use separate partition for /boot? What filesystem do you use for /boot? Syslinux has a few problems with btrfs and sometimes it fails to load initrd/kernel from btrfs filesystem.
Read it before posting http://www.catb.org/esr/faqs/smart-questions.html
Ruby gems repository done right https://bbs.archlinux.org/viewtopic.php?id=182729
Fast initramfs generator with security in mind https://wiki.archlinux.org/index.php/Booster
Offline
Try reinstalling syslinux and update the boot record. Run
# syslinux-install_update -i -a -mfrom chroot. See wiki for more information.
I don't think that'll do any good seeing that doing same with GRUB didn't help and that my initramfs is 50 bytes.
What is your partition table? Do you use separate partition for /boot? What filesystem do you use for /boot? Syslinux has a few problems with btrfs and sometimes it fails to load initrd/kernel from btrfs filesystem.
MBR, no I don't use separate /boot, it's ext4 and everything should be ok with that?
Last edited by Matalonder (2014-07-19 16:46:25)
Offline
I don't think that'll do any good seeing that doing same with GRUB didn't help and that my initramfs is 50 bytes.
Ah, missed this. Yes the problem is in your initramfs image. Now we need to understand why it does not generated properly. A few things to look at:
mkinitcpio sources https://projects.archlinux.org/mkinitcpio.git/ check where modprobe is called
Run mkinitcpio with strace, it will show you what files and what program it starts:
strace -f -e trace=process,file $CMD
where $CMD is mkinitcpio that fails. Check all failed syscalls/file accesses. It should give you pointers what is going wrong there.
Read it before posting http://www.catb.org/esr/faqs/smart-questions.html
Ruby gems repository done right https://bbs.archlinux.org/viewtopic.php?id=182729
Fast initramfs generator with security in mind https://wiki.archlinux.org/index.php/Booster
Offline
mkinitcpio sources https://projects.archlinux.org/mkinitcpio.git/ check where modprobe is called
Too bad I'm not a programmer yet to understand at least something there... Though thanks for suggestion
Run mkinitcpio with strace, it will show you what files and what program it starts:
strace -f -e trace=process,file $CMD
where $CMD is mkinitcpio that fails. Check all failed syscalls/file accesses. It should give you pointers what is going wrong there.
Well, I done that, parsed with grep for "No such file" and got 22526 lines with that error. Much of them really are modules as were suggested but then how do I get them back?
If modules are part of linux-headers then I already reinstalled that.
By the way, I spotted gcc among those 'no file' lines, though I have gcc installed and it works.
Offline
Are you talking about modules in /usr/lib/modules/$VERSION-ARCH? Are they missed on your machine? The modules come with 'linux' package but you already reinstalled this. Hmm... Do you see any errors when you reinstall linux? Do you have enough space on your disk?
Read it before posting http://www.catb.org/esr/faqs/smart-questions.html
Ruby gems repository done right https://bbs.archlinux.org/viewtopic.php?id=182729
Fast initramfs generator with security in mind https://wiki.archlinux.org/index.php/Booster
Offline
Are you talking about modules in /usr/lib/modules/$VERSION-ARCH? Are they missed on your machine? The modules come with 'linux' package but you already reinstalled this. Hmm... Do you see any errors when you reinstall linux? Do you have enough space on your disk?
Yes, those modules. They are not missed. Not all of them. For example
/usr/lib/modules/3.15.5-2-ARCH/build/arch/x86/include/generated/autoconf.h
is missing, and is reported as missing by strace output. Though it is missing also on my laptop, but mkinitcpio works okay here.
Also for comparison the same strace test on laptop gives only 178 'no file' lines. And not a single one about modules - though at least one (autoconf.h) is missing here too.
No, no errors aside from the usual fglrx one (it's always 'Failed!' for some reason, but later works after reboot...) and 'libOpenCL.so is empty, not checked' which pacman outputs whenever I install anything at all. It'll be fixed as soon as I'm able to login as user (aur package), and shouldn't break anything.
Yes, enough space. 42 GiB free to be exact.
Offline
Have you tried running mkinitcpio with the -v flag?
-edit-
Please put the output of mkinitcpio with -v and of that strace command mentioned above in pastebin, and link us to it.
Last edited by lucke (2014-07-19 18:46:56)
Offline
Have you tried running mkinitcpio with the -v flag?
-edit-
Please put the output of mkinitcpio with -v and of that strace command mentioned above in pastebin, and link us to it.
mkinitcpio -v -p linux: http://pastebin.com/gSizGstB (disregard 'Note' part)
Full strace&parsed strace: too big for pastebin, so I just upload txt's in archive: http://rghost.net/56987428
res — full, res2 — parsed.
Offline
As you mentioned in your first post, you could try booting with initramfs-linux-fallback.img from your other computer - if it boots, you won't have to play with chroot and whatnot, and it might be easier to solve your problem.
Is your arch x86 or x86_64?
What's the output of "du -hs /usr/lib/modules/3.15.5-2-ARCH"?
Offline
lucke wrote:Have you tried running mkinitcpio with the -v flag?
-edit-
Please put the output of mkinitcpio with -v and of that strace command mentioned above in pastebin, and link us to it.
mkinitcpio -v -p linux: http://pastebin.com/gSizGstB (disregard 'Note' part)
Full strace&parsed strace: too big for pastebin, so I just upload txt's in archive: http://rghost.net/56987428
res — full, res2 — parsed.
I see a lot of ENOENT errors when accessing /tmp/mkinitcpio.XXX folders. Could it be that you ran out of space in the build directory? /tmp is tmpfs and has limited size. Run mkinitcpio with '--builddir' flag and specify other builddir, point it to non-tmpfs place e.g. ~/mytmp.
Read it before posting http://www.catb.org/esr/faqs/smart-questions.html
Ruby gems repository done right https://bbs.archlinux.org/viewtopic.php?id=182729
Fast initramfs generator with security in mind https://wiki.archlinux.org/index.php/Booster
Offline
I see a lot of ENOENT errors when accessing /tmp/mkinitcpio.XXX folders. Could it be that you ran out of space in the build directory? /tmp is tmpfs and has limited size. Run mkinitcpio with '--builddir' flag and specify other builddir, point it to non-tmpfs place e.g. ~/mytmp.
That didn't help, still 50 bytes.
As you mentioned in your first post, you could try booting with initramfs-linux-fallback.img from your other computer - if it boots, you won't have to play with chroot and whatnot, and it might be easier to solve your problem.
Is your arch x86 or x86_64?
What's the output of "du -hs /usr/lib/modules/3.15.5-2-ARCH"?
x86_64.
du gives 132M.
Also it booted with fallback image from laptop, in X even, though it threw out many errors. How can I access those messages? I made one photo while it fsck'd, but that's it. There were much more: http://i.imgur.com/Q2TWwxy.jpg
Too bad I can't even use that tip because dbus doesn't work.
Offline
You can look at journalctl for non-X errors.
For reference, "du -hs /usr/lib/modules/3.15.5-2-ARCH/" on my x86 machine shows 71 MB, and this is the output of mkinitcpio -v creating proper images. Matalonder's mkinitcpio adds only 4 (usb-related) modules to the images.
Have you tried running mkinitcpio after booting with that borrowed image, Matalonder?
-edit-
Show us the output of "journalctl -b --no-pager" and of "systemctl status systemd-modules-load.service". Or "journalctl -u systemd-modules-load.service". journalctl should work without dbus, I would presume.
Last edited by lucke (2014-07-19 20:12:18)
Offline
You can look at journalctl for non-X errors.
Aha, looks like it's mostly 'failed to connect to system bus: no such file' and 'failed to start Login service'. Also 'Cannot open /var/lib/rpcbind/rpcbind.xdr /var/lib/rpcbind/portmap.xdr file for reading, errno 2 (no such file or directory)' once.
Have you tried running mkinitcpio after booting with that borrowed image, Matalonder?
Yes, still 50 bytes.
Show us the output of "journalctl -b --no-pager"
"systemctl status systemd-modules-load.service".
● systemd-modules-load.service - Load Kernel Modules
Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
Active: failed (Result: exit-code) since Sat 2014-07-19 22:41:38 EEST; 36min ago
Docs: man:systemd-modules-load.service(8)
man:modules-load.d(5)
Process: 133 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
Main PID: 133 (code=exited, status=1/FAILURE)
Jul 19 22:41:37 alicex systemd-modules-load[133]: Failed to insert 'fglrx': Invalid argument
Jul 19 22:41:38 alicex systemd-modules-load[133]: Inserted module 'vboxdrv'
Jul 19 22:41:38 alicex systemd-modules-load[133]: Inserted module 'snd_seq_oss'
Jul 19 22:41:38 alicex systemd-modules-load[133]: Inserted module 'snd_pcm_oss'
Jul 19 22:41:38 alicex systemd-modules-load[133]: Inserted module 'microcode'
Jul 19 22:41:38 alicex systemd-modules-load[133]: Inserted module 'f71882fg'
Jul 19 22:41:38 alicex systemd-modules-load[133]: Inserted module 'nfs'
Jul 19 22:41:38 alicex systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE
Jul 19 22:41:38 alicex systemd[1]: Failed to start Load Kernel Modules.
Jul 19 22:41:38 alicex systemd[1]: Unit systemd-modules-load.service entered failed state.Looks like fglrx went berserk?
Update: It's pretty late here so I'll continue trying to do something tomorrow.
Last edited by Matalonder (2014-07-19 20:26:41)
Offline
I'd remove fglrx/catalyst and see if there is any improvement. Also, look for any pacnew (save?) files in /etc. See what is in /etc/modprobe.d/. "modprobe -r nfs; modprobe nfs" to confirm that modprobe itself works properly (as it seems to do, at least outside of mkinitcpio). "pacman -S base" and "pacman -S" whatever "pacman -Qi mkinitcpio" shows as depends. To save yourself trouble, make sure the laptop's initcpio image isn't named in such a way that running mkinitcpio overwrites it.
Offline
Also, look for any pacnew (save?) files in /etc.
There were no suspicious .pacnew's (these exist only for files modified by me as they should), so I just looked through /etc/ for files which got yesterday's timestamp (otherwise they couldn't do anything wrong, right?) and found only plausible ones.
While doing the file listing I found out that 'find' was broken and reinstalled it though.
"modprobe -r nfs; modprobe nfs" to confirm that modprobe itself works properly (as it seems to do, at least outside of mkinitcpio)
No errors.
"pacman -S base" and "pacman -S" whatever "pacman -Qi mkinitcpio" shows as depends.
Filesystem got messed up
error: failed to commit transaction (conflicting files)
filesystem: /bin exists in filesystem
filesystem: /etc/arch-release exists in filesystem
filesystem: /etc/mtab exists in filesystem
filesystem: /etc/profile.d/locale.sh exists in filesystem
filesystem: /lib exists in filesystem
filesystem: /lib64 exists in filesystem
filesystem: /sbin exists in filesystem
filesystem: /usr/lib/modprobe.d/usb-load-ehci-first.conf exists in filesystem
filesystem: /usr/lib/os-release exists in filesystem
filesystem: /usr/lib64 exists in filesystem
filesystem: /usr/local/share/man exists in filesystem
filesystem: /usr/sbin exists in filesystem
filesystem: /usr/share/man/man7/archlinux.7.gz exists in filesystem
filesystem: /var/lock exists in filesystem
filesystem: /var/mail exists in filesystem
filesystem: /var/run exists in filesystemWhen I fixed it+diffutils and reinstalled base, initramfs generated okay and fglrx too. Huh. Why didn't I do that earlier?..
Thank you very much. I'll update opening post with info about broken packages (for people with same problem) and mark it solved if reboot now goes okay.
Packages that I found to be broken and fixed:
diffutils
filesystem
findutils [mkinitcpio dependency!]
gtk-aurora-engineOffline
Does it seem to you that pacman forgot that these packages had been installed, and that's why "pacman -Qk" didn't help?
In truth, you did mention reinstalling essential packages in the first post.
Offline
Does it seem to you that pacman forgot that these packages had been installed, and that's why "pacman -Qk" didn't help?
In truth, you did mention reinstalling essential packages in the first post.
Looks like with just -k it tests existence of the files — which were there, just zeroed. While with -kk it found that file lists for packages (or something else, I'm not sure) were corrupted and that helped. So files exist, package is noted as installed and updated — pacman does not reinstall in as mkinitcpio dependency and thinks it's allright.
In truth if I just reinstalled everything (especially find — it was broken such a way that didn't even output errors — just nothing, like it found no files) from that first -Qkk output it would be ok already then.
So you actually solved everything with your first post here. It's just my knowledge was too weak to tell right away that some of mine 'Nothing that could break system so badly at least...' packages actually could and did break image generation. Sorry, hehe...
Last edited by Matalonder (2014-07-20 14:09:02)
Offline
Well, you didn't go for the easy way of reinstalling Arch, and you learnt something ;-) Good for you.
Last edited by lucke (2014-07-20 14:11:11)
Offline