System doesn't boot after upgrade

nathanb · 2013-09-06 16:26:03

I just did a full pacman -Syu, and after reboot the system just hangs after the fsck stage.

It displays a message that the root volume is not read-write and may be fsck'd again (which it used to display before with no problems), and then it shows fsck results for each local partition. After that, it just hangs. No output (I left it sitting for 15 minutes), no response to keys. Trying to use the magic SysRq keys displays a message saying that the sysrq operation is not enabled.

Soft reboot doesn't work. I have to hold down the power switch. Booting the fallback image has the same result.

I can boot off a USB drive, but I'm not sure what's going wrong so I don't know where to start looking. Should I try reinstalling grub? Downgrading the kernel? I admit that I know less about the Arch initialization sequence than I should. What is supposed to be happening after fsck that's not?

WonderWoofy · 2013-09-06 16:58:48

Chroot in and then check your journal. Of course, the first step you should probably take is to remove the 'quiet' from your kernel command line so that you better see where the boot is stalling. Of course, with the parallel starting of services, this is probably not going to be entirely clear. So if that is the case, use the chroot/journal method.

Alternatively, you could simply mount your rootfs from the live media (assuming it is the Arch live media or something else systemd based), then use journalctl --directory=/mnt/var/log/journal to access it without actually chrooting.

The warning about the rootfs not being rw is just because systemd has changed the way that the systemd-fsck@.service decides what to fsck. If the rootfs is mounted read-only, then the rootfs will be fsck'ed. But if you mount it rw, it will assume that it has been fsck'ed by the initramfs and will skip it. The warning will only show if you have the fsck hook built into your initramfs anyway, so it is safe to make mount it rw. Simply omitting the 'ro' from the kernel command line is not enough. You have to explicitly inidcate 'rw'.

If you want to enable the Magic SysRq key, then you need to do so in /etc/sysctl.conf or create a similar file in /etc/sysctl.d that is parsed lexicographically before 50-default.conf. Of course, for the functionality to be enabled, the systemd-sysctl.service has to have run. From what you have posted, it is not entirely clear how far into the boot you have gotten.

nathanb · 2013-09-06 21:32:11

Took the quiet flag off the kernel parameters, and I see that ypbind and nfsd are having trouble starting...I'll troubleshoot those services once it actually gives up on trying to start them. Thanks for the help.

cfr · 2013-09-07 01:18:44

If you are using grub and autogenerate grub.cfg you should not be seeing that warning. In itself, that is not a problem - the warning is harmless. But it may mean that you have not reinstalled grub to disk and/or generated a new grub.cfg and/or updated /default/grub in light of any .pacnew etc. That raises further questions about whether you've followed other post-upgrade instructions from pacman and dealt with other .pacnew/.pacsave files.

nathanb · 2013-09-07 01:26:13

Yes, yes, I'm a horrible person and should probably be flogged for my disobedience.

I try to inspect any .pacnew files that pacman creates, but I don't exactly sit there watching the bloody thing as it installs 300 packages. After install I'll grep /etc for any .pacnew files, but if it's in a weird place like /boot/grub, I may not see it. If pacman were actually helpful and displayed a summary of actions I needed to take at the end of the install, I'd be much more likely to actually take said actions.

Because I still have a hosed system, I used updatedb and locate to find all pacnew files and resolve all of them, but that doesn't seem to have fixed nfsd and ypbind.

WonderWoofy · 2013-09-07 01:29:47

cfr wasn't trying to tell you that you were a bad person for not perfectly maintainging your system. Rather, just that your problems seem to indicate that there might be further issues that may or may not be affecting your situation.

For the *.pac* files, there is a tool that ships with pacman that can handle this for you. It gives you the choice of using find or mlocate, and allows you to see the differences between the two.

So no one here is trying to attack you, but part of solving a problem is determining exactly what the problem is.

jcasper · 2013-09-09 18:17:49

I had the same problem. I was able to get into the system and get things working by disabling ypbind in systemctl while in single user mode (add 'single' to the kernel command line, then 'systemctl disable ypbind' when you get to a prompt, then 'systemctl default' to continue booting). I then started ypbind manually after the system booted and it seems to be working okay. I don't have much time right now to look into it further to see what the problem is, but thought that might help someone. Did you ever figure out what was causing your problems nathanb?

cfr · 2013-09-10 00:34:57

WonderWoofy wrote:

cfr wasn't trying to tell you that you were a bad person for not perfectly maintainging your system. Rather, just that your problems seem to indicate that there might be further issues that may or may not be affecting your situation.

^ This .

For the *.pac* files, there is a tool that ships with pacman that can handle this for you. It gives you the choice of using find or mlocate, and allows you to see the differences between the two.

I recommend pacmatic. It does not catch kdmrc.pacnew but it seems to get pretty much everything else I use.

nathanb · 2013-09-10 03:13:28

jcasper wrote:

Did you ever figure out what was causing your problems nathanb?

No, not really. ypbind consistently fails on boot, but it will work (eventually) when manually enabled. So here's what I have to do:
* Disable ypbind from automatic loading
* Boot the system
* Go make some tea
* Type 'systemctl start ypbind'
* If it barfs due to timeout, you did not let your tea steep long enough. Try again.
* Eventually it will work

This is pretty sucky and I am not at all satisfied with this situation. Sometime when I have some time (this is my work PC, so gotta wait for work to die down) I'll just nuke it from space and install a fresh Arch, which will hopefully fix the problem.

Last edited by nathanb (2013-09-10 03:13:46)

ewaller · 2013-09-10 04:14:33

nathanb wrote:

[* If it barfs due to timeout, you did not let your tea steep long enough. Try again.
* Eventually it will work

As an engineer who likes things to be deterministic, and one with no concept as to how long tea should steep, there must be a better way.

Are you sure that your network is starting properly? Are you on a LAN? Is it Wired or Wireless? Earl Grey or English Breakfast?

BTW: I am a coffee drinker. Tea, for me, is Earl Grey in one of those unforgivable bags, allowed to steep for the entire life cycle of the tea whilst I drink it. I know, "What a barbarian? "

nathanb · 2013-09-10 04:19:51

ewaller wrote:

Are you sure that your network is starting properly?

Yes, even when ypbind fails I can browse the web with links and even ping the yp server.

ewaller wrote:

Are you on a LAN? Is it Wired or Wireless?

Wired LAN.

ewaller wrote:

Earl Grey or English Breakfast?

Rooibos.

ewaller · 2013-09-10 04:33:38

nathanb wrote:

ewaller wrote:
Are you sure that your network is starting properly?
Yes, even when ypbind fails I can browse the web with links and even ping the yp server.

Not what I would have expected.

* Type 'systemctl start ypbind'
* If it barfs due to timeout, you did not let your tea steep long enough. Try again.

Clearly, there is a problem with dependencies. I would look into which services were, or were not running when starting ypbind fails and when it does not. I know I am not being specific -- again, sorry, I don't use yp. I am only offering some things I would investigate were this my system.

nathanb wrote:

ewaller wrote:
Earl Grey or English Breakfast?
Rooibos.

Never heard of it (ewaller does basic research).

google wrote:

For centuries, the indigenous people of South Africa have enjoyed "red teas" made from the needle-like leaves of the Rooibos (pronounced roy-boss) bush. Today, red tea goes by many names, including rooibos tea, red rose tea, red clover tea, and red diamond tea. But no matter what you call it, our unique Rooibos blends are caffeine-free with a pleasing sweetness and distinctive taste.

NO CAFFEINE Yikes !

nathanb · 2013-09-10 18:59:38

I don't know if this is related, but my network adapter appears to have been grandfathered in from the interface renaming shenanigans perpetrated on an unwitting public some months ago:

[nathanb@nathanb-box ~] ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fc:4d:d4:3d:2f:0b brd ff:ff:ff:ff:ff:ff
    inet 10.62.197.120/22 brd 10.102.79.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::fe4d:d4ff:fe3d:2f0b/64 scope link 
       valid_lft forever preferred_lft forever
3: vboxnet0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff

I think (as you do) that the root cause may very well be network related, but I'm having trouble finding the exact problem. The kernel init messages seem to indicate that the network has reached a fully operational state before it starts the ypbind service, and other network services like openntpd seem to be starting happily.

I have a booted system now so I'm not going to disturb it at the moment, but once I have a free couple of hours I'll play around with it some more and see if I can find the root of the problem.

ewaller wrote:

NO CAFFEINE Yikes !

I quit caffeine while in grad school and haven't looked back. I have to be more disciplined about my sleep schedule and exercise regimen, etc., but not being dependent on chemicals for my alertness is (IMO) worth it

Arch Linux

#1 2013-09-06 16:26:03

System doesn't boot after upgrade

#2 2013-09-06 16:58:48

Re: System doesn't boot after upgrade

#3 2013-09-06 21:32:11

Re: System doesn't boot after upgrade

#4 2013-09-07 01:18:44

Re: System doesn't boot after upgrade

#5 2013-09-07 01:26:13

Re: System doesn't boot after upgrade

#6 2013-09-07 01:29:47

Re: System doesn't boot after upgrade

#7 2013-09-09 18:17:49

Re: System doesn't boot after upgrade

#8 2013-09-10 00:34:57

Re: System doesn't boot after upgrade

#9 2013-09-10 03:13:28

Re: System doesn't boot after upgrade

#10 2013-09-10 04:14:33

Re: System doesn't boot after upgrade

#11 2013-09-10 04:19:51

Re: System doesn't boot after upgrade

#12 2013-09-10 04:33:38

Re: System doesn't boot after upgrade

#13 2013-09-10 18:59:38

Re: System doesn't boot after upgrade

Board footer