System fails to boot after routine upgrade.

warl0ck · 2020-04-29 02:19:33

Here is a reddit thread with the history of what I've done so far. I was also in the IRC room if anyone remembers that lovely chat.

https://www.reddit.com/r/archlinux/comm … _any_help/

TL;DR, My system is behaving as if someone deleted a core directory, or I pulled the plug during an update or something. No such thing has occurred. I simply updated my system (that has been stable for well over two years), and when I booted back up, pretty much every service was failing.

[FAILED] Failed to start dhcpcd on eno1.
[FAILED] Failed to start Docker Application Container Engine.
[FAILED] Failed to start Network Time Service
[FAILED] Failed to start Permit User Sessions
[FAILED] Failed to start MariaDB 10.3.12 database server
[FAILED] Failed to start Light Display Manager

This will hang and the only thing I can do is reset it again. When I investigate the logs, it seems like the thing is consistently failing is systemd-logind, which in turn is causing all other services to fail, most notably lightDM, so I am unable to access the GUI. a log I took: http://ix.io/2jQZ

When I press the reset button, one of three things happens:

1. I boot into the previously described mode, where I am unable to access a TTY or boot into any sort of GUI.

2. I boot into a "halfway" working system. I am able to reach lightDM and even log into i3, but things don't work correctly. Most notably my keybind to launch urxvt does nothing, and Firefox (which is set to autolaunch) fails instantly, and will not restart.

Notably the i3 "placeholder" windows when used when using a layout (which are urxvt windows) are appearing, but the commands that are set to run and populate them with data fail to execute. Qt programs like keepasscx and ckb-next are loading fine.

3. I boot into a completely working state, indistinguishable from normal behavior. This happens maybe 1/20 reboots.

I have absolutely no clue what is causing this behavior. When I google these errors, it's mostly people accidentally nuking a directory or pulling the plug during a critical update. If I had gone and done one of these things, I fail to see how the system would start correctly even once.

Please take a look at the linked thread for everything that has been suggested and tried, but a short list:
* rolled back kernel
* rolled back nvidia drivers
* checked SMART status of disks
* check for missing /dev/pts

Any help would be much appreciated as I am right in the middle of a semester and don't have the luxury of just nuking this and starting from scratch atm.

EDIT: someone tells me they have had similar symptoms when their fstab file got messed up. They say I should have an entry for /boot, but I do not. I haven't touched the fstab in years so I don't know how this one is supposed to look like.

Last edited by warl0ck (2020-04-29 03:24:04)

dakota · 2020-04-29 03:59:55

Is it normal that both of these services would be enabled? Or is there a conflict?

Apr 27 18:18:10 archlinux systemd[1]: Starting dhcpcd on all interfaces...
-- Subject: A start job for unit dhcpcd.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit dhcpcd.service has begun execution.
-- 
-- The job identifier is 73.
Apr 27 18:18:10 archlinux crond[455]: (CRON) STARTUP (1.5.5)
Apr 27 18:18:10 archlinux systemd[1]: Starting dhcpcd on eno1...
-- Subject: A start job for unit dhcpcd@eno1.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit dhcpcd@eno1.service has begun execution.
-- 
-- The job identifier is 82.

Followed by...

Apr 27 18:18:10 archlinux systemd[1]: Started dhcpcd on all interfaces.
-- Subject: A start job for unit dhcpcd.service has finished successfully
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit dhcpcd.service has finished successfully.
-- 
-- The job identifier is 73.

~~~~~~

Apr 27 18:18:40 archlinux systemd[1]: dhcpcd@eno1.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- An ExecStart= process belonging to unit dhcpcd@eno1.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Apr 27 18:18:40 archlinux systemd[1]: dhcpcd@eno1.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The unit dhcpcd@eno1.service has entered the 'failed' state with result 'exit-code'.
Apr 27 18:18:40 archlinux systemd[1]: Failed to start dhcpcd on eno1.
-- Subject: A start job for unit dhcpcd@eno1.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit dhcpcd@eno1.service has finished with a failure.
-- 
-- The job identifier is 82 and the job result is failed.

On my system, only dhcpcd@enp0s25.service is running. I seem to remember a thread very recently about this, but I can't find it right now. You might try reading through the recent dhcpcd threads. (Or you could wait and someone much smarter than I am will come along and point you in the right direction).

Cheers,

warl0ck · 2020-04-29 04:09:28

dakota wrote:

Is it normal that both of these services would be enabled? Or is there a conflict?

Yeah, someone on reddit pointed that out as well. I haven't really touched my network settings since I got wifi to work 3 years ago, i haven't needed to change it.

I certainly don't remember duplicating the dhcpcd service.

I do always see a dhcpcd error when I boot though, for months now. It has never affected anything, someone just told me it means it's listening to an ethernet port that I'm not using.

dakota · 2020-04-29 04:35:38

Yes, but dhcpcd was recently upgraded. For some people this upgrade has caused problems with network-related services. You could try one of the following:

1. look in your pacman log to see if the most recent dhcpcd upgrade coincided with the problem you are now facing. And if it does, see if downgrading the package helps.
2. disable just the dhcpcd.service and reboot (leaving dhcpcd@xxx.service enabled).

(Note: I may be out of line. I believe forum etiquette discourages commenting unless one has a good idea what the problem/solution are... but since no one else has answered, I thought I'd take a shot.)

Cheers,

V1del · 2020-04-29 06:32:44

You are not out of line, and your advice is sound. It isn't in general discouraged to participate, but you should be prepared to receive criticism/have it be pointed out should your advice be completely out of the left field (this is often misconstrued as an attack, most of us here just want to ensure that problems get fixed and prevent that people go off on completely unrelated tangents, potentially wasting hours of time on a solution that cannot possibly help with the problem)

@warl0ck, like I mentioned on reddit, it is never a good idea to have multiple concurrent services running. It isn't completely certain why this particular update led to the kind of fallout you are seeing but it's a quite likely culprit. I'd assume you want the wifi be controlled by netctl, so disable both of these dhcpcd services, reboot and report back if there are still issues. See this example for a thread with the same symptoms, the resolution of disabling concurrent services and some extensive discussion to the possible causes: https://bbs.archlinux.org/viewtopic.php?id=255094

warl0ck · 2020-04-29 16:16:04

Thanks all for the helpful replies, they are *all* appreciated. I'm looking through the dhcpcd bugs and

https://bugs.archlinux.org/task/66397

looks strikingly similar. I will try to downgrade dhcpcd and see if it fixes it!

warl0ck · 2020-04-29 18:48:36

Ok, so downgrading dhcpcd by one version didn't seem to work, but disabling it completely did.

I've booted three times in a row successfully, and I still have wifi so I guess I wasn't using it in the first place!

Should I mark this as resolved or not since the bug definitely still exists?

seth · 2020-04-29 19:12:02

so I guess I wasn't using it in the first place

Multiple services per NIC aren't supported anyway (the trip over each other)

systemctl list-unit-files --state=enabled

JimDeadlock · 2020-04-30 06:44:37

warl0ck wrote:

Should I mark this as resolved or not since the bug definitely still exists?

Speaking as someone who is more often seeking answers than providing them, your problem is solved so you should mark it as such - the purpose being so that people like me can easily spot it when we're in trouble

Arch Linux

#1 2020-04-29 02:19:33

System fails to boot after routine upgrade.

#2 2020-04-29 03:59:55

Re: System fails to boot after routine upgrade.

#3 2020-04-29 04:09:28

Re: System fails to boot after routine upgrade.

#4 2020-04-29 04:35:38

Re: System fails to boot after routine upgrade.

#5 2020-04-29 06:32:44

Re: System fails to boot after routine upgrade.

#6 2020-04-29 16:16:04

Re: System fails to boot after routine upgrade.

#7 2020-04-29 18:48:36

Re: System fails to boot after routine upgrade.

#8 2020-04-29 19:12:02

Re: System fails to boot after routine upgrade.

#9 2020-04-30 06:44:37

Re: System fails to boot after routine upgrade.

Board footer