Arch Linux

Emess

Member

From: Tokyo, Japan

Registered: 2008-01-20

Posts: 21

Systemd complete boot failure

Hi all, after following wiki instructions for the update to systemd, my system is now completely unbootable. I added the flags for debug level logging but I'm not seeing anything further in the logs, it looks like the system just hangs. The initial boot after installation showed more detail in the journal log but still nothing super far down the boot sequence.

May 25 10:14:57 localhost systemd-journal[1962]: Allowing runtime journal files to grow to 1.1G.
May 25 10:14:57 localhost systemd-journal[1962]: Journal started
May 25 10:14:57 localhost systemd-sysctl[1956]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-fsck[2119]: / has gone 300 days without being checked, check forced.
May 25 10:14:58 localhost systemd-sysctl[3147]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3145]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3146]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3148]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3151]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3152]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3153]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:58 localhost systemd-sysctl[3154]: Duplicate assignment of kernel/sysrq in file '/usr/lib/sysctl.d/50-default.conf', ignoring.
May 25 10:14:59 localhost systemd-fsck[2119]: /: 41871/655360 files (0.3% non-contiguous), 444725/2621440 blocks
May 25 10:15:00 localhost systemd[1]: Started File System Check on Root Device.
May 25 10:15:00 localhost systemd[1]: Starting Remount Root and Kernel File Systems...
May 25 10:15:00 localhost systemd[1]: Started Remount Root and Kernel File Systems.
May 25 10:15:00 localhost systemd[1]: Starting Local File Systems (Pre).
May 25 10:15:00 localhost systemd[1]: Reached target Local File Systems (Pre).
May 25 10:15:00 localhost systemd[1]: Starting Load Random Seed...
May 25 10:15:00 localhost systemd[1]: Started Load Random Seed.
May 25 10:15:00 localhost systemd[1]: Found device ST2000DM001-9YN164.
May 25 10:15:00 localhost systemd[1]: Starting File System Check on /dev/sda2...
May 25 10:15:00 localhost systemd-fsck[3166]: /home has gone 300 days without being checked, check forced.
May 25 10:15:29 localhost systemd-fsck[3166]: /home: 58971/121413632 files (1.6% non-contiguous), 452446934/485624576 blocks
May 25 10:15:30 localhost systemd[1]: Started File System Check on /dev/sda2.
May 25 10:15:30 localhost systemd[1]: Mounting /home...
May 25 10:15:30 localhost systemd[1]: Mounted /home.
May 25 10:15:30 localhost systemd[1]: Starting Local File Systems.
May 25 10:15:30 localhost systemd[1]: Reached target Local File Systems.
May 25 10:15:30 localhost systemd[1]: Starting Recreate Volatile Files and Directories...
May 25 10:15:30 localhost systemd[1]: Starting Trigger Flushing of Journal to Persistent Storage...
May 25 10:15:30 localhost systemd[1]: Started Recreate Volatile Files and Directories.
May 25 10:15:30 localhost systemd[1]: Starting Update UTMP about System Reboot/Shutdown...
May 25 10:15:31 localhost systemd-journal[1962]: Allowing system journal files to grow to 1021.2M.
May 25 10:15:31 localhost systemd[1]: Started Trigger Flushing of Journal to Persistent Storage.
May 25 10:15:31 localhost systemd[1]: Started Update UTMP about System Reboot/Shutdown.
May 25 10:15:31 localhost systemd[1]: Starting System Initialization.
May 25 10:15:31 localhost systemd[1]: Reached target System Initialization.
May 25 10:15:31 localhost systemd[1]: Starting Daily Cleanup of Temporary Directories.
May 25 10:15:31 localhost systemd[1]: Started Daily Cleanup of Temporary Directories.
May 25 10:15:31 localhost systemd[1]: Starting Timers.
May 25 10:15:31 localhost systemd[1]: Reached target Timers.
May 25 10:15:31 localhost systemd[1]: Starting D-Bus System Message Bus Socket.
May 25 10:15:31 localhost systemd[1]: Listening on D-Bus System Message Bus Socket.
May 25 10:15:31 localhost systemd[1]: Starting Sockets.
May 25 10:15:31 localhost systemd[1]: Starting D-Bus System Message Bus...
May 25 10:15:31 localhost systemd[1]: Started D-Bus System Message Bus.
May 25 10:15:31 localhost systemd[1]: Starting Permit User Sessions...
May 25 10:15:31 localhost systemd-logind[3184]: New seat seat0.
May 25 10:15:31 localhost systemd[1]: Started Login Service.
May 25 10:15:31 localhost systemd[1]: Started Permit User Sessions.
May 25 10:15:31 localhost systemd[1]: Starting Getty on tty1...
May 25 10:15:31 localhost systemd[1]: Started Getty on tty1.
May 25 10:15:31 localhost systemd[1]: Starting Login Prompts.
May 25 10:15:31 localhost systemd[1]: Reached target Login Prompts.
May 25 10:15:31 localhost systemd[1]: Starting Multi-User System.
May 25 10:15:31 localhost systemd[1]: Reached target Multi-User System.
May 25 10:15:31 localhost systemd[1]: Starting Graphical Interface.
May 25 10:15:31 localhost systemd[1]: Reached target Graphical Interface.
May 25 10:15:31 localhost systemd[1]: Startup finished in 9.511s (kernel) + 35.001s (userspace) = 44.513s.

After setting the hostname and making sure the network was configured (as above it was completely inaccessible still), the boot sequence changed to as it is below with nothing more or less. I'm at a complete loss as to how to get the system back up short of a reinstall. I'm not sure why it mentions graphical interfaces when the system has no GPU card, monitor, or X11 installation.

-- Reboot --
May 25 17:40:18 aya systemd-journal[1972]: Allowing runtime journal files to grow to 1.1G.
May 25 17:40:18 aya systemd-journal[1972]: Journal started
May 25 17:40:18 aya systemd-fsck[2129]: /: clean, 41469/655360 files, 366671/2621440 blocks
May 25 17:40:20 aya systemd-journal[1972]: Allowing system journal files to grow to 1021.2M.
May 25 17:40:20 aya systemd-logind[3191]: New seat seat0.
May 25 18:06:49 aya systemd-journal[1972]: Journal stopped

cfr

Member

From: Cymru

Registered: 2011-11-27

Posts: 7,148

Re: Systemd complete boot failure

Systemd defaults to GUI (graphical.target). You need to switch the default target to multi-user.target to stop it trying to start X.

---

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Emess

Member

From: Tokyo, Japan

Registered: 2008-01-20

Posts: 21

Re: Systemd complete boot failure

Thanks cfr, I think that's fixed at least the X11 messages but they're still not showing up in the journal log either way. The system shows only the following in the log after I attempt to boot it.

-- Reboot --
May 28 03:34:12 aya systemd-journal[1962]: Allowing runtime journal files to grow to 1.1G.
May 28 03:34:12 aya systemd-journal[1962]: Journal started
May 28 03:34:12 aya systemd-fsck[2119]: /: clean, 55752/655360 files, 390255/2621440 blocks
May 28 03:34:14 aya systemd-journal[1962]: Allowing system journal files to grow to 1021.2M.
May 28 03:34:14 aya systemd-logind[3171]: New seat seat0.
May 28 03:40:40 aya systemd-journal[1962]: Journal stopped

It doesn't seem to progress past this and after roughly 10-15 minutes of no ping response or anything else of notable value, the system reboots into the recovery disk (it's in a datacentre). Any thoughts on how to proceed?

WonderWoofy

Member

From: Los Gatos, CA

Registered: 2012-05-19

Posts: 8,414

Re: Systemd complete boot failure

What flags did you add for debug level logging?  And what does /etc/systemd/journald.conf look like?

Emess

Member

From: Tokyo, Japan

Registered: 2008-01-20

Posts: 21

Re: Systemd complete boot failure

Added the following to my kernel boot parameters: systemd.log_level=debug systemd.unit=multi-user.target systemd.log_target=kmsg log_buf_len=1M

[root@rescue /]# cat /etc/systemd/journald.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=login
#SyncIntervalSec=5m
#RateLimitInterval=10s
#RateLimitBurst=200
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info

Looks like everything is commented out and just relies on defaults and kernel params, but that could be why I'm not getting anything useful.