Boot hangs on "sent stream file descriptor to Service Manager"[SOLVED]

schaetzc · 2017-10-17 20:18:15

Background Story

It all started with another problem a few days ago. During normal usage (browsing, reading eMails, ...) the system froze and the caps-lock and num-lock indicator lights started flashing. The system did not react to anything. Even ctrl+alt+f2 did not work. I forgot to try SysRQ. After a forced shutdown (pressing and holding power button for 10 seconds) the system booted normally.

Over time, the freeze happened multiple times. I thought the freeze might be related to my latest update, so I tried to update (pacman -Syu) a few days later. During the update, the system froze again and I had to force a shutdown again. Now the system did not boot anymore. There was only a black screen after the boot manager. Usually I see some console messages during boot, but even they did not show up.

There seems to be no problem with my hardware. The BIOS/UEFI on-board diagnostics did not find any problems and the dual booting windows system and a linux mint live system run flawless.

I used an linux mint live image to do the following:

Check filesystem with fsck — no problems detected
chroot and pacman -Syu
During upgrade, pacman showed warnings like "ldconfig: File /usr/lib/libQt... is empty, not checked." I ignored them and rebooted. Still only a black screen.
chroot again and force a re-install of the packages mentioned in the error messages (some qt5 packages and xf86-video-intel). Additionally, I reinstalled linux (because I though the freeze problem was related to the kernel) and systemd (because I thought that could help solving the boot problem). After a reboot, the boot started with the familar console messages, but did not finish. However, there was no warning or error.
chroot yet again and reinstall intel-ucode and create a new initramfs-linux.img. There were no signs that these could be the origin of the problem. I just tried it.
After rebooting, no changes. The boot started with the usual console messages and got stuck without a visible error.

Most of the time, the boot process hangs after one of these messages

[  O K  ] Started CUPS scheduler.
[  O K  ] Started Hostname Service.
[  O K  ] Started TLP system startup/shutdown.

The order of these messages changes. Most of the time TLP comes last and is the last message before the boot process hangs. I played a bit with the kernel parameters. Using the "debug" parameter during normal boot, I see that the system hangs in the following loop:

[...] systemd-journald[217] Successfully sent stream file descriptor to Service Manager.
[...] systemd-journald[217] Successfully sent stream file descriptor to Service Manager.
[...] systemd-journald[217] Successfully sent stream file descriptor to Service Manager.
[...] systemd-journald[217] Sent WATCHDOG=1 notification
[...] systemd-journald[217] Successfully sent stream file descriptor to Service Manager.
[...] systemd-journald[217] Successfully sent stream file descriptor to Service Manager.
[...] systemd-journald[217] Successfully sent stream file descriptor to Service Manager.
[...] systemd-journald[217] Sent WATCHDOG=1 notification
 
// .    .    .     ( and so on )
// In reality, the "sent stream" message is printed around 20 times
// between the "watchdog" messages.

I waited for 1000s. The loop is always the same.

Disabling intel-ucode or acpi did not change anything.

With the "rescue" parameter, the system boots into a shell without any visible problems. Checking dmesg afterwards doesn't show a problem (obviously, since the problem occurs only after the point where the rescue shell is started).

When the boot process hangs, I can open up a terminal using ctrl+alt+f2. Every three seconds or so the screen jumps back to the boot console, so I can't do much. I checked dmesg and dmesg -l err,warn but couldn't find any hints. Click the commands for pastebins.

Question

How can I restore my system? I think a full re-install would solve the problem, however I don't want to download and configure everything again. Apart from that, data loss is not a problem. I have a backup of my personal data.
Thank you in advance.

Data

If you need additional data/logs, please say so.

I have a laptop with arch linux and windows 10. It has an old Intel i5 CPU, one Samsung EVO SSD, and no dedicated graphics card.
The UEFI partition (FAT) is on /dev/sda2 and mounted under /boot.
The arch partition (ext4) is on /dev/sda6 and mounted under /.
The other partitions are related to the windows installation.
The bootmanager is systemd-boot. Arch has the following boot entry (which worked fine until the problem from above occured):

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options root=PARTUUID=7a520b43-6910-4a7a-ad8e-7aad97e50205 rw acpi_backlight=native

After successfully booting, the lxdm display manager should show up.

Last edited by schaetzc (2017-10-18 20:17:46)

seth · 2017-10-17 20:37:05

https://wiki.archlinux.org/index.php/Pa … an_upgrade

schaetzc · 2017-10-17 22:00:14

Thank you seth for helping.

The linked article did not help much. As I said, I already re-installed some packages and generated a new initramfs. However, I tried to follow the steps.

On "pacman -Syyu" I got the errors

error: failed to commit transaction (conflicting files)
systemd-sysvcompat: /usr/bin/halt exists in filesystem
systemd-sysvcompat: /usr/bin/init exists in filesystem
              .   .   .
Errors occurred, no packages were upgraded.

First I installed all packages but "systemd-sysvcompat", then I read this this wiki entry and used "pacman -S --force systemd-sysvcompat" to install the remaining package (--root and --cachedir are not needed due to chroot).

In the next step "find /usr/lib -size 0" listed more than 200 empty files:

/usr/lib/thunderbird/plugin-container
/usr/lib/thunderbird/libxul.so
. . .
/usr/lib/calibre/calibre/devices/usbms/__init__.py
/usr/lib/calibre/calibre/devices/irexdr/__init__.py
. . .
/usr/lib/python2.7/bsddb/test/__init__.py
/usr/lib/python2.7/test/nullcert.pem
. . .
/usr/lib/python3.6/tkinter/test/test_tkinter/__init__.py
/usr/lib/python3.6/test/nullcert.pem

I tried "pacman -S thunderbird" and got "exists in filesystem"-errors, so I used "pacman -S --force thunderbird". Afterwards "find /usr/lib -size 0" listed only 100 files. But I had no luck with calibre, python2/3, or any of the other files. "pacman -S (--force) package" did not reduce the number of empty files, even though there were no errors. One problem is, that I have to guess which path corresponds to which package. For the paths containing "calibre" I'm sure that the package has to be "calibre", nevertheless re-installing the package did not help.

What should I do next?

The boot process is still the same. No changes so far.

Last edited by schaetzc (2017-10-17 22:07:20)

loqs · 2017-10-17 22:59:12

If pacman -Qkk calibre or paccheck --quiet --sha256sum (requires pacutils) do not find any errors the file may be intended to be 0 in size.
Pacman#Search_for_a_package_that_contains_a_specific_file should help determine which package if any a file belongs to.
Would suggest disabling/removing TLP to see if that resolves the issue. Please post the journal from a boot with the issue.

schaetzc · 2017-10-18 09:20:53

»pacman -Qkk calibre« found no errors. »pacman -Qkk« showed some altered files for three packages, see output of pacman -Qkk | grep -P '[1-9]+0* altered files'.
I re-installed these packages with »pacman -S package«, but the number of altered files did not change. The boot process did not change.

I tried disabling some systemd services. Disabling TLP or CUPS did not change anything, so I enabled them again. Finally I disabled my login/display manager (lxdm). Now the boot succeeds! Since I can boot now, I don't think you need the boot journal anymore.

However, I cannot start X. I tried to start X using the »startx« command with the following ~/.xinitrc files

exec cinnamon-session

exec i3

In both cases, all I got was the output

/etc/X11/xinit/xserverrc: line 2: /usr/bin/X: Success
xinit: giving up
xinit: unable to connect to X server: Connection refused
xinit: server error

Thank you loqs! Now I made some progress. How should I proceed from here?
Since the problem is not different, should I change the title of this thread?

Last edited by schaetzc (2017-10-18 09:21:47)

seth · 2017-10-18 12:34:05

You should open a new thread (for the new problem) and attach your Xorg log there.
Also: https://wiki.archlinux.org/index.php/Xinit#xinitrc - read the second note.

schaetzc · 2017-10-18 20:16:54

I started investigating the Xorg problem but had no luck. Attempting to start X did not log anything. Re-installing xorg showed more "exists in filesystem" errors. With a forced re-install, X did at least some logging before crashing.

I give up. So far I spent more time trying to repair the system than I would have needed to completely re-install everything. I will just do a clean re-install :'(

Thank you for your time.

Last edited by schaetzc (2017-10-18 20:18:32)

Arch Linux

#1 2017-10-17 20:18:15