[SOLVED] Booting fails with various errors (deleted /usr/ by accident)

mclang · 2017-09-07 11:15:30

Yesterday my computer rebooted itself spontaneously.

Since then the booting process halts in "Failed to start Network Manager" error. In addition, there is also two other errors before that, "Failed to start Create Volatile Files and Directories" and "Failed to start Login Service". I booted my computer using newest Arch Install USB, used `arch-chroot` to chroot into the root partition of my Arch installation and tried to dig deeper with `journalctl` as the error messages adviced.

Here is what I found:

-- Logs begin at Tue 2016-03-29 17:06:35 UTC, end at Thu 2017-09-07 07:48:30 UTC. --
Sep 07 07:36:05 WammerIIC systemd[1]: Starting Create Volatile Files and Directories...
Sep 07 07:36:05 WammerIIC systemd-tmpfiles[391]: Failed to copy files to /etc/pam.d: No such file or directory
Sep 07 07:36:05 WammerIIC systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Sep 07 07:36:05 WammerIIC systemd[1]: Failed to start Create Volatile Files and Directories.
Sep 07 07:36:05 WammerIIC systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Sep 07 07:36:05 WammerIIC systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.

-- Logs begin at Tue 2016-03-29 17:06:35 UTC, end at Thu 2017-09-07 07:48:30 UTC. --
Sep 07 07:36:09 WammerIIC systemd[1]: Starting Login Service...
Sep 07 07:36:34 WammerIIC systemd-logind[408]: Failed to add match for JobRemoved: Connection timed out
Sep 07 07:36:34 WammerIIC systemd-logind[408]: Failed to fully start up daemon: Connection timed out
Sep 07 07:36:34 WammerIIC systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE
Sep 07 07:36:34 WammerIIC systemd[1]: Failed to start Login Service.
Sep 07 07:36:34 WammerIIC systemd[1]: systemd-logind.service: Unit entered failed state.
Sep 07 07:36:34 WammerIIC systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Sep 07 07:36:34 WammerIIC systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Sep 07 07:48:08 WammerIIC systemd[1]: Stopped Login Service.

-- Logs begin at Tue 2016-03-29 17:06:35 UTC, end at Thu 2017-09-07 07:48:30 UTC. --
Sep 07 07:36:34 WammerIIC systemd[1]: Starting Network Manager...
Sep 07 07:36:34 WammerIIC NetworkManager[422]: <info>  [1504769794.0979] NetworkManager (version 1.8.3dev+18+gf43222d41-1, Arch Linux) is starting... (for the first time)
Sep 07 07:36:34 WammerIIC NetworkManager[422]: <info>  [1504769794.0980] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 20-connectivity.conf)
Sep 07 07:38:04 WammerIIC systemd[1]: NetworkManager.service: Start operation timed out. Terminating.
Sep 07 07:39:34 WammerIIC systemd[1]: NetworkManager.service: State 'stop-sigterm' timed out. Killing.
Sep 07 07:39:34 WammerIIC systemd[1]: NetworkManager.service: Killing process 422 (NetworkManager) with signal SIGKILL.
Sep 07 07:39:34 WammerIIC systemd[1]: NetworkManager.service: Main process exited, code=killed, status=9/KILL
Sep 07 07:39:34 WammerIIC systemd[1]: Failed to start Network Manager.
Sep 07 07:39:34 WammerIIC systemd[1]: NetworkManager.service: Unit entered failed state.
Sep 07 07:39:34 WammerIIC systemd[1]: NetworkManager.service: Failed with result 'timeout'.
Sep 07 07:39:35 WammerIIC systemd[1]: NetworkManager.service: Service hold-off time over, scheduling restart.
Sep 07 07:48:08 WammerIIC systemd[1]: Stopped Network Manager.

The whole journal of that failed boot can be viewed here:
https://pastebin.com/rLdNfJcC

I'm normally running kernel-ck, but the same thing happens also with normal Arch kernel.

Does anybody have ideas what to try to get this sorted out?

Last edited by mclang (2017-09-09 19:17:18)

jasonwryan · 2017-09-07 17:27:46

Sep 07 07:36:05 WammerIIC systemd-tmpfiles[391]: Failed to copy files to /etc/pam.d: No such file or directory

mclang · 2017-09-07 19:25:59

Sorry, forgot to mention that checking `/etc/pam.d/` was the first thing I did after reading those errors.

It exists indeed and contains various files that seems to (can't be sure) belong there. And they all are readable and writable by root.

jasonwryan · 2017-09-07 19:41:08

That's not the only error about not finding required directories. Could be some sort of BTRFS failure.

mclang · 2017-09-08 14:17:05

Hmm. that may well be the case

I read about ACL problems with `systemd-tmpfiles-setup` when using btrfs, but I'm using defaults option in fstab, so I think that is not the problem. In addition, those problems were two years back, and I haven't had this kind of issues before.

My root partition is simple btrfs volume without snapshots or other advanced things if I remember right. I can paste my mount options and btrfs information if it is of any help. And I definitely read more about btrfs now, maybe something pops up!

I wonder though why mounting using Arch Install USB works, but normal boot fails... Can it be that some thing in installed kernel, initramfs or related file is corrupted? But how come BOTH my kernels, the normal Arch one and kernel-ck has same boot problem?

Is the amount of these messages normal:

systemd[1]: Looping too fast. Throttling execution a little.

Luckily I have enabled persistent journal, so maybe I find something when I look back when my computer suddenly rebooted.

Last edited by mclang (2017-09-08 14:28:21)

Blasphemist · 2017-09-08 20:01:57

https://wiki.archlinux.org/index.php/Sy … rt_at_boot
I doubt that this is the whole problem but it should be eliminated just the same.

mclang · 2017-09-08 20:37:27

Blasphemist wrote:

https://wiki.archlinux.org/index.php/Sy … rt_at_boot
I doubt that this is the whole problem but it should be eliminated just the same.

That is just where I read about one possible reason to "Failed to start Create Volatile Files and Directories" is ACL, after which I found out that btrfs uses them by default:
https://wiki.archlinux.org/index.php/Ac … abling_ACL

I tried to fix thing by doing `btrfsck --repair`, `btrfs scrub start /` and `btrfs filesystem defragment -r -v -clzo /` but the problem remains.I'm out of ideas what to try

Last edited by mclang (2017-09-08 20:52:38)

loqs · 2017-09-08 20:52:13

mclang wrote:

Is the amount of these messages normal:
systemd[1]: Looping too fast. Throttling execution a little.

No having any of those messages in the journal is not normal

Sep 07 07:36:09 WammerIIC alsactl[406]: /usr/bin/alsactl: sysfs_init:48: sysfs path '/sys' is invalid

Is /sys mounted

Sep 07 07:36:09 WammerIIC dbus-daemon[412]: Failed to start message bus: Failed to open "/usr/share/dbus-1/system.conf": No such file or directory

Does /usr/share/dbus-1/system.conf exist?
I would suggest to check for missing or altered files tracked by pacman with

# pacman -Qkk

mclang · 2017-09-08 21:13:21

I `arch-chroot`:ed back inside the root partition of my Arch installation.

At least doing it this way the '/sys' directory exists and contains directories like 'block', 'bus', 'class', ... 'kernel', etc.

But file '/usr/share/dbus-1/system.conf' does NOT exist, and `pacman -Qkk` reports a whole LOT of missing files and directories

What the heck has happened? Can I try to restore all missing system files by installing packages one by one, or how should I proceed?

loqs · 2017-09-08 21:42:52

I would first check the filesystem and the device itself for errors. Then yes resintalling the packages that provide the missing files should restore them in most cases ( I am unsure if files that are marked as backup by pacman will be )

mclang · 2017-09-09 19:15:49

Problem solved!

First I checked my SSD drive with Kingston SSD Manager, no problems there.

Next I reinstalled all native packages using command found from Pacman wiki:

 pacman -Qnq | pacman -S -

Then I rebooted, biting my fingernails... And look and behold, after a bit of wait LightDM came up!

Now I must still reinstall all foreign AUR packages before things are normal. And maybe I do some system maintenance in same go, i.e delete orphaned packages and clear pacman cache, etc. After I make btrfs snapshot that is.

BTW, the reason of my problems was some kind of SNAFU with copying and pasting delete command... How do I know? I found this in my bash history:

  998  2017-09-06 14:10:53 - sudo rm /usr/share/fonts/font-awesome-4.7.0/
  999  2017-09-06 14:11:15 - sudo rm /usr/ -rf /usr/share/fonts/font-awesome-4.7.0/

The first is what I did, the second came from somewhere... Maybe misplaced keycombination while learning i3, or something.

Arch Linux

#1 2017-09-07 11:15:30

[SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#2 2017-09-07 17:27:46

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#3 2017-09-07 19:25:59

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#4 2017-09-07 19:41:08

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#5 2017-09-08 14:17:05

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#6 2017-09-08 20:01:57

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#7 2017-09-08 20:37:27

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#8 2017-09-08 20:52:13

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#9 2017-09-08 21:13:21

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#10 2017-09-08 21:42:52

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

#11 2017-09-09 19:15:49

Re: [SOLVED] Booting fails with various errors (deleted /usr/ by accident)

Board footer