You are not logged in.
TLDR: I fought a wild rm. I believe I have not lost anything important, but now I am seeing units failing at boot. I am not sure whether that is normal.
---
I accidentally ran something like `rm -fr /*`. I interrupted it after a couple of seconds. Now I am trying to assess the casualties.
Thankfully, I ran it as a normal user, and most parts of my home directory have backups. So I only need to figure out the missing files among:
- directories to which I have write access in the system directories;
- directories in the parts of my home which are not backup-ed.
and restrict the search to recently modified directories (because removing a file is modifying its parent directory).
So I searched for all recently modified directories:
find / -type d -printf '%T+\t%p\0' |sort -zrn |tr '\0' '\n > recently-written-dirs.txtI did not find much. If my method is to be believed, most deletions have been absorbed by my web browser’s cache and another directory with a lot of inodes. Well, if that is true, I am lucky.
However I prefer to double-check. I cannot do much more for the 2nd point, but for the 1st point I identified all directories to which I could write, with:
find / -path ~ -prune -o -type d -writable -printf '%T+\t%p\0' -prune |sort -zrn |tr '\0' '\n' > writable-dirs.txtThe only ones with recent changes were these:
/tmp
/var/tmp
/dev/mqueue
/dev/shm
/run/user/1000
/sys/fs/cgroup/systemd/user.slice/user-1000.slice/user@1000.service
/sys/fs/cgroup/unified/user.slice/user-1000.slice/user@1000.serviceUnless I am mistaken, those are all temporary files, and although altering them may leave my computer in a strange state (and indeed, Vim and other programs started to behave strangely) it should come back to normal after a reboot.
I also checked the integrity of all my packages. `sudo pacman -Qk` reported no missing file, and `sudo pacman -Qkk` reported no modified file (excepted the expected ones, such as `/etc/fstab`).
Then I rebooted. Everything appears to work, but at boot I noticed that several systemd units failed to start. I do not remember that it was the case before. The ones that I can still read on the screen are the following:
Load Kernel Modules / systemd-modules-load.service
FUSE Control File System / sys-fs-fuse-connections.mount
Rebuild Dynamic Linker Cache / ldconfig.service
Rebuild Journal Catalog / systemd-journal-catalog-update.service
Update is Completed / systemd-update-done.service
First Boot Wizard / systemd-firstboot.service
Commit a transcient machine-id on disk / systemd-machine-id-commit.service
Create System Users / systemd-sysusers.service
Rebuild Hardware Database / systemd-hwdb-update.service
Set Up Additional Binary Formats / systemd-binfmt.serviceAccording to `systemctl status`, all of them fail because some condition is not met. For example:
● systemd-modules-load.service - Load Kernel Modules
Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Sun 2019-09-22 00:48:59 CEST; 11min ago
├─ ConditionDirectoryNotEmpty=|/lib/modules-load.d was not met
├─ ConditionDirectoryNotEmpty=|/usr/lib/modules-load.d was not met
├─ ConditionDirectoryNotEmpty=|/usr/local/lib/modules-load.d was not met
├─ ConditionDirectoryNotEmpty=|/etc/modules-load.d was not met
├─ ConditionDirectoryNotEmpty=|/run/modules-load.d was not met
├─ ConditionKernelCommandLine=|modules-load was not met
└─ ConditionKernelCommandLine=|rd.modules-load was not met
● sys-fs-fuse-connections.mount - FUSE Control File System
Loaded: loaded (/usr/lib/systemd/system/sys-fs-fuse-connections.mount; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Sun 2019-09-22 00:48:59 CEST; 12min ago
└─ ConditionPathExists=/sys/fs/fuse/connections was not met
● ldconfig.service - Rebuild Dynamic Linker Cache
Loaded: loaded (/usr/lib/systemd/system/ldconfig.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Sun 2019-09-22 00:48:59 CEST; 13min ago
├─ ConditionNeedsUpdate=|/etc was not met
└─ ConditionFileNotEmpty=|!/etc/ld.so.cache was not met
● systemd-update-done.service - Update is Completed
Loaded: loaded (/usr/lib/systemd/system/systemd-update-done.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Sun 2019-09-22 00:48:59 CEST; 28min ago
├─ ConditionNeedsUpdate=|/etc was not met
└─ ConditionNeedsUpdate=|/var was not met
● systemd-firstboot.service - First Boot Wizard
Loaded: loaded (/usr/lib/systemd/system/systemd-firstboot.service; static; vendor preset: disabled)
Active: inactive (dead)
Condition: start condition failed at Sun 2019-09-22 00:48:59 CEST; 29min ago
└─ ConditionFirstBoot=yes was not metMany —but not all— of these conditions have to do with existence of some file. I do not remember having seen these units failing at boot before, but I might be mistaken. So, is this normal? If not, should I care, and can I do something about it?
Offline
The output from systemctl status does not show any of the services to be in a failed state.
What is the output of `systemctl --failed`
Offline
You are right:
$ systemctl --failed
0 loaded units listed.It’s then confusing that the message at boot says “[FAILED] Failed to start XXX.”
So you’re saying that everything is okay?
Offline
Does the journal contain any information on what failed?
Offline
The last entry of `journalctl` (without options) predates the incident by one day. The journal does not even mention that I rebooted since (twice), so maybe something is wrong here (one of the failed services is precisely about the journal, "systemd-journal-catalog-update.service").
Offline
What is the status of the journal service?
systemctl status systemd-journald.servicePerhaps the system has switched its machine-id and is not showing journal entries from the new machine-id.
Does the following show entries from the current boot?
# journalctl -meOffline
`journalctl -me` only shows events from 10 days ago, up to the last reboot before the incident.
`systemctl status systemd-journald.service` reports the following (sept. 22 00:48 is my 2nd reboot since the incident; 19dcadbedb314b7a80e78e5f319ab473 is the current contents of /etc/machine-id, which has been untouched since installation):
● systemd-journald.service - Journal Service
Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
Active: active (running) since Sun 2019-09-22 00:48:58 CEST; 1 day 12h ago
Docs: man:systemd-journald.service(8)
man:journald.conf(5)
Main PID: 467 (systemd-journal)
Status: "Processing requests..."
Tasks: 1 (limit: 4915)
Memory: 28.4M
CGroup: /system.slice/systemd-journald.service
└─467 /usr/lib/systemd/systemd-journald
sept. 22 00:48:58 mymachine systemd-journald[467]: Journal started
sept. 22 00:48:58 mymachine systemd-journald[467]: Runtime journal (/run/log/journal/19dcadbedb314b7a80e78e5f319ab473) is 8.0M, max 795.9M, 787.9M free.
sept. 22 00:48:58 mymachine systemd-journald[467]: Time spent on flushing to /var is 66.900ms for 674 entries.
sept. 22 00:48:58 mymachine systemd-journald[467]: System journal (/var/log/journal/19dcadbedb314b7a80e78e5f319ab473) is 680.0M, max 3.1G, 2.4G free.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.I also had a look at what is in /var/log/journal:
$ ls -FlAhtR /var/log/journal/
/var/log/journal/:
total 16K
drwxr-sr-x+ 2 root systemd-journal 4.0K Sep 20 11:00 19dcadbedb314b7a80e78e5f319ab473/
drwxr-sr-x+ 2 root systemd-journal-remote 4.0K Jul 21 15:55 remote/
/var/log/journal/19dcadbedb314b7a80e78e5f319ab473:
total 689M
-rw-r-----+ 1 root systemd-journal 8.0M Sep 21 2029 'system@56ed22d536ac4293b6c5d90de5caccb6-000000000001746e-0005920af8dbd8ac.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 17 2029 'system@56ed22d536ac4293b6c5d90de5caccb6-000000000001749e-0006b211034f9995.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 9 2029 'system@56ed22d536ac4293b6c5d90de5caccb6-00000000000174c0-0005920b058609bf.journal'
-rw-r-----+ 1 root systemd-journal 24M Sep 23 16:27 system.journal
-rw-r-----+ 1 root systemd-journal 8.0M Sep 23 15:32 user-1000.journal
-rw-r-----+ 1 root systemd-journal 88M Sep 20 11:00 'system@c0b212a3782849ad8ac1b263df1a2adc-0000000000000001-000592459566ebaf.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 20 09:49 'user-1000@c8a51e55bc4f41e2bc3d7faa19da5ff4-0000000000019b45-000592346c88593c.journal'
-rw-r-----+ 1 root systemd-journal 24M Sep 11 13:47 'system@0005924595674c9b-cd87e09d7ecc81a6.journal~'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 9 23:57 'system@56ed22d536ac4293b6c5d90de5caccb6-00000000000174e0-00059225de07044f.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 9 23:57 'system@56ed22d536ac4293b6c5d90de5caccb6-00000000000174d1-0006b133b053e45f.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 8 16:03 'system@56ed22d536ac4293b6c5d90de5caccb6-00000000000174e3-00059225de07045f.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 8 15:55 'system@56ed22d536ac4293b6c5d90de5caccb6-00000000000174ad-0006b1d49f0ee43b.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 8 15:51 'system@56ed22d536ac4293b6c5d90de5caccb6-0000000000017468-0005920af5735a55.journal'
-rw-r-----+ 1 root systemd-journal 88M Sep 8 15:50 'system@56ed22d536ac4293b6c5d90de5caccb6-0000000000000001-0005913353ef3186.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Sep 8 12:51 'user-1000@c8a51e55bc4f41e2bc3d7faa19da5ff4-0000000000000423-0005911fd7976bd8.journal'
-rw-r-----+ 1 root systemd-journal 16M Aug 28 22:35 'system@0005913353ef9465-20f07cf752f336bd.journal~'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 27 23:20 'user-1000@0005911fd7977d57-02fa687ccf7e3833.journal~'
-rw-r-----+ 1 root systemd-journal 72M Aug 27 23:20 'system@0005911fd5dfce06-95e8bd2ba7860384.journal~'
-rw-r-----+ 1 root systemd-journal 72M Aug 12 13:53 'system@00058fea2d23ed15-d568287c449db0ac.journal~'
-rw-r-----+ 1 root systemd-journal 24M Aug 5 23:42 'system@00058f6594000eae-4e856b1c647ce621.journal~'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 5 10:59 'user-1000@d7728633c9104990b629a85ce69c44b3-0000000000021fc9-00057e45423c26a2.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 5 10:59 'system@f733d250d8e34dadae60b070c31438ae-0000000000021fc8-00057e45417b5d84.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 4 18:08 'user-1000@d7728633c9104990b629a85ce69c44b3-0000000000021872-00058f3acf2d07ae.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 3 20:27 'system@f733d250d8e34dadae60b070c31438ae-0000000000020eab-00058f3a23492fc3.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 3 20:25 'user-1000@d7728633c9104990b629a85ce69c44b3-00000000000212b2-00058f3a256a356d.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Aug 3 19:52 'user-1000@d7728633c9104990b629a85ce69c44b3-000000000000350b-00058e5b94e4c1ed.journal'
-rw-r-----+ 1 root systemd-journal 104M Aug 3 19:52 'system@f733d250d8e34dadae60b070c31438ae-0000000000002f43-00058e58c1895fb6.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Jul 23 15:42 'user-1000@d7728633c9104990b629a85ce69c44b3-0000000000000391-00058e31d931970e.journal'
-rw-r-----+ 1 root systemd-journal 16M Jul 23 14:58 'system@f733d250d8e34dadae60b070c31438ae-0000000000000001-00058e31d7dacfde.journal'
-rw-r-----+ 1 root systemd-journal 8.0M Dec 31 2018 'system@f733d250d8e34dadae60b070c31438ae-000000000002185c-00058f3aa17c62d1.journal'
/var/log/journal/remote:
total 0I see nothing suspect. Some files are dated from 2029 because some 14 days ago, I set the date to 2029 by mistake, but that shouldn’t cause any problems (I can see that the messages from 2029 are interleaved with the others in the actual order of events) (according to `find / -newermt 2019-09-24`, no other file is affected).
The journal catalog is not enlightening to me:
$ journalctl --list-catalog
0027229ca0644181a76c4e92458afa2e systemd: One or more messages could not be forwarded to syslog
1675d7f172174098b1108bf8c7dc8f5d systemd: DNSSEC validation failed
1dee0369c7fc4736b7099b38ecb46ee7 systemd: Mount point is not empty
24d8d4452573402496068381a6312df2 systemd: A virtual machine or container has been started
3354939424b4456d9802ca8333ed424a systemd: Session @SESSION_ID@ has been terminated
36db2dfa5a9045e1bd4af5f93e1cf057 systemd: DNSSEC mode has been turned off, as server doesn't support it
39f53479d3a045ac8e11786248231fbf systemd: A start job for unit @UNIT@ has finished successfully
45f82f4aef7a4bbf942ce861d1f20990 systemd: Time zone change to @TIMEZONE@
4d4408cfd0d144859184d1e65d7c8a65 systemd: A DNSSEC trust anchor has been revoked
50876a9db00f4c40bde1a2ad381c3a1b systemd: The system is configured in a way that might cause problems
58432bd3bace477cb514b56381b8a758 systemd: A virtual machine or container has been terminated
5aadd8e954dc4b1a8c954d63fd9e1137 systemd: Core file was truncated to @SIZE_LIMIT@ bytes.
5eb03494b6584870a536b337290809b3 systemd: Automatic restarting of a unit has been scheduled
641257651c1b4ec9a8624d7a40a9e1e7 systemd: Process @EXECUTABLE@ could not be executed
6bbd95ee977941e497c48be27c254128 systemd: System sleep state @SLEEP@ entered
7ad2d189f7e94e70a38c781354912448 systemd: Unit succeeded
7b05ebc668384222baa8881179cfda54 systemd: A reload job for unit @UNIT@ has finished
7d4958e842da4a758f6c1cdc7b36dcc5 systemd: A start job for unit @UNIT@ has begun execution
8811e6df2a8e40f58a94cea26f8ebf14 systemd: System sleep state @SLEEP@ left
8d45620c1a4348dbb17410da57c60c66 systemd: A new session @SESSION_ID@ has been created for user @USER_ID@
98268866d1d54a499c4e98921d93bc40 systemd: System shutdown initiated
98e322203f7a4ed290d09fe03c09fe15 systemd: Unit process exited
9d1aaa27d60140bd96365438aad20286 systemd: A stop job for unit @UNIT@ has finished
a596d6fe7bfa4994828e72309e95d61e systemd: Messages from a service have been suppressed
ae8f7b866b0347b9af31fe1c80b127c0 systemd: Resources consumed by unit runtime
b07a249cd024414a82dd00cd181378ff systemd: System start-up is now complete
be02cf6855d2428ba40df7e9d022f03d systemd: A start job for unit @UNIT@ has failed
c7a787079b354eaaa9e77b371893cd27 systemd: Time change
d34d037fff1847e6ae669a370e694725 systemd: A reload job for unit @UNIT@ has begun execution
d93fb3c9c24d451a97cea615ce59c00b systemd: The journal has been stopped
d9b373ed55a64feb8242e02dbe79a49c systemd: Unit failed
de5b426a63be47a7b6ac3eaac82e2f6f systemd: A stop job for unit @UNIT@ has begun execution
e7852bfe46784ed0accde04bc864c2d5 systemd: Seat @SEAT_ID@ has now been removed
e9bf28e6e834481bb6f48f548ad13606 systemd: Journal messages have been missed
ec387f577b844b8fa948f33cad9a75e6 systemd: Disk space used by the journal
eed00a68ffd84e31882105fd973abdd1 systemd: User manager start-up is now complete
f77379a8490b408bbe5f6940505a777b systemd: The journal has been started
fc2e22bc6ee647b6b90729ab34a250b1 systemd: Process @COREDUMP_PID@ (@COREDUMP_COMM@) dumped core
fcbefc5da23d428093f97c82a9290f7b systemd: A new seat @SEAT_ID@ is now availableAnd running `sudo journalctl --update -catalog` did not change the output of previous commands.
Offline