[SOLVED] Files/directories randomly disappearing in /etc

nezu · 2022-12-13 13:11:13

This is a bit of a weird one. I've noticed that some stuff randomly disappears from /etc.

First I noticed this some time ago when /etc/sudoers disappeared and i was confused as to why i suddenly can't suso. I just replaced it and continued using the system like normal.
The next time this happened was today when the entire /etc/wireguard disappeared (realized when i tried connecting to something behind a vpn). Now this is a bit more annoying since i had to re-generate keys for multiple tunnels.
There might be more that I haven't noticed yet.

Another thing that might or might not be related is that somewhere in a similar timeframe zsh would sometimes take a few seconds to load while consuming 100% cpu usage (I know that it can be slow to start with plugins but this didn't happen before not in the past on any of my other machines. This might or might not be related.

Some background info:
kernel: latest (6.0 rn)
Single btrfs partition for root + fat32 /boot + swap
the system has been sometimes shut down improperly but I would assume this would at most affect files that were not fully saved/were being modified (maybe i'm wrong, btrfs is relatively new to me)

I'm starting to get suspicious that this might not be an accident and something malicious is going on. I don't think pacman would ever touch a user modified file let alone delete it.

Have you guys ever seen anything like this before? Ideas?

Last edited by nezu (2022-12-14 11:01:40)

Trilby · 2022-12-13 14:24:06

Do you use any tools marketted as "system clean up" aids like bleachbit, or any configuration management tools like etckeeper?

nezu · 2022-12-13 14:26:04

nope. I manage almost everything myself. That's why i'm so confused as to what's happening. No funny symlinks or anything like that as well

frostschutz · 2022-12-13 15:20:04

The last time I had files randomly disappearing, it was systemd deleting them. RemoveIPC was active and I was experimenting with mountpoints and bind mounts under /dev/shm... which means systemd stupidly walks through all filesystems and deletes everything.

Don't use /dev/shm for anything, ever. It belongs to systemd now. If you want to experiment in a tmpfs, mount your own. And disable RemoveIPC anyway.

seth · 2022-12-13 15:24:10

the system has been sometimes shut down improperly

In chronological relation to the file loss?

I would assume

this would at most affect files that were not fully saved/were being modified

You haven't (necessarily) lost the data, but the inode links in the (corrupted) FS.
Alternative causes might be an overly aggressive trimming strategy (it's an SSD and you do use discard?)

If there's a pattern in the affected files, you could audit them to see whether some malicious userspace process deletes them, https://wiki.archlinux.org/title/Audit_framework ("unlink" syscall, likely)

Edit:

frostschutzu wrote:

systemd stupidly walks through all filesystems and deletes everything

"systemd-cleanupafterpulseaudio"…

Last edited by seth (2022-12-13 15:38:51)

Slithery · 2022-12-13 16:52:15

Run a SMART test on the drive and post the results.

nezu · 2022-12-13 16:56:08

In chronological relation to the file loss?

That's the funny part, no. Both times it happened the system was shut down properly and the files were there on the previous boot.

Alternative causes might be an overly aggressive trimming strategy (it's an SSD and you do use discard?)

hmm, is it possible that data loss like that can happen under normal operation or is that only in relation to other factors like sudden power loss?
I do have

rw,relatime,ssd,compress=zstd,discard=async

in my btrfs mount options

If there's a pattern in the affected files, you could audit them to see whether some malicious userspace process deletes them, https://wiki.archlinux.org/title/Audit_framework ("unlink" syscall, likely)

Thanks for that. Will set it up and see what i get

nezu · 2022-12-13 17:04:40

Slithery wrote:

Run a SMART test on the drive and post the results.

here: https://pastebin.com/aMGeqS4L

I'm pretty sure this is a logical/software issue and not a hardware one. And possibly data corruption due to user error, not hardware failure.

there is the "Unsafe Shutdowns: 193" but most of that was on my previous OS

Edit I take that back (partially). A good chunk of this is probably on this install but almost all are after the system has shut down. I have an issue with ACPI that's preventing the system from powering off properly after shutdown but at this point all filesystems are unmounted. Essentially where the power off should happen normally but doesn't.

ACPI issues is a topic for another day (and section) unless someone belies it's related.

Last edited by nezu (2022-12-13 17:10:58)

seth · 2022-12-13 17:12:06

https://wiki.archlinux.org/title/Solid_state_drive#TRIM

Warning: Users need to be certain that their SSD supports TRIM before attempting to use it. Data loss can occur otherwise!

Note: Continuous TRIM is not the most preferred way to issue TRIM commands among the Linux community. For example, Ubuntu enables periodic TRIM by default [7], Debian does not recommend using continuous TRIM and Red Hat recommends using periodic TRIM over using continuous TRIM if feasible [8].

The kernel seems to go for discard=async as default w/ 6.2, but still…

cfr · 2022-12-14 06:26:46

seth wrote:

The kernel seems to go for discard=async as default w/ 6.2, but still…

The wiki is adamant that continuous trim should not be used on nvme devices. See https://wiki.archlinux.org/title/Solid_ … e#Discards.

wiki wrote:

Warning: Although continuous TRIM is an option (albeit not recommended) for SSDs, NVMe devices should not be issued discards.
Discards are disabled by default on typical setups that use ext4 and LVM, but other file systems might need discards to be disabled explicitly.

Is 6.2 really going to issue that by default? If so, should it now be disabled explicitly regardless of filesystem?

In any case, the OP should definitely not be enabling it on such a drive.

Last edited by cfr (2022-12-14 06:32:03)

seth · 2022-12-14 08:03:58

https://www.phoronix.com/news/Btrfs-Asy … rd-Default

nezu · 2022-12-14 11:01:12

https://imgur.com/a/KkAE50W

Yeah, there is a lot of actual corruption here.

I'll just hack the system together for today and wipe it clean once I get home.

Well, I guess topic solved. Thanks for the info on TRIM as well. Might be a good idea to update the wiki if it's going to be the default soon.

frostschutz · 2022-12-14 11:08:48

that's quite weird, especially for btrfs ... run a memtest, too

cfr · 2022-12-14 16:58:34

seth wrote:

https://www.phoronix.com/news/Btrfs-Asy … rd-Default

Thanks, so it is still file system-dependent, but btrfs is moving into the class of file systems for which it should be explicitly disabled.

Unless, of course, the warning in the wiki is outdated, but that seems unlikely?

Last edited by cfr (2022-12-14 16:58:57)

seth · 2022-12-14 20:23:23

The continuous trim isn't advised anyway and on top of that one needs to make sure that the SSD/nvme actually supports the feature (well)
idk. what the patch is going to be about, "mounted with a capable solid-state drive" hopefully has some reasonable checks included.
But "Facebook/Meta engineers" advocate it, so it's certainly gonna be as much of a blessing as facebook and a smashing success like the metaverse

Arch Linux

#1 2022-12-13 13:11:13

[SOLVED] Files/directories randomly disappearing in /etc

#2 2022-12-13 14:24:06

Re: [SOLVED] Files/directories randomly disappearing in /etc

#3 2022-12-13 14:26:04

Re: [SOLVED] Files/directories randomly disappearing in /etc

#4 2022-12-13 15:20:04

Re: [SOLVED] Files/directories randomly disappearing in /etc

#5 2022-12-13 15:24:10

Re: [SOLVED] Files/directories randomly disappearing in /etc

#6 2022-12-13 16:52:15

Re: [SOLVED] Files/directories randomly disappearing in /etc

#7 2022-12-13 16:56:08

Re: [SOLVED] Files/directories randomly disappearing in /etc

#8 2022-12-13 17:04:40

Re: [SOLVED] Files/directories randomly disappearing in /etc

#9 2022-12-13 17:12:06

Re: [SOLVED] Files/directories randomly disappearing in /etc

#10 2022-12-14 06:26:46

Re: [SOLVED] Files/directories randomly disappearing in /etc

#11 2022-12-14 08:03:58

Re: [SOLVED] Files/directories randomly disappearing in /etc

#12 2022-12-14 11:01:12

Re: [SOLVED] Files/directories randomly disappearing in /etc

#13 2022-12-14 11:08:48

Re: [SOLVED] Files/directories randomly disappearing in /etc

#14 2022-12-14 16:58:34

Re: [SOLVED] Files/directories randomly disappearing in /etc

#15 2022-12-14 20:23:23

Re: [SOLVED] Files/directories randomly disappearing in /etc

Board footer