You are not logged in.

#1 2022-12-13 13:11:13

nezu
Member
From: Poland
Registered: 2022-12-13
Posts: 5
Website

[SOLVED] Files/directories randomly disappearing in /etc

This is a bit of a weird one. I've noticed that some stuff randomly disappears from /etc.

First I noticed this some time ago when /etc/sudoers disappeared and i was confused as to why i suddenly can't suso. I just replaced it and continued using the system like normal.
The next time this happened was today when the entire /etc/wireguard disappeared (realized when i tried connecting to something behind a vpn). Now this is a bit more annoying since i had to re-generate keys for multiple tunnels.
There might be more that I haven't noticed yet.

Another thing that might or might not be related is that somewhere in a similar timeframe zsh would sometimes take a few seconds to load while consuming 100% cpu usage (I know that it can be slow to start with plugins but this didn't happen before not in the past on any of my other machines. This might or might not be related.

Some background info:
kernel: latest (6.0 rn)
Single btrfs partition for root + fat32 /boot + swap
the system has been sometimes shut down improperly but I would assume this would at most affect files that were not fully saved/were being modified (maybe i'm wrong, btrfs is relatively new to me)

I'm starting to get suspicious that this might not be an accident and something malicious is going on. I don't think pacman would ever touch a user modified file let alone delete it.

Have you guys ever seen anything like this before? Ideas?

Last edited by nezu (2022-12-14 11:01:40)

Offline

#2 2022-12-13 14:24:06

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,111
Website

Re: [SOLVED] Files/directories randomly disappearing in /etc

Do you use any tools marketted as "system clean up" aids like bleachbit, or any configuration management tools like etckeeper?


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2022-12-13 14:26:04

nezu
Member
From: Poland
Registered: 2022-12-13
Posts: 5
Website

Re: [SOLVED] Files/directories randomly disappearing in /etc

nope. I manage almost everything myself. That's why i'm so confused as to what's happening. No funny symlinks or anything like that as well

Offline

#4 2022-12-13 15:20:04

frostschutz
Member
Registered: 2013-11-15
Posts: 1,355

Re: [SOLVED] Files/directories randomly disappearing in /etc

The last time I had files randomly disappearing, it was systemd deleting them. RemoveIPC was active and I was experimenting with mountpoints and bind mounts under /dev/shm... which means systemd stupidly walks through all filesystems and deletes everything.

Don't use /dev/shm for anything, ever. It belongs to systemd now. If you want to experiment in a tmpfs, mount your own. And disable RemoveIPC anyway.

Online

#5 2022-12-13 15:24:10

seth
Member
Registered: 2012-09-03
Posts: 45,436

Re: [SOLVED] Files/directories randomly disappearing in /etc

the system has been sometimes shut down improperly

In chronological relation to the file loss?

I would assume

e28dffa3c5396b5064779e3f589cf255.png

this would at most affect files that were not fully saved/were being modified

You haven't (necessarily) lost the data, but the inode links in the (corrupted) FS.
Alternative causes might be an overly aggressive trimming strategy (it's an SSD and you do use discard?)

If there's a pattern in the affected files, you could audit them to see whether some malicious userspace process deletes them, https://wiki.archlinux.org/title/Audit_framework ("unlink" syscall, likely)

Edit:

frostschutzu wrote:

systemd stupidly walks through all filesystems and deletes everything

"systemd-cleanupafterpulseaudio"…

Last edited by seth (2022-12-13 15:38:51)

Offline

#6 2022-12-13 16:52:15

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: [SOLVED] Files/directories randomly disappearing in /etc

Run a SMART test on the drive and post the results.


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#7 2022-12-13 16:56:08

nezu
Member
From: Poland
Registered: 2022-12-13
Posts: 5
Website

Re: [SOLVED] Files/directories randomly disappearing in /etc

In chronological relation to the file loss?

That's the funny part, no. Both times it happened the system was shut down properly and the files were there on the previous boot.

Alternative causes might be an overly aggressive trimming strategy (it's an SSD and you do use discard?)

hmm, is it possible that data loss like that can happen under normal operation or is that only in relation to other factors like sudden power loss?
I do have

rw,relatime,ssd,compress=zstd,discard=async

in my btrfs mount options

If there's a pattern in the affected files, you could audit them to see whether some malicious userspace process deletes them, https://wiki.archlinux.org/title/Audit_framework ("unlink" syscall, likely)

Thanks for that. Will set it up and see what i get

Offline

#8 2022-12-13 17:04:40

nezu
Member
From: Poland
Registered: 2022-12-13
Posts: 5
Website

Re: [SOLVED] Files/directories randomly disappearing in /etc

Slithery wrote:

Run a SMART test on the drive and post the results.

here: https://pastebin.com/aMGeqS4L

I'm pretty sure this is a logical/software issue and not a hardware one. And possibly data corruption due to user error, not hardware failure.

there is the "Unsafe Shutdowns: 193" but most of that was on my previous OS

Edit I take that back (partially). A good chunk of this is probably on this install but almost all are after the system has shut down. I have an issue with ACPI that's preventing the system from powering off properly after shutdown but at this point all filesystems are unmounted. Essentially where the power off should happen normally but doesn't.

ACPI issues is a topic for another day (and section) unless someone belies it's related.

Last edited by nezu (2022-12-13 17:10:58)

Offline

#9 2022-12-13 17:12:06

seth
Member
Registered: 2012-09-03
Posts: 45,436

Re: [SOLVED] Files/directories randomly disappearing in /etc

https://wiki.archlinux.org/title/Solid_state_drive#TRIM

Warning: Users need to be certain that their SSD supports TRIM before attempting to use it. Data loss can occur otherwise!

Note: Continuous TRIM is not the most preferred way to issue TRIM commands among the Linux community. For example, Ubuntu enables periodic TRIM by default [7], Debian does not recommend using continuous TRIM and Red Hat recommends using periodic TRIM over using continuous TRIM if feasible [8].

The kernel seems to go for discard=async as default w/ 6.2, but still…

Offline

#10 2022-12-14 06:26:46

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,129

Re: [SOLVED] Files/directories randomly disappearing in /etc

seth wrote:

The kernel seems to go for discard=async as default w/ 6.2, but still…

The wiki is adamant that continuous trim should not be used on nvme devices. See https://wiki.archlinux.org/title/Solid_ … e#Discards.

wiki wrote:

Warning: Although continuous TRIM is an option (albeit not recommended) for SSDs, NVMe devices should not be issued discards.
Discards are disabled by default on typical setups that use ext4 and LVM, but other file systems might need discards to be disabled explicitly.

Is 6.2 really going to issue that by default? If so, should it now be disabled explicitly regardless of filesystem?

In any case, the OP should definitely not be enabling it on such a drive.

Last edited by cfr (2022-12-14 06:32:03)


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#11 2022-12-14 08:03:58

seth
Member
Registered: 2012-09-03
Posts: 45,436

Offline

#12 2022-12-14 11:01:12

nezu
Member
From: Poland
Registered: 2022-12-13
Posts: 5
Website

Re: [SOLVED] Files/directories randomly disappearing in /etc

https://imgur.com/a/KkAE50W

Yeah, there is a lot of actual corruption here.

I'll just hack the system together for today and wipe it clean once I get home.

Well, I guess topic solved. Thanks for the info on TRIM as well. Might be a good idea to update the wiki if it's going to be the default soon.

Offline

#13 2022-12-14 11:08:48

frostschutz
Member
Registered: 2013-11-15
Posts: 1,355

Re: [SOLVED] Files/directories randomly disappearing in /etc

that's quite weird, especially for btrfs ... run a memtest, too

Online

#14 2022-12-14 16:58:34

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,129

Re: [SOLVED] Files/directories randomly disappearing in /etc

Thanks, so it is still file system-dependent, but btrfs is moving into the class of file systems for which it should be explicitly disabled.

Unless, of course, the warning in the wiki is outdated, but that seems unlikely?

Last edited by cfr (2022-12-14 16:58:57)


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#15 2022-12-14 20:23:23

seth
Member
Registered: 2012-09-03
Posts: 45,436

Re: [SOLVED] Files/directories randomly disappearing in /etc

The continuous trim isn't advised anyway and on top of that one needs to make sure that the SSD/nvme actually supports the feature (well)
idk. what the patch is going to be about, "mounted with a capable solid-state drive" hopefully has some reasonable checks included.
But "Facebook/Meta engineers" advocate it, so it's certainly gonna be as much of a blessing as facebook and a smashing success like the metaverse tongue

Offline

Board footer

Powered by FluxBB