Arch Linux

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

[SOLVED] systemd-logind suddenly deletes all files

So, today on my ArchLinux server, one of my user's files vanished - all several gigabytes of them - and it's gone, poof (Southpark moment).

Problem is, I currently fail to understand why. It happened right after a rsync backup task ran over SSH. So it backed up the files, and it's gone, poof.

So I started digging around and the only related thing I found is one single message in my logs:

Sep 15 19:22:46 VPS-arch systemd-logind[303]: Failed to remove POSIX shared memory directory home/username: Directory not empty

And I'm lucky to have this message at all, because there was a file in that home directory which didn't belong to the user, so that's the only file that's not gone, poof. Otherwise it seems it would just have deleted it silently.

Now this box has been running for a while (since 2019 or so) and the backup script hasn't been touched since March or so... what changed?

I updated it earlier today (pacman.log):

[2022-09-15T07:42:42+0200] [PACMAN] starting full system upgrade
[2022-09-15T07:42:45+0200] [ALPM] transaction started
[2022-09-15T07:42:46+0200] [ALPM] upgraded ca-certificates-utils (20210603-1 -> 20220905-1)
[2022-09-15T07:42:46+0200] [ALPM] upgraded ca-certificates (20210603-1 -> 20220905-1)
[2022-09-15T07:42:46+0200] [ALPM] upgraded curl (7.85.0-1 -> 7.85.0-2)
[2022-09-15T07:42:46+0200] [ALPM] upgraded gsasl (2.0.1-1 -> 2.2.0-1)
[2022-09-15T07:42:46+0200] [ALPM] transaction completed
[2022-09-15T07:42:46+0200] [ALPM] running '30-systemd-update.hook'...
[2022-09-15T07:42:46+0200] [ALPM] running '40-update-ca-trust.hook'...
[2022-09-15T07:42:47+0200] [ALPM] running 'texinfo-install.hook'...

I don't know systemd too well, to be honest. I don't use it much, it's mostly just there, and hopefully, nothing happens. The config files are all untouched / standard, as far as I can tell.

But now it suddenly started deleting everything, and it happens every time the user logs out, so I can't even restore the files... they just gone poof, again.

Is it just me, or anyone else?

Last edited by frostschutz (2022-09-15 19:11:37)

loqs

Member

Registered: 2014-03-06

Posts: 18,964

Re: [SOLVED] systemd-logind suddenly deletes all files

If you log that user in is there anything in /dev/shm that references home/username ?

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: [SOLVED] systemd-logind suddenly deletes all files

*facepalm*

Thanks. I owe you one...

Is it just me, or anyone else?

So turns out it's just me doing stupid things in /dev/shm - out of lazyness/convenience I sometimes use /dev/shm for tinkering (why mount another tmpfs when you already have one - right?) and for reasons I don't remember I had a bind mount of / to /dev/shm/foobar. Which means all files are under /dev/shm... well, not really, but that's what it looks like. And of course systemd goes ahead and cleans that up. Aaaand it's gone. Poof.

Whoopsie-daisy. I simply did not know systemd did that... I will stop using /dev/shm for experiments in the future.

Mystery solved (thanks to loqs for pointing it out).

Last edited by frostschutz (2022-09-15 20:06:20)