You are not logged in.

#1 2024-01-20 02:17:21

ectospasm
Member
Registered: 2015-08-28
Posts: 273

[SOLVED] systemd-homed and weird journal errors

SOLVED

I had to run the following command:

sudo homectl update --auto-resize-mode=off --rebalance-weight=off <user>

I was prompted for my user password, and then "Auto Resize" and "Rebalance" showed in the output of homectl inspect, both were off (they're on by default).  After a reboot, I no longer got the "no space left on device" message, and journalctl no longer complains about a corrupt journal file.

Original Post

This is a relatively new installation, and I've had this problem ever since I first set it up.  Also, this is my first time using systemd-homed, and I'm not sure where to look to address this.

Whenever I log in with my systemd-homed user, I immediately get a printk stating that the journal has run out of space (i.e. "no space left on device").  That's not the message verbatim since I'm not sure how to copy it from tty1, but I do see this in my journal:

$ journalctl -b0 | grep -i 'no space left'
Journal file /var/log/journal/490336ad3ce04b7bae989340da3fe01b/system@00000000000000000000000000000000-0000000000000000-0000000000000000.journal corrupted, ignoring file.
Jan 19 20:32:16 fluorine systemd-journald[445]: Failed to create new system journal: No space left on device

That corrupt journal file is spit out to stderr by journalctl, but I'm including it as a clue.  I also see a similar message about lastlog, it can't be written to because there's no space left on the device.  Again, not verbatim, the message appears in tty1 when I log in (I'm not using a DM, just the plain Linux console, and I launch X.org with startx).

I don't think it's out of space, df -h reports plenty of space:

Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/top        1.9T  1.6T  305G  85% /
devtmpfs               4.0M     0  4.0M   0% /dev
tmpfs                   32G   58M   32G   1% /dev/shm
efivarfs               246K  108K  134K  45% /sys/firmware/efi/efivars
tmpfs                   13G  1.9M   13G   1% /run
/dev/mapper/top        1.9T  1.6T  305G  85% /home
/dev/mapper/top        1.9T  1.6T  305G  85% /root
/dev/mapper/top        1.9T  1.6T  305G  85% /swap
/dev/mapper/top        1.9T  1.6T  305G  85% /var
tmpfs                   32G   36M   32G   1% /tmp
/dev/nvme0n1p1         510M   41M  470M   8% /efi
tmpfs                  6.3G   16K  6.3G   1% /run/user/0
/dev/mapper/home-trey  1.6T   48G  1.5T   4% /home/trey
tmpfs                  6.3G   36K  6.3G   1% /run/user/60147
none                   100M     0  100M   0% /home/trey/bitwarden
/dev/mapper/home-trey  1.6T   48G  1.5T   4% /home/trey/bugs
/dev/mapper/home-trey  1.6T   48G  1.5T   4% /home/trey/.cache
/dev/mapper/home-trey  1.6T   48G  1.5T   4% /home/trey/downloads
/dev/mapper/home-trey  1.6T   48G  1.5T   4% /home/trey/hosts
/dev/mapper/home-trey  1.6T   48G  1.5T   4% /home/trey/src

I may have been too fancy when I set this up.  I'm using Btrfs subvolumes, and the top-level subvolume (/dev/mapper/top) is not mounted directly.  My root subvolume is a subvolume with parent ID=5, and a lot of my other "partitions" are subvolumes under subvol=/root (i.e., /root/home, /root/var, /root/swap, /root/root [root's home directory]).  subvol=/root/home is mounted to /home, and has NOCOW set on /home (chattr -C /home before creating the first systemd-homed user/directory).

Later on (after I had installed the first system), I set NOCOW on /var (chattr -RC /var).  I know that doesn't do anything for existing files, but that may also be part of the problem.

I also got fancy with my systemd-homed directory.  I'm using Btrfs as the home directory filesystem (in a LUKS2 container), and I have subvolumes on that as well.  You can see those in the df -h output above.  I did have to hack some stuff in my .zshrc to get the ownership and permissions of the mounted homed subvolumes to be correct;  I couldn't figure out a better way to do so for now.  Basically I'm preparing to use snapper to snapshot my mounted home directory, and stuff like downloads and .cache I don't want to be in the snapshots.

Some things to note:  this printk and no space left on device messages only appear when I log in as my systemd-homed user.  I've tried logging in directly as root and the printk doesn't happen (though I didn't fully check the journal when I tried).  Also, I waited ten minutes to log in with my systemd-homed user the last time, in case it was some kind of timing issue.  I waited more than ten minutes, but that had no effect on the problem (same printk and lastlog messages about no space left on device).

I'm sure I'm not providing everything to figure this out.  What else do you need?

Last edited by ectospasm (2024-01-21 10:27:41)

Offline

#2 2024-01-21 06:55:27

swsnr
Member
Registered: 2024-01-21
Posts: 17

Re: [SOLVED] systemd-homed and weird journal errors

Run "homectl inspect" on your homed user account, and verify that "Auto Resize" and "Rebalance" are "off".  If these are listed as on, or aren't listed (these settings default to on), turn them off per homectl manpage.

Offline

#3 2024-01-21 10:22:02

ectospasm
Member
Registered: 2015-08-28
Posts: 273

Re: [SOLVED] systemd-homed and weird journal errors

swsnr wrote:

Run "homectl inspect" on your homed user account, and verify that "Auto Resize" and "Rebalance" are "off".  If these are listed as on, or aren't listed (these settings default to on), turn them off per homectl manpage.

Yep, that fixed it!  Also, it doesn't appear that journalctl complains about the corrupt journal file anymore!  WIN/WIN!

Offline

Board footer

Powered by FluxBB