You are not logged in.

#1 2024-10-24 06:54:57

broken pipe
Member
Registered: 2010-12-10
Posts: 243

journald full of "overlayfs: "xino" feature enabled using 3 up"

Hi all,

I think it's related to a recent kernel update, but my journald log is completely full of the message:

kernel: overlayfs: "xino" feature enabled using 3 upper inode bits.

It's generated every second and I can't figure out where it's coming from. The server is running multiple docker containers, nfs, samba.

I couldn't find any info about this online either. Do you have any ideas?


CPU: 6-core 12th Gen Intel Core i5-12400T (-MT MCP-) speed/min/max: 800/800/4200 MHz
Kernel: 6.11.5-arch1-1 x86_64 Up: 32m Mem: 7.14/15.31 GiB (46.6%) Storage: 4.09 TiB (27.3% used)
Procs: 474 Shell: Bash inxi: 3.3.36

Drives:
  Local Storage: total: 4.09 TiB used: 1.12 TiB (27.3%)
  ID-1: /dev/nvme0n1 vendor: Crucial model: CT500P1SSD8 size: 465.76 GiB
  ID-2: /dev/sda vendor: Crucial model: CT4000MX500SSD1 size: 3.64 TiB

Best regards!

Offline

#2 2024-10-24 07:38:48

-thc
Member
Registered: 2017-03-15
Posts: 685

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

That looks like one of your docker containers runs on an overlay filesystem - right?

Offline

#3 2024-10-24 08:59:38

duffydack
Member
Registered: 2012-06-17
Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I'm getting this problem since the other day.  It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal.  Not sure how to get around it.  I've removed and pruned all the images and spun them up again, same issue.

Last edited by duffydack (2024-10-24 09:00:27)

Offline

#4 2024-10-24 09:03:27

broken pipe
Member
Registered: 2010-12-10
Posts: 243

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

duffydack wrote:

I'm getting this problem since the other day.  It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal.  Not sure how to get around it.  I've removed and pruned all the images and spun them up again, same issue.

ok, i can confirm that. the fewer containers are running, the fewer messages. when immich is running, there are significantly more.

Offline

#5 2024-10-24 11:19:47

duffydack
Member
Registered: 2012-06-17
Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

broken pipe wrote:
duffydack wrote:

I'm getting this problem since the other day.  It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal.  Not sure how to get around it.  I've removed and pruned all the images and spun them up again, same issue.

ok, i can confirm that. the fewer containers are running, the fewer messages. when immich is running, there are significantly more.

As far as my other containers, I never see the message again once they are up and running, it's just immich.  Also, mine is random intervals, every few to maybe 30s.

journal spam

Offline

#6 2024-10-25 16:34:24

SubS0
Member
Registered: 2015-02-10
Posts: 39

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Same here (no immich container).

Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.

I’m still running 6.9.7-arch1-1 and didn’t reboot the server. I’ll report here if a reboot change something with runc 1.2.0

Edit :Same with a device running 6.11.5-arch1-1 and runc 1.2.0-1

Last edited by SubS0 (2024-10-25 16:59:33)

Offline

#7 2024-10-25 17:35:27

slip
Member
Registered: 2019-09-09
Posts: 11

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

SubS0 wrote:

Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.

Thanks for that. This did, indeed, fix it for me as well.

Offline

#8 2024-10-25 18:45:24

duffydack
Member
Registered: 2012-06-17
Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

SubS0 wrote:

Same here (no immich container).

Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.

I’m still running 6.9.7-arch1-1 and didn’t reboot the server. I’ll report here if a reboot change something with runc 1.2.0

Edit :Same with a device running 6.11.5-arch1-1 and runc 1.2.0-1

Nice one!

Last edited by duffydack (2024-10-25 18:56:13)

Offline

#9 2024-10-25 19:09:07

SubS0
Member
Registered: 2015-02-10
Posts: 39

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I have no idea what's going on, and the problem is far beyond my competence.

I don’t see anything explicitely related on  https://github.com/opencontainers/runc/releases

It would be nice if someone with a better understanding of docker’s overlays FS  can dig into it and help formatting a bug report and/or identity where’s the best place to open one.

Offline

#10 2024-10-28 20:16:45

Heyian
Member
Registered: 2024-10-28
Posts: 2

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Thx a lot for this thread, I downgraded to get rid of this too. I tried looking up for a solution but couldn't find anything. I hope someone figure this out...

The PR#4448 seems to be the one that implemented the overlayfs but I have no idea how this could be related to our issue. No issues about our problem on the repo either.

Offline

#11 2024-11-02 05:23:30

kode54
Member
Registered: 2013-10-21
Posts: 28

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I am experiencing this with a bitwarden container. linux-lts. How many of you have a btrfs rootfs? Because I have one.

Offline

#12 2024-11-02 09:20:07

broken pipe
Member
Registered: 2010-12-10
Posts: 243

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I opened a discussion on the project's github page: https://github.com/opencontainers/runc/discussions/4506

Offline

#13 2024-11-02 11:20:57

duffydack
Member
Registered: 2012-06-17
Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

The runc update today doesn't fix anything btw

Offline

#14 2024-11-04 10:08:52

cyphar
Member
Registered: 2013-08-25
Posts: 5

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

(Resurrecting my ancient Arch Forums account...)

This patch should fix the issue: https://github.com/opencontainers/runc/pull/4509. It fixes the issue on my machinem, and the patch will be in runc 1.2.2.

This was caused by the change to using overlayfs to protect the runc binary, but it would only manifest on systems that have xino_auto=y set for the overlayfs module and have /usr/bin and /run on different filesystems. I also looked through the rest of the overlayfs params and it seems xino is the only one that is relevant to our binary protection that will produce dmesg spam.

Last edited by cyphar (2024-11-04 10:09:44)

Offline

#15 2024-11-05 17:14:52

broken pipe
Member
Registered: 2010-12-10
Posts: 243

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

cyphar wrote:

(Resurrecting my ancient Arch Forums account...)

This patch should fix the issue: https://github.com/opencontainers/runc/pull/4509. It fixes the issue on my machinem, and the patch will be in runc 1.2.2.

This was caused by the change to using overlayfs to protect the runc binary, but it would only manifest on systems that have xino_auto=y set for the overlayfs module and have /usr/bin and /run on different filesystems. I also looked through the rest of the overlayfs params and it seems xino is the only one that is relevant to our binary protection that will produce dmesg spam.

Thank you very much for your quick help!

Offline

#16 2024-11-06 05:07:48

cyphar
Member
Registered: 2013-08-25
Posts: 5

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I've also sent a kernel patch to port overlayfs to log all of these messages to the fsopen handle if we are using the new mount API. For new mount API users (like runc) filesystems shouldn't be logging information about the mount configuration to dmesg anyway.

Offline

#17 2024-11-16 05:18:16

cyphar
Member
Registered: 2013-08-25
Posts: 5

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

runc 1.2.2 includes the fix for this issue, btw.

Offline

#18 2024-11-18 13:30:51

Heyian
Member
Registered: 2024-10-28
Posts: 2

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I confirm that 1.2.2 has solved the issue for me too. Thx!

Offline

#19 2024-11-18 19:10:03

SRG
Member
Registered: 2012-12-09
Posts: 27

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Came here just before runc 1.2.2 reached my servers, and i was also annoyed by the so many messages generated since a few days.

Offline

Board footer

Powered by FluxBB