Arch Linux

broken pipe

Member

Registered: 2010-12-10

Posts: 243

journald full of "overlayfs: "xino" feature enabled using 3 up"

Hi all,

I think it's related to a recent kernel update, but my journald log is completely full of the message:

kernel: overlayfs: "xino" feature enabled using 3 upper inode bits.

It's generated every second and I can't figure out where it's coming from. The server is running multiple docker containers, nfs, samba.

I couldn't find any info about this online either. Do you have any ideas?

CPU: 6-core 12th Gen Intel Core i5-12400T (-MT MCP-) speed/min/max: 800/800/4200 MHz
Kernel: 6.11.5-arch1-1 x86_64 Up: 32m Mem: 7.14/15.31 GiB (46.6%) Storage: 4.09 TiB (27.3% used)
Procs: 474 Shell: Bash inxi: 3.3.36

Drives:
  Local Storage: total: 4.09 TiB used: 1.12 TiB (27.3%)
  ID-1: /dev/nvme0n1 vendor: Crucial model: CT500P1SSD8 size: 465.76 GiB
  ID-2: /dev/sda vendor: Crucial model: CT4000MX500SSD1 size: 3.64 TiB

Best regards!

-thc

Member

Registered: 2017-03-15

Posts: 714

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

That looks like one of your docker containers runs on an overlay filesystem - right?

duffydack

Member

Registered: 2012-06-17

Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I'm getting this problem since the other day.  It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal.  Not sure how to get around it.  I've removed and pruned all the images and spun them up again, same issue.

Last edited by duffydack (2024-10-24 09:00:27)

broken pipe

Member

Registered: 2010-12-10

Posts: 243

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I'm getting this problem since the other day.  It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal.  Not sure how to get around it.  I've removed and pruned all the images and spun them up again, same issue.

ok, i can confirm that. the fewer containers are running, the fewer messages. when immich is running, there are significantly more.

duffydack

Member

Registered: 2012-06-17

Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I'm getting this problem since the other day.  It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal.  Not sure how to get around it.  I've removed and pruned all the images and spun them up again, same issue.

ok, i can confirm that. the fewer containers are running, the fewer messages. when immich is running, there are significantly more.

As far as my other containers, I never see the message again once they are up and running, it's just immich.  Also, mine is random intervals, every few to maybe 30s.

SubS0

Member

Registered: 2015-02-10

Posts: 39

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Same here (no immich container).

Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.

I’m still running 6.9.7-arch1-1 and didn’t reboot the server. I’ll report here if a reboot change something with runc 1.2.0

Edit :Same with a device running 6.11.5-arch1-1 and runc 1.2.0-1

Last edited by SubS0 (2024-10-25 16:59:33)

slip

Member

Registered: 2019-09-09

Posts: 11

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.

Thanks for that. This did, indeed, fix it for me as well.

duffydack

Member

Registered: 2012-06-17

Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Same here (no immich container).

Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.

I’m still running 6.9.7-arch1-1 and didn’t reboot the server. I’ll report here if a reboot change something with runc 1.2.0

Edit :Same with a device running 6.11.5-arch1-1 and runc 1.2.0-1

Nice one!

Last edited by duffydack (2024-10-25 18:56:13)

SubS0

Member

Registered: 2015-02-10

Posts: 39

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I have no idea what's going on, and the problem is far beyond my competence.

I don’t see anything explicitely related on  https://github.com/opencontainers/runc/releases

It would be nice if someone with a better understanding of docker’s overlays FS  can dig into it and help formatting a bug report and/or identity where’s the best place to open one.

Heyian

Member

Registered: 2024-10-28

Posts: 2

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Thx a lot for this thread, I downgraded to get rid of this too. I tried looking up for a solution but couldn't find anything. I hope someone figure this out...

The PR#4448 seems to be the one that implemented the overlayfs but I have no idea how this could be related to our issue. No issues about our problem on the repo either.

kode54

Member

Registered: 2013-10-21

Posts: 28

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I am experiencing this with a bitwarden container. linux-lts. How many of you have a btrfs rootfs? Because I have one.

broken pipe

Member

Registered: 2010-12-10

Posts: 243

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I opened a discussion on the project's github page: https://github.com/opencontainers/runc/discussions/4506

duffydack

Member

Registered: 2012-06-17

Posts: 12

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

The runc update today doesn't fix anything btw

cyphar

Member

Registered: 2013-08-25

Posts: 5

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

(Resurrecting my ancient Arch Forums account...)

This patch should fix the issue: https://github.com/opencontainers/runc/pull/4509. It fixes the issue on my machinem, and the patch will be in runc 1.2.2.

This was caused by the change to using overlayfs to protect the runc binary, but it would only manifest on systems that have xino_auto=y set for the overlayfs module and have /usr/bin and /run on different filesystems. I also looked through the rest of the overlayfs params and it seems xino is the only one that is relevant to our binary protection that will produce dmesg spam.

Last edited by cyphar (2024-11-04 10:09:44)

broken pipe

Member

Registered: 2010-12-10

Posts: 243

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

(Resurrecting my ancient Arch Forums account...)

This patch should fix the issue: https://github.com/opencontainers/runc/pull/4509. It fixes the issue on my machinem, and the patch will be in runc 1.2.2.

This was caused by the change to using overlayfs to protect the runc binary, but it would only manifest on systems that have xino_auto=y set for the overlayfs module and have /usr/bin and /run on different filesystems. I also looked through the rest of the overlayfs params and it seems xino is the only one that is relevant to our binary protection that will produce dmesg spam.

Thank you very much for your quick help!

cyphar

Member

Registered: 2013-08-25

Posts: 5

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I've also sent a kernel patch to port overlayfs to log all of these messages to the fsopen handle if we are using the new mount API. For new mount API users (like runc) filesystems shouldn't be logging information about the mount configuration to dmesg anyway.

cyphar

Member

Registered: 2013-08-25

Posts: 5

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

runc 1.2.2 includes the fix for this issue, btw.

Heyian

Member

Registered: 2024-10-28

Posts: 2

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

I confirm that 1.2.2 has solved the issue for me too. Thx!

SRG

Member

Registered: 2012-12-09

Posts: 27

Re: journald full of "overlayfs: "xino" feature enabled using 3 up"

Came here just before runc 1.2.2 reached my servers, and i was also annoyed by the so many messages generated since a few days.