You are not logged in.
Hi all,
I think it's related to a recent kernel update, but my journald log is completely full of the message:
kernel: overlayfs: "xino" feature enabled using 3 upper inode bits.
It's generated every second and I can't figure out where it's coming from. The server is running multiple docker containers, nfs, samba.
I couldn't find any info about this online either. Do you have any ideas?
CPU: 6-core 12th Gen Intel Core i5-12400T (-MT MCP-) speed/min/max: 800/800/4200 MHz
Kernel: 6.11.5-arch1-1 x86_64 Up: 32m Mem: 7.14/15.31 GiB (46.6%) Storage: 4.09 TiB (27.3% used)
Procs: 474 Shell: Bash inxi: 3.3.36
Drives:
Local Storage: total: 4.09 TiB used: 1.12 TiB (27.3%)
ID-1: /dev/nvme0n1 vendor: Crucial model: CT500P1SSD8 size: 465.76 GiB
ID-2: /dev/sda vendor: Crucial model: CT4000MX500SSD1 size: 3.64 TiB
Best regards!
Offline
That looks like one of your docker containers runs on an overlay filesystem - right?
Offline
I'm getting this problem since the other day. It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal. Not sure how to get around it. I've removed and pruned all the images and spun them up again, same issue.
Last edited by duffydack (2024-10-24 09:00:27)
Offline
I'm getting this problem since the other day. It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal. Not sure how to get around it. I've removed and pruned all the images and spun them up again, same issue.
ok, i can confirm that. the fewer containers are running, the fewer messages. when immich is running, there are significantly more.
Offline
duffydack wrote:I'm getting this problem since the other day. It is with docker, and appears once or twice when starting up my other containers (npm, freshrss, sonarr, lidarr, radarr) but it's immich that's making it spam my journal. Not sure how to get around it. I've removed and pruned all the images and spun them up again, same issue.
ok, i can confirm that. the fewer containers are running, the fewer messages. when immich is running, there are significantly more.
As far as my other containers, I never see the message again once they are up and running, it's just immich. Also, mine is random intervals, every few to maybe 30s.
Offline
Same here (no immich container).
Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.
I’m still running 6.9.7-arch1-1 and didn’t reboot the server. I’ll report here if a reboot change something with runc 1.2.0
Edit :Same with a device running 6.11.5-arch1-1 and runc 1.2.0-1
Last edited by SubS0 (2024-10-25 16:59:33)
Offline
Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.
Thanks for that. This did, indeed, fix it for me as well.
Offline
Same here (no immich container).
Downgrading runc from 1.2.0-1 -> 1.1.15-1 seems to stop stops the spamming.
I’m still running 6.9.7-arch1-1 and didn’t reboot the server. I’ll report here if a reboot change something with runc 1.2.0
Edit :Same with a device running 6.11.5-arch1-1 and runc 1.2.0-1
Nice one!
Last edited by duffydack (2024-10-25 18:56:13)
Offline
I have no idea what's going on, and the problem is far beyond my competence.
I don’t see anything explicitely related on https://github.com/opencontainers/runc/releases
It would be nice if someone with a better understanding of docker’s overlays FS can dig into it and help formatting a bug report and/or identity where’s the best place to open one.
Offline
Thx a lot for this thread, I downgraded to get rid of this too. I tried looking up for a solution but couldn't find anything. I hope someone figure this out...
The PR#4448 seems to be the one that implemented the overlayfs but I have no idea how this could be related to our issue. No issues about our problem on the repo either.
Offline
I am experiencing this with a bitwarden container. linux-lts. How many of you have a btrfs rootfs? Because I have one.
Offline
I opened a discussion on the project's github page: https://github.com/opencontainers/runc/discussions/4506
Offline
The runc update today doesn't fix anything btw
Offline
(Resurrecting my ancient Arch Forums account...)
This patch should fix the issue: https://github.com/opencontainers/runc/pull/4509. It fixes the issue on my machinem, and the patch will be in runc 1.2.2.
This was caused by the change to using overlayfs to protect the runc binary, but it would only manifest on systems that have xino_auto=y set for the overlayfs module and have /usr/bin and /run on different filesystems. I also looked through the rest of the overlayfs params and it seems xino is the only one that is relevant to our binary protection that will produce dmesg spam.
Last edited by cyphar (2024-11-04 10:09:44)
Offline
(Resurrecting my ancient Arch Forums account...)
This patch should fix the issue: https://github.com/opencontainers/runc/pull/4509. It fixes the issue on my machinem, and the patch will be in runc 1.2.2.
This was caused by the change to using overlayfs to protect the runc binary, but it would only manifest on systems that have xino_auto=y set for the overlayfs module and have /usr/bin and /run on different filesystems. I also looked through the rest of the overlayfs params and it seems xino is the only one that is relevant to our binary protection that will produce dmesg spam.
Thank you very much for your quick help!
Offline
I've also sent a kernel patch to port overlayfs to log all of these messages to the fsopen handle if we are using the new mount API. For new mount API users (like runc) filesystems shouldn't be logging information about the mount configuration to dmesg anyway.
Offline