Docker issues after docker-1:23.0.1-1 update

Zack7777 · 2023-02-12 14:24:03

After updating from 1:20.0 to 1:23.0.1-1 a bunch of containers started to experience crashes.
The app inside catches SIGTERM out of the blue.
I thought it to be the problem with a particular container, so I opened a bug on their gh: https://github.com/linuxserver/docker-w … issues/243
But later on I started to notice that other containers are behaving weird and restarting randomly, too. It would seem that the package is broken.

benvosper · 2023-02-13 11:10:33

I'm having similar issues with 1:23.0.1, but in my case, the docker service is unable to start at all:

> journalctl -xeu docker.service
Feb 13 10:39:26 beta systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Feb 13 10:39:26 beta systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 1229 and the job result is failed.

Rolling back to 1:20.10.23 seems to work fine.

Zack7777 · 2023-02-13 11:38:50

Opened a bug report: https://bugs.archlinux.org/task/77481

programmador · 2023-02-13 11:58:47

For me it causes 100% cpu load for rabbitmq containers. They look as started but erl_child_setup process just eats cpu and rabbitmq server actually does not start. Rolling back to 1.20 helped.
Strange thing is that only reboot helps after downgrading.
This DOES NOT HELP:
* systemctl stop docker
* downgrade
* systemctl daemon-reload
* systemctl start docker
Erlang is still broken in docker after performing these actions. Only rebooting after a downgrade helps.

Humandoodlebug · 2023-02-15 07:31:09

Same here, on two different devices. Based on this thread, I suspect rabbitmq is to blame (which is running in the container in question).

TantricShades · 2023-02-15 23:09:36

Does anyone know where I can find the older version of this package? I have a fresh install of arch and I am fairly certain I don't have the old version in my cache to rollback to. I could be misunderstanding how that works. When I go to github for the source, I can't find an older version. Any help and or advice would be appreciated, thank you.

Zack7777 · 2023-02-16 00:13:20

TantricShades wrote:

Does anyone know where I can find the older version of this package? I have a fresh install of arch and I am fairly certain I don't have the old version in my cache to rollback to. I could be misunderstanding how that works. When I go to github for the source, I can't find an older version. Any help and or advice would be appreciated, thank you.

There you go: https://archive.archlinux.org/repos/202 … kg.tar.zst
Bookmark this: https://wiki.archlinux.org/title/Arch_Linux_Archive

TantricShades · 2023-02-16 00:16:36

Zack7777 wrote:

TantricShades wrote:
Does anyone know where I can find the older version of this package? I have a fresh install of arch and I am fairly certain I don't have the old version in my cache to rollback to. I could be misunderstanding how that works. When I go to github for the source, I can't find an older version. Any help and or advice would be appreciated, thank you.
There you go: https://archive.archlinux.org/repos/202 … kg.tar.zst
Bookmark this: https://wiki.archlinux.org/title/Arch_Linux_Archive

GEATLY appreciated, thank you

gileri · 2023-02-16 10:09:45

programmador wrote:

For me it causes 100% cpu load for rabbitmq containers. They look as started but erl_child_setup process just eats cpu and rabbitmq server actually does not start. Rolling back to 1.20 helped.
Erlang is still broken in docker after performing these actions. Only rebooting after a downgrade helps.

Same here. One thing that alleviate the issue is setting lower nofile ulimits when running the rabbitmq container (by default there was 1073741816):

docker run --rm --ulimit nofile=65536:65536 rabbitmq:3.6.6-management

Similar issue here.

Last edited by gileri (2023-02-16 10:11:28)

iMon · 2023-02-18 10:02:31

What fixed the issue for me was to add the following configurations:

/etc/systemd/system/docker.service.d/override.conf:

[Service]                                                                                                                     
LimitNOFILE=1048576

/etc/systemd/system/containerd.service.d/override.conf:

[Service]                                                                                                                     
LimitNOFILE=1048576

Now my containers run like before the update to Docker 23.0.

To make it take effect, I had to restart my computer. Restarting both services was not enough.

Last edited by iMon (2023-02-19 09:21:56)

MB4E · 2023-02-18 20:53:27

@iMon : Didn't work for me.

iMon · 2023-02-19 09:22:30

I had a typo in the path above. It should be "/etc/systemd/system/docker.service.d/override.conf" not "/etc/systemd/system/dockerd.service.d/override.conf".

bennypt · 2023-02-20 14:30:07

iMon wrote:

What fixed the issue for me was to add the following configurations:
/etc/systemd/system/docker.service.d/override.conf:
[Service]                                                                                                                     
LimitNOFILE=1048576
/etc/systemd/system/containerd.service.d/override.conf:
[Service]                                                                                                                     
LimitNOFILE=1048576
Now my containers run like before the update to Docker 23.0.
To make it take effect, I had to restart my computer. Restarting both services was not enough.

@iMon that worked for me

Thank you for sharing.

Zarch · 2023-03-01 14:52:45

I had a similar issue after updating to 1:23.0.1. My CPU would spike to 100% and repeatedly freeze my screen after running the Docker containers, so it's not just RabbitMQ. Also, the MySQL container would exit with the message: "mysqld failed while attempting to check config".

iMon wrote:

I had a typo in the path above. It should be "/etc/systemd/system/docker.service.d/override.conf" not "/etc/systemd/system/dockerd.service.d/override.conf".

Thanks, this solved the problem for me.

rogedelgado · 2023-03-14 00:10:10

Hello,

I can confirm that after the upgrade to docker 1:23.0.1-1 I had a problem while building my images. The error was:

library initialization failed - unable to allocate file descriptor table - out of memory

Using the fix that @iMon mention fix the problem also. Thanks iMon

Last edited by rogedelgado (2023-03-14 00:10:54)

marrs · 2023-03-18 23:50:53

I had a different issue. My configuration had been using the deprecated (presumably now removed) devicemapper by default and I had to configure my docker daemon to use a different storage driver instead. I went with overlay2.

To do this, I updated my /etc/docker/daemon.json config as follows

{
    "storage-driver": "overlay2"
}

More info here: https://wiki.archlinux.org/title/Docker#Storage_driver

Last edited by marrs (2023-03-18 23:54:24)

daizhirui · 2023-07-12 06:29:56

iMon wrote:

What fixed the issue for me was to add the following configurations:
/etc/systemd/system/docker.service.d/override.conf:
[Service]                                                                                                                     
LimitNOFILE=1048576
/etc/systemd/system/containerd.service.d/override.conf:
[Service]                                                                                                                     
LimitNOFILE=1048576
Now my containers run like before the update to Docker 23.0.
To make it take effect, I had to restart my computer. Restarting both services was not enough.

This also works for Docker 24.0.2
Thank you!

diogobaeder · 2023-08-10 00:15:10

iMon wrote:

What fixed the issue for me was to add the following configurations:
/etc/systemd/system/docker.service.d/override.conf:
[Service]                                                                                                                     
LimitNOFILE=1048576
/etc/systemd/system/containerd.service.d/override.conf:
[Service]                                                                                                                     
LimitNOFILE=1048576
Now my containers run like before the update to Docker 23.0.
To make it take effect, I had to restart my computer. Restarting both services was not enough.

I can confirm that this solved the dreadfully slow Docker builds I was having on my computer, too. Using Docker 24.0.5. Thanks for the solution, man! :-)

2ManyDogs · 2023-08-10 11:14:33

The OP has not been back since March, so I am going to consider this thread abandoned and close it now.

Arch Linux

#1 2023-02-12 14:24:03

Docker issues after docker-1:23.0.1-1 update

#2 2023-02-13 11:10:33

Re: Docker issues after docker-1:23.0.1-1 update

#3 2023-02-13 11:38:50

Re: Docker issues after docker-1:23.0.1-1 update

#4 2023-02-13 11:58:47

Re: Docker issues after docker-1:23.0.1-1 update

#5 2023-02-15 07:31:09

Re: Docker issues after docker-1:23.0.1-1 update

#6 2023-02-15 23:09:36

Re: Docker issues after docker-1:23.0.1-1 update

#7 2023-02-16 00:13:20

Re: Docker issues after docker-1:23.0.1-1 update

#8 2023-02-16 00:16:36

Re: Docker issues after docker-1:23.0.1-1 update

#9 2023-02-16 10:09:45

Re: Docker issues after docker-1:23.0.1-1 update

#10 2023-02-18 10:02:31

Re: Docker issues after docker-1:23.0.1-1 update

#11 2023-02-18 20:53:27

Re: Docker issues after docker-1:23.0.1-1 update

#12 2023-02-19 09:22:30

Re: Docker issues after docker-1:23.0.1-1 update

#13 2023-02-20 14:30:07

Re: Docker issues after docker-1:23.0.1-1 update

#14 2023-03-01 14:52:45

Re: Docker issues after docker-1:23.0.1-1 update

#15 2023-03-14 00:10:10

Re: Docker issues after docker-1:23.0.1-1 update

#16 2023-03-18 23:50:53

Re: Docker issues after docker-1:23.0.1-1 update

#17 2023-07-12 06:29:56

Re: Docker issues after docker-1:23.0.1-1 update

#18 2023-08-10 00:15:10

Re: Docker issues after docker-1:23.0.1-1 update

#19 2023-08-10 11:14:33

Re: Docker issues after docker-1:23.0.1-1 update

Board footer