You are not logged in.

Pages: 1

#1 2025-03-06 02:39:56

tmoore
Member
Registered: 2012-06-03
Posts: 8

Strange dockerd crashing bug when pulling images

I hope this is the proper forum for this.

I have a strange bug that I've reproduced on two different arch linux installations.  It seems to happen when doing docker pulls, either when deploying something through Portainer, or when simply doing a "docker run" on the command line.   I initially thought it was bad hardware on a new Beelink minipc I got, but then my old workhorse machine started experiencing it as well.

OS: Arch Linux
Docker version: Docker version 28.0.0, build f9ced58158

Here's the journalctl log with the dockerd debug enabled at the time

Mar 03 21:06:23 beelink dockerd[592863]: time="2025-03-03T21:06:23.026678534-05:00" level=debug msg="unpigz binary not found, falling back to go gzip library"
Mar 03 21:06:23 beelink dockerd[592863]: time="2025-03-03T21:06:23.033101700-05:00" level=debug msg="Applying tar in /data/docker/overlay2/fb372f724b92eb6bee47b80926f1616bb71882c217d84acb2e5887be14550825/diff" storage-driver=overlay2
Mar 03 21:06:23 beelink dockerd[592863]: time="2025-03-03T21:06:23.037843318-05:00" level=debug msg="Applied tar sha256:bcd8ddbd4e1213f888ec015f6da863e06b47adfba3d587020a4d8dc38a382869 to fb372f724b92eb6bee47b80926f1616bb71882c217d84acb2e5887be14550825, size: 0"
Mar 03 21:06:23 beelink dockerd[592863]: time="2025-03-03T21:06:23.052054757-05:00" level=debug msg="unpigz binary not found, falling back to go gzip library"
Mar 03 21:06:23 beelink dockerd[592863]: time="2025-03-03T21:06:23.058428742-05:00" level=debug msg="Applying tar in /data/docker/overlay2/71884ad97b2325e16bf9c058c2fc46092ee1d69e27a1428a12a4bbf4494e0fe9/diff" storage-driver=overlay2
Mar 03 21:06:23 beelink systemd-coredump[1731698]: [?] Process 592863 (dockerd) of user 0 dumped core.

                                                   Stack trace of thread 843724:
                                                   #0  0x0000571f006d0098 n/a (n/a + 0x0)
                                                   #1  0x0000571f0072f419 n/a (n/a + 0x0)
                                                   #2  0x0000571f00734869 n/a (n/a + 0x0)
                                                   #3  0x0000571f0072bd56 n/a (n/a + 0x0)
                                                   #4  0x0000571f006ff625 n/a (n/a + 0x0)
                                                   #5  0x0000571f006ff2df n/a (n/a + 0x0)
                                                   ELF object binary architecture: AMD x86-64
Mar 03 21:06:23 beelink systemd[1]: docker.service: Main process exited, code=dumped, status=11/SEGV
Mar 03 21:06:23 beelink systemd[1]: docker.service: Failed with result 'core-dump'.
Mar 03 21:06:23 beelink systemd[1]: docker.service: Unit process 1381876 (docker-proxy) remains running after unit stopped.
Mar 03 21:06:23 beelink systemd[1]: docker.service: Consumed 54min 31.858s CPU time, 552.1M memory peak.
Mar 03 21:06:25 beelink systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Mar 03 21:06:25 beelink systemd[1]: Starting Docker Application Container Engine...

Then I've run "coredumpctl gdb" on the crash, but the only hint it gives is:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000571f006d0098 in runtime[mallocgcSmallNoscan] ()
[Current thread is 1 (Thread 0x7c6c3e7fc6c0 (LWP 843724))

I've tried to google "mallocgcSmallNoscan" but haven't discovered exactly what that means.

It just crashed again on my other computer with the same error.  Unfortunately it didn't save the coredump for some reason

Mar 05 21:29:04 trinity dockerd[5612]: time="2025-03-05T21:29:04.864380182-05:00" level=warning msg="close stream failed" error="NotFound: task eb45389dc1b27365c9875524a8137268a6524e6a1a95be9ce5be7a9de37b0f87 not found" stream=stdin
Mar 05 21:29:50 trinity dockerd[5612]: time="2025-03-05T21:29:50.263026308-05:00" level=error msg="Not continuing with pull after error" error="errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Mar 05 21:29:50 trinity dockerd[5612]: time="2025-03-05T21:29:50.263107519-05:00" level=info msg="Ignoring extra error returned from registry" error="unauthorized: authentication required"
Mar 05 21:30:45 trinity systemd-coredump[2130139]: [?] Process 5612 (dockerd) of user 0 dumped core.
░░ Subject: Process 5612 (dockerd) dumped core
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ Documentation: man:core(5)
░░
░░ Process 5612 (dockerd) crashed and dumped core.
░░
░░ This usually indicates a programming error in the crashing program and
░░ should be reported to its vendor as a bug.
Mar 05 21:30:45 trinity systemd[1]: docker.service: Main process exited, code=dumped, status=11/SEGV

I'm thinking this is a bug of some sort, but I'm not sure of the next steps to trace.  It doesn't always happen.. just once in a while, and always when I'm deploying something.

First computer has docker on nvme drive.  Second computer has on ssd
Both have daemon.json configured with:

{
    "data-root": "/data/docker",
    "log-driver": "local",
    "log-opts": {
        "max-size": "10m",
        "max-file": "10"
    },
    "ip6tables": false,
}

Thanks!

Last edited by tmoore (2025-03-06 02:59:41)

Offline

#2 2025-03-06 06:34:02

partizann
Member
Registered: 2007-07-28
Posts: 10

Re: Strange dockerd crashing bug when pulling images

Have exactly the same issue - crashing docker with core dumped when pulling images.

Offline

#3 2025-03-06 13:44:24

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

I have this issue as well when pulling the Immich server image, full discussion and logs: https://github.com/immich-app/immich/discussions/16648

I also came across this: https://github.com/moby/moby/issues/49513

Offline

#4 2025-03-06 14:18:12

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

Rolling back to docker 27.5.1 fixes this issue for me.  Appears to be a go related bug in docker 28.0.0

Offline

#5 2025-03-06 15:34:24

tmoore
Member
Registered: 2012-06-03
Posts: 8

Re: Strange dockerd crashing bug when pulling images

Thanks for the bugreport link AltyCoder!  That's exactly what's going on.  I'll downgrade my docker as well.  Appreciate the response.

Offline

#6 2025-03-06 22:31:18

meilon
Member
Registered: 2025-03-06
Posts: 2

Re: Strange dockerd crashing bug when pulling images

Same issue, still persists with 28.0.1

Offline

#7 2025-03-07 08:24:27

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

28.0.1 is flagged as out of date and has been for over a week but a new version doesn't appear to be in testing so downgrade to 27.5.1 for the time being and block docker from being upgraded in pacman.conf for the time being.  I also blocked docker-compose and containerd from being upgraded for the time being as well.

Offline

#8 2025-03-07 11:29:01

gromit
Package Maintainer (PM)
From: Germany
Registered: 2024-02-10
Posts: 1,036
Website

Re: Strange dockerd crashing bug when pulling images

@AltyCoder docker 1:28.0.1-1 is in the stable repositories already

Online

#9 2025-03-07 11:47:39

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

@gromit does this fix the go issue?

Offline

#10 2025-03-07 11:53:05

gromit
Package Maintainer (PM)
From: Germany
Registered: 2024-02-10
Posts: 1,036
Website

Re: Strange dockerd crashing bug when pulling images

I don't know, would be nice if you could test it!

Online

#11 2025-03-07 12:58:34

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

It seems to work based on pulling the Immich server image which was causing me issues for the past 2-3 weeks.

I don't see a mention of this issue being fixed in the release notes though: https://docs.docker.com/engine/release-notes/28/

Offline

#12 2025-03-07 22:33:26

meilon
Member
Registered: 2025-03-06
Posts: 2

Re: Strange dockerd crashing bug when pulling images

28.0.1-1 isn't flagged as out of date (that was for the 28.0.0 release) and, as I already said, it isn't fixed. The problem isn't related to any particular image or used file system, not even if pigz is used or not.

The only way to fix this would be either
- Wait for Docker to fix whatever is the problem with building with go 1.24 (see linked github issue above)
- The package maintainers build the package with go 1.23
- Build docker yourself with go 1.23

Going back to 27.5.1 is a workaround to wait out until this root cause is found

Offline

#13 2025-03-08 07:37:17

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

It's fixed, you didn't see my last post

Offline

#14 2025-03-08 08:33:26

insanemal
Member
From: Brisbane/Australia
Registered: 2009-05-05
Posts: 141

Re: Strange dockerd crashing bug when pulling images

Oh how I wish that were true

Mar 08 18:26:13 cosmos systemd-coredump[133316]: [?] Process 556 (dockerd) of user 0 dumped core.
                                                 
                                                 Stack trace of thread 566:
                                                 #0  0x000057ccf461e098 n/a (n/a + 0x0)
                                                 #1  0x000057ccf467cbd9 n/a (n/a + 0x0)
                                                 #2  0x000057ccf4682029 n/a (n/a + 0x0)
                                                 #3  0x000057ccf4679516 n/a (n/a + 0x0)
                                                 #4  0x000057ccf464d625 n/a (n/a + 0x0)
                                                 #5  0x000057ccf464d2df n/a (n/a + 0x0)
                                                 ELF object binary architecture: AMD x86-64

Offline

#15 2025-03-08 11:33:19

midixinga
Member
Registered: 2014-01-18
Posts: 194

Re: Strange dockerd crashing bug when pulling images

It's not fixed, it can't be fixed; I assume archs docker version docker 1:28.0.1-1 (2025-03-06 19:09 UTC) https://archlinux.org/packages/extra/x86_64/docker/ is still build with go 1.24.

so this docker version isn't usable in my case

after 15 minutes (time is random)

Mar 08 12:16:36 <host> systemd-coredump[8797]: Process 386 (dockerd) of user 0 terminated abnormally with signal 11/SEGV, processing...
Mar 08 12:16:39 <host> systemd-coredump[8798]: Process 386 (dockerd) of user 0 dumped core
Mar 08 12:16:39 <host> systemd[1]: docker.service: Main process exited, code=dumped, status=11/SEGV
Mar 08 12:16:39 <host> systemd[1]: docker.service: Failed with result 'core-dump

Last edited by midixinga (2025-03-09 10:54:03)

Offline

#16 2025-03-09 21:05:13

AltyCoder
Member
Registered: 2025-03-06
Posts: 7

Re: Strange dockerd crashing bug when pulling images

My docker stacks have just randomly crashed after ~48 hours of no issues so rolling back to 27.5.1. Again.

Offline

#17 2025-03-19 16:50:56

DragonX256
Member
Registered: 2015-11-07
Posts: 9

Re: Strange dockerd crashing bug when pulling images

My god, I'm not the only one. This issue is still relevant.

Offline

#18 2025-03-19 17:25:52

TheLovinator
Member
Registered: 2025-03-14
Posts: 1

Re: Strange dockerd crashing bug when pulling images

DragonX256 wrote:

My god, I'm not the only one. This issue is still relevant.

You can do this:

wget https://archive.archlinux.org/packages/d/docker/docker-1%3A27.5.1-1-x86_64.pkg.tar.zst
sudo pacman -U docker-1:27.5.1-1-x86_64.pkg.tar.zst
sudo systemctl restart docker

Offline

#19 2025-03-19 21:46:37

tmoore
Member
Registered: 2012-06-03
Posts: 8

Re: Strange dockerd crashing bug when pulling images

If you use yay, I advise to install the "downgrade" app.. It's great.

Offline

#20 2025-03-21 08:19:47

GaaH2
Member
Registered: 2022-01-08
Posts: 4

Re: Strange dockerd crashing bug when pulling images

TheLovinator wrote:
DragonX256 wrote:

My god, I'm not the only one. This issue is still relevant.

You can do this:

wget https://archive.archlinux.org/packages/d/docker/docker-1%3A27.5.1-1-x86_64.pkg.tar.zst
sudo pacman -U docker-1:27.5.1-1-x86_64.pkg.tar.zst
sudo systemctl restart docker

And do not forget to add

IgnorePkg   = docker

to your pacman.conf

Offline

#21 2025-03-21 16:14:08

doums
Member
Registered: 2019-03-15
Posts: 29

Re: Strange dockerd crashing bug when pulling images

Hi, just sharing, same issue here. Since the last few days I was experiencing these strange crashes/segv with docker `28.0.1`

systemd-coredump[1136133]: [?] Process 744 (dockerd) of user 0 dumped core.

                                                Stack trace of thread 264286:
                                                #0  0x0000570338fe6098 n/a (/usr/bin/dockerd + 0x39098)
                                                #1  0x0000570339044bd9 n/a (/usr/bin/dockerd + 0x97bd9)
                                                #2  0x000057033904a029 n/a (/usr/bin/dockerd + 0x9d029)
                                                #3  0x0000570339041516 n/a (/usr/bin/dockerd + 0x94516)
                                                #4  0x0000570339015625 n/a (/usr/bin/dockerd + 0x68625)
                                                #5  0x00005703390152df n/a (/usr/bin/dockerd + 0x682df)
                                                ELF object binary architecture: AMD x86-64
systemd[1]: docker.service: Main process exited, code=dumped, status=11/SEGV
systemd[1]: docker.service: Failed with result 'core-dump'.

Last edited by doums (2025-03-21 16:16:29)

Offline

#22 2025-03-21 19:23:34

loqs
Member
Registered: 2014-03-06
Posts: 18,380

Re: Strange dockerd crashing bug when pulling images

https://github.com/moby/moby/issues/495 … 2744164263

If I could reproduce it, I would git-bisect golang between v1.23.0 to v1.24.0 and see where it breaks.

Is anyone who can reproduce the issue bisecting or need help with that?

Offline

#23 Today 11:58:11

mathes.s
Member
Registered: 2021-03-05
Posts: 4

Re: Strange dockerd crashing bug when pulling images

Hi,

same issue here with docker just crashing without any idea why. Just tried the downgrade now and will see if it fixes things. The onyl thing I can identify is that it is happening after the full hour, e.g. 00:00, 12:00 and so on.

är 26 12:00:29 server systemd[1]: var-lib-docker-overlay2-26d3642ca944c0d77aab36599da5283dac61f3bb5bd54a8a66f561495cbcff90-merged.mount: Deactivated successfully.
Mär 26 12:00:33 server systemd[1]: var-lib-docker-overlay2-407420de88acb87d7d07fff2b05dd2a7d46fb52fed11047f20fb5e37d1f44e54-merged.mount: Deactivated successfully.
Mär 26 12:00:34 server systemd-coredump[506099]: Process 399751 (dockerd) of user 0 terminated abnormally with signal 11/SEGV, processing...
Mär 26 12:00:34 server systemd[1]: Started Process Core Dump (PID 506099/UID 0).
Mär 26 12:00:35 server systemd-coredump[506100]: [?] Process 399751 (dockerd) of user 0 dumped core.
                                                  
                                                  Stack trace of thread 456192:
                                                  #0  0x00005db1ef01e098 n/a (/usr/bin/dockerd + 0x39098)
                                                  #1  0x00005db1ef07cbd9 n/a (/usr/bin/dockerd + 0x97bd9)
                                                  #2  0x00005db1ef082029 n/a (/usr/bin/dockerd + 0x9d029)
                                                  #3  0x00005db1ef079516 n/a (/usr/bin/dockerd + 0x94516)
                                                  #4  0x00005db1ef04d625 n/a (/usr/bin/dockerd + 0x68625)
                                                  #5  0x00005db1ef04d2df n/a (/usr/bin/dockerd + 0x682df)
                                                  ELF object binary architecture: AMD x86-64
Mär 26 12:00:35 server systemd[1]: systemd-coredump@1-506099-0.service: Deactivated successfully.
Mär 26 12:00:35 server systemd[1]: systemd-coredump@1-506099-0.service: Consumed 799ms CPU time, 384.8M memory peak.
Mär 26 12:00:35 server systemd[1]: docker.service: Main process exited, code=dumped, status=11/SEGV
Mär 26 12:00:35 server systemd[1]: docker.service: Failed with result 'core-dump'.

BG Mathias

Offline

Pages: 1

Board footer

Powered by FluxBB