[SOLVED] Docker using wrong runc version

josephholten · 2024-06-12 08:52:54

maybe someone could help me with a strange issue I have appearing suddenly after a full system update: building a Dockerfile with `docker build` works flawlessly, but using `docker-compose build` with the same (very simple) Dockerfile throws an error on the very first RUN command in the Dockerfile

Incorrect Usage: flag provided but not defined: -keep
NAME:
    runc run - create and run a container

USAGE:
    runc run [command options] <container-id>

I have to machines running arch linux, I updated one from docker v25 to v26, the other is still on v25. On the machine with the older version (which in other ways also might be slightly different), the same compose script works fine.
As far as I understand, the docker on the updated machine is somehow using the wrong runc version after the update, which doesn't support `-keep`.

`docker version` reports:

 Version:           26.1.4
 API version:       1.45
 Go version:        go1.22.3
 Git commit:        5650f9b102
 Built:             Thu Jun  6 18:42:55 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.22.3
  Git commit:       de5c9cf0b9
  Built:            Thu Jun  6 18:42:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.18
  GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e.m
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

where the runc version does not match the version reported by `runc --version` which says

spec: 1.0.2-dev
go: go1.21.6
libseccomp: 2.5.5

On the non updated machine, the runc version is reported correctly by docker.
I have even tried to run `runc run -keep` which doesn't give the error flag provided but not defined, but when running `runc run -blabla` it does. Hence, by the globally installed runc, `-keep` seems to be recognized as a valid flag.

I have also tried to install docker and docker-compose in a fresh arch virtual machine, where it works flawlessly and the runc version is reported correctly. Hence, the arch package itself seems not to be broken, only my install.

My idea was maybe to manually override the container runtime with /sbin/runc which seems to have the flag, or somehow find out which runc docker is using.

As an info, the arch package versions are:

docker 1:26.1.4-1
docker-compose 2.27.1-1

Last edited by josephholten (2024-06-12 11:52:52)

Lone_Wolf · 2024-06-12 09:29:02

runc last update was around january and man runc doesn't mention keep .

For clarity :
docker build works on both systems with that file but docker-compose only works on the slightly outdated system ?

josephholten · 2024-06-12 09:45:33

Thanks for your quick reply! That is an interesting point about man. docker build works on both, docker-compose only on the slightly outdated machine. I have just double checked.

You can find the source code under https://github.com/josephholten/chat.git if you want an example. Just make sure to clone it with "--recursive".

Lone_Wolf · 2024-06-12 10:01:01

I don't use docker myself, so others will need to help out.

Please run pacman -Qkk docker-compose on both systems and post the outputs if they don't match.

josephholten · 2024-06-12 10:04:35

I don't have access to my outdated machine right now, first tonight again. I'll post the results then.
For the updated machine it's:

docker-compose: 11 total files, 0 altered files

gromit · 2024-06-12 10:26:20

I suspect that either runc or docker compose might be not installed via repo package, you could for example check the output of "type -a docker-compose" contains paths that are in "/home/joseph/.local" etc.

Also note that docker-compose itself is usually just a wrapper to the compose plugin available under "docker compose". The difference is that v1 (docker-compose) is written in python and v2 was written in go, hence my pip guess further up. Besides that also make sure to check that both machines are fully updated ("pacman -Syu").

gromit · 2024-06-12 10:27:44

Re-reading the issue maybe also do the above checks for runc .. Maybe you "go install"ed something..?

Please also posts the results here so we can interpret them with you!

Last edited by gromit (2024-06-12 10:28:04)

josephholten · 2024-06-12 10:44:51

That was precisely the issue! I didn't know the command 'type -a ...' super helpful though for finding conflicts.
I get:

type -a runc

/sbin/runc
/bin/runc
/usr/local/bin/runc
/usr/bin/runc
/usr/sbin/runc

And the bad-guy is just as you suspected /usr/local/bin/runc which says it's version 1.0.2! The culprit.
In fact now I vaguely remember trying out docker as non-root on the updated machine, which I might have installed in some strange way. That must have been the issue.

Arch Linux

#1 2024-06-12 08:52:54

[SOLVED] Docker using wrong runc version

#2 2024-06-12 09:29:02

Re: [SOLVED] Docker using wrong runc version

#3 2024-06-12 09:45:33

Re: [SOLVED] Docker using wrong runc version

#4 2024-06-12 10:01:01

Re: [SOLVED] Docker using wrong runc version

#5 2024-06-12 10:04:35

Re: [SOLVED] Docker using wrong runc version

#6 2024-06-12 10:26:20

Re: [SOLVED] Docker using wrong runc version

#7 2024-06-12 10:27:44

Re: [SOLVED] Docker using wrong runc version

#8 2024-06-12 10:44:51

Re: [SOLVED] Docker using wrong runc version

Board footer