[SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

klaim · 2016-02-24 22:52:14

Hi, I failed to find any info anywhere about this issue so here I am.

I first reported the issue to docker people who said it might be an incompatibility with Go and to point it to you in case it's the problem (they are investigating my issue too).

Docker's github issue: https://github.com/docker/docker/issues/20653

To summarize: I have a fresh ArchLinux install, almost nothing going on. I install docker through. I run it's hello world test. It fails. I discovered this while trying to install Discourse, but apparently it's Docker or devmapper which fails to "register layer".
It looks like this:

╰─$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world

03f4658f8b78: Extracting [==================================================>]    601 B/601 B
a3ed95caeb02: Download complete
docker: failed to register layer: devmapper: Error mounting '/dev/mapper/docker-8:2-6555021-f67bdeaadec34a41c7a3e2d
fc70e6cad1b15832966fd060c7f57295dcd6ecd98' on '/var/lib/docker/devicemapper/mnt/f67bdeaadec34a41c7a3e2dfc70e6cad1b1
5832966fd060c7f57295dcd6ecd98': invalid argument.
See 'docker run --help'.

I tried it after reinstalling completely docker, still fails.
I tried to restart the daemon with no success.

Any idea?

(not sure if it's the right subforum, I can't find one which is about specific packages)

Last edited by klaim (2016-02-27 01:37:02)

firecat53 · 2016-02-25 00:19:31

A few things to try (first make sure your user is in the docker group or you are running the docker commands with sudo or as root):

1. Stop Docker, delete /var/lib/docker, restart and then try pulling the image again. Also try another common image such as ubuntu or busybox. Frequently just wiping /var/lib/docker does the trick.

2. Stop Docker, then start from the command line (see /usr/lib/systemd/system/docker.service for the normal options), but set the storage driver option to overlayfs (default is devicemapper) with '-s overlayfs' and run from the command line. Try the image pulls again and see if that changes anything.

Scott

Last edited by firecat53 (2016-02-25 00:21:09)

klaim · 2016-02-26 01:21:51

Thanks for your time!

firecat53 wrote:

A few things to try (first make sure your user is in the docker group or you are running the docker commands with sudo or as root):

My login is part of the docker group and I use sudo too.

1. Stop Docker, delete /var/lib/docker, restart and then try pulling the image again. Also try another common image such as ubuntu or busybox. Frequently just wiping /var/lib/docker does the trick.

I did that already before because it was suggested in some documentations or some answers to other issues.
It was not a success.
I just tried again after updating everything (because there is a new version of the docker package that someone pointed on github that it could have fixed it).
It was not a success either.

2. Stop Docker, then start from the command line (see /usr/lib/systemd/system/docker.service for the normal options), but set the storage driver option to overlayfs (default is devicemapper) with '-s overlayfs' and run from the command line. Try the image pulls again and see if that changes anything.
Scott

If I understand correctly, the command I must run is

 /usr/bin/docker daemon -H fd://

If I run just this (with sudo) I get

 FATA[0000] No sockets found

If I add '-s overlayfs' I get the same result.

I tried:

 
╰─$ sudo docker daemon -s overlayfs                                                                                               1 ↵
ERRO[0000] Failed to GetDriver graph overlayfs /var/lib/docker
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported

Then I tried to modify the .service file by just adding these arguments to the original start command.
I started docker through systemctl, then got this error:

╰─$ sudo systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

'journalctl -u docker' shows:

Feb 26 02:24:12 klaimslab systemd[1]: Starting Docker Application Container Engine...
Feb 26 02:24:12 klaimslab docker[27819]: time="2016-02-26T02:24:12.793512136+01:00" level=error msg="Failed to GetDriver graph overlayfs /var/lib/docker"
Feb 26 02:24:12 klaimslab docker[27819]: time="2016-02-26T02:24:12.793637730+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: driver not supported"
Feb 26 02:24:12 klaimslab systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Feb 26 02:24:12 klaimslab systemd[1]: Failed to start Docker Application Container Engine.
Feb 26 02:24:12 klaimslab systemd[1]: docker.service: Unit entered failed state.
Feb 26 02:24:12 klaimslab systemd[1]: docker.service: Failed with result 'exit-code'.

I return docker's service file to it's initial state now.

Basically, no progress as overlayfs seems to not work as is with docker.

Last edited by klaim (2016-02-26 01:27:51)

klaim · 2016-02-26 01:39:16

It was suggested on GitHub issue comments that I try the very new version which fix a compatibility issue with the Go version used by docker, but it seems unrelated to my problem which persists: https://github.com/docker/docker/issues/20653

firecat53 · 2016-02-26 03:04:28

Sorry, I didn't test before I wrote The overlay module needs to be loaded first. Although I'm still not sure why the default devicemapper storage driver isn't working for you. What filesystem are you using? I can't test the overlay storage driver because I'm using BTRFS and that's not supported. You need to be using something like ext4 as the 'backing filesystem'.

Try this first and see what errors it throws:

sudo docker daemon -s devicemapper

Then

sudo modprobe overlay
sudo docker daemon -s overlay

Scott

klaim · 2016-02-26 21:44:55

firecat53 wrote:

Sorry, I didn't test before I wrote The overlay module needs to be loaded first. Although I'm still not sure why the default devicemapper storage driver isn't working for you. What filesystem are you using?

`less /etc/fstab`:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/sda2       /       ext4    errors=remount-ro       0       1
/dev/sda3       /home   ext4    defaults        1       2
/dev/sda4       swap    swap    defaults        0       0
proc            /proc   proc    defaults                0       0
sysfs           /sys    sysfs   defaults                0       0
tmpfs           /dev/shm        tmpfs   defaults        0       0
devpts          /dev/pts        devpts  defaults        0       0

I know almost nothing about linux filesystems, unfortunately.

I can't test the overlay storage driver because I'm using BTRFS and that's not supported. You need to be using something like ext4 as the 'backing filesystem'.

I do not know how to check the 'backing filesystem'?

Try this first and see what errors it throws:
sudo docker daemon -s devicemapper

Looks like argument is ignored:

╰─$ sudo docker daemon -s devicemapper
WARN[0000] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
WARN[0000] devmapper: Base device already exists and has filesystem xfs on it. User specified filesystem  will be ignored.
INFO[0000] Graph migration to content-addressability took 0.00 seconds
WARN[0000] Running modprobe nf_nat failed with message: `modprobe: WARNING: Module nf_nat not found in directory /lib/modules/3.14.32-xxxx-grs-ipv6-64`, error: exit status 1
INFO[0000] Firewalld running: false
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
WARN[0000] Your kernel does not support kernel memory limit.
WARN[0000] Your kernel does not support cgroup cfs period
WARN[0000] Your kernel does not support cgroup cfs quotas
INFO[0000] Loading containers: start.

INFO[0000] Loading containers: done.
INFO[0000] Daemon has completed initialization
INFO[0000] Docker daemon                                 commit=c3959b1 execdriver=native-0.2 graphdriver=devicemapper version=1.10.2
INFO[0000] API listen on /var/run/docker.sock

Then

sudo modprobe overlay
sudo docker daemon -s overlay

Scott

╰─$ sudo modprobe overlay                                                                                                         1 ↵
modprobe: FATAL: Module overlay not found in directory /lib/modules/3.14.32-xxxx-grs-ipv6-64

I have no idea what this means....

╭─klaim@klaimslab  ~
╰─$ sudo modprobe overlay                                                                                                         1 ↵
modprobe: FATAL: Module overlay not found in directory /lib/modules/3.14.32-xxxx-grs-ipv6-64
╭─klaim@klaimslab  ~
╰─$ sudo modprobe overlayfs                                                                                                       1 ↵
modprobe: FATAL: Module overlayfs not found in directory /lib/modules/3.14.32-xxxx-grs-ipv6-64
╭─klaim@klaimslab  ~
╰─$ sudo pacman -S overlayfs                                                                                                      1 ↵
error: target not found: overlayfs
╭─klaim@klaimslab  ~
╰─$ sudo docker daemon -s overlay                                                                                                 1 ↵
ERRO[0000] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported
╭─klaim@klaimslab  ~
╰─$ sudo docker daemon -s overlayfs                                                                                               1 ↵
ERRO[0000] Failed to GetDriver graph overlayfs /var/lib/docker
FATA[0000] Error starting daemon: error initializing graphdriver: driver not supported

Well.....halp?

Last edited by klaim (2016-02-26 22:30:33)

firecat53 · 2016-02-26 23:04:01

Wow, what kernel are you using?? That might be part of the problem. You're missing the 'nf_nat' module (see the error for devicemapper), and the kernel you're using doesn't have the overlayfs module available. You're going to have to either switch kernels (stock Arch kernel works fine) or learn how to compile a kernel with the correct modules

You're using ext4 which is fine for use with devicemapper or overlayfs.

Scott

klaim · 2016-02-26 23:10:25

firecat53 wrote:

Wow, what kernel are you using?? That might be part of the problem. You're missing the 'nf_nat' module (see the error for devicemapper), and the kernel you're using doesn't have the overlayfs module available. You're going to have to either switch kernels (stock Arch kernel works fine) or learn how to compile a kernel with the correct modules

I have no idea why there would be an unusual kernel, but this is a Kimsuffi (OVH) box so maybe the inital image was modified. Onced installed I upgraded the whole thing but never thought about upgrading a kernel (I thought it was automatic with pacman upgrade?)
Anyway, I'll look for how to switch kernel then.

klaim · 2016-02-26 23:19:45

This is ... "fun":

╭─klaim@klaimslab  ~
╰─$ sudo pacman -Qi linux
error: package 'linux' was not found
╭─klaim@klaimslab  ~
╰─$ sudo pacman -Q | grep linux                                                                                                   1 ↵
archlinux-keyring 20160215-1
libutil-linux 2.27.1-1
linux-api-headers 4.4.1-1
util-linux 2.27.1-1

I don't know what is happening. Still searching for how to change kernel.

klaim · 2016-02-27 00:51:08

OK I fixed it.

It was indeed the kernel version.
I had to learn on-the-fly how to install an image and then how to setup grub to actually boot with it (I didn't know it was grub doing this!!!!)

Now I have:

╭─klaim@klaimslab ~
╰─$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
03f4658f8b78: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:8be990ef2aeb16dbcb9271ddfe2610fa6658d13f6dfb8bc72074cc1ca36966a7
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/userguide/

I will now proceed to try to (finally!!) install my discourse instance.

BTW, would there be a way to automatically detect this kind of issues better in the future?

firecat53 · 2016-02-27 01:34:22

Glad you fixed it. Only experience will help find problems like that. Obviously I didn't key in on it right away either :-)
Make sure you update your github issue and edit the title of your first post hrre to reflect that problem is solved.

Scott

Arch Linux

#1 2016-02-24 22:52:14

[SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#2 2016-02-25 00:19:31

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#3 2016-02-26 01:21:51

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#4 2016-02-26 01:39:16

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#5 2016-02-26 03:04:28

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#6 2016-02-26 21:44:55

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#7 2016-02-26 23:04:01

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#8 2016-02-26 23:10:25

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#9 2016-02-26 23:19:45

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#10 2016-02-27 00:51:08

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

#11 2016-02-27 01:34:22

Re: [SOLVED] Docker's Hello-World test fails after fresh ArchLinux install

Board footer