Arch Linux

cmtonkinson

Member

Registered: 2013-11-15

Posts: 18

[solved] docker: Error mounting: invalid argument

I'm relatively new to using docker. When I had initially set it up (weeks ago) I was able to pull and manipulate containers. Today I came back to my docker project and now I can't even successfully pull an image.

$ docker pull ubuntu:14.04
ubuntu:14.04: The image you are pulling has been verified
511136ea3c5a: Download complete 
511136ea3c5a: Error downloading dependent layers 
d497ad3926c8: Downloading [============================>                      ] 115.7 MB/201.6 MB 7s
e791be0477f2: Download complete 
3680052c0f5c: Download complete 
22093c35d77b: Download complete 
5506de2b643b: Download complete 
5506de2b643b: Error pulling image (14.04) from ubuntu, endpoint: https://registry-1.docker.io/v1/, Error mounting '/dev/mapper/dock5506de2b643b: Error pulling image (14.04) from ubuntu, Error mounting '/dev/mapper/docker-8:98-380441-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/docker/devicemapper/mnt/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d26982014/11/14 20:14:12 Error pulling image (14.04) from ubuntu, Error mounting '/dev/mapper/docker-8:98-380441-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/docker/devicemapper/mnt/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158': invalid argument

This is the log:

$ journalctl -e
Nov 14 20:24:47 example.com docker[18225]: 2014/11/14 20:24:47 docker daemon: 1.3.1 4e9bbfa; execdriver: native; graphdriver:
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] +job serveapi(fd://)
Nov 14 20:24:47 example.com docker[18225]: [info] Listening for HTTP on fd ()
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] +job init_networkdriver()
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] -job init_networkdriver() = OK (0)
Nov 14 20:24:47 example.com docker[18225]: 2014/11/14 20:24:47 WARNING: Your kernel does not support cgroup swap limit.
Nov 14 20:24:47 example.com docker[18225]: [info] Loading containers:
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 07b2a034632f0ddb8f01ebdcf1c0cc4ba0ade7a6d927b544754b44aa3b58f877: open /var/lib/docker/containers/07b2a034632f0ddb8f01ebdcf1c0cc4ba0ade7a6d927b544754b44aa3b58f877/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 0bb0b50393dd951caeaa8386eb8ec8be1cc60e0822167ebed4dfdce9724b618c: open /var/lib/docker/containers/0bb0b50393dd951caeaa8386eb8ec8be1cc60e0822167ebed4dfdce9724b618c/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 15732a4b2a97b91a2c590cab260f91dde7708b50059942221d349a778f94c892: open /var/lib/docker/containers/15732a4b2a97b91a2c590cab260f91dde7708b50059942221d349a778f94c892/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 17b4471d4b031cd24feba6a2dc46e1524bcd9a6bb3b03342a642a5968caff3fa: open /var/lib/docker/containers/17b4471d4b031cd24feba6a2dc46e1524bcd9a6bb3b03342a642a5968caff3fa/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 20686bda195d61f2c76c3e3c809f8b07df9b830d8bc650e10593e1863c82b7f3: open /var/lib/docker/containers/20686bda195d61f2c76c3e3c809f8b07df9b830d8bc650e10593e1863c82b7f3/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 2fe7db68f38445c8ac0124be87174f66ccd1729880c5fa94b93d77ac0aa547fe: open /var/lib/docker/containers/2fe7db68f38445c8ac0124be87174f66ccd1729880c5fa94b93d77ac0aa547fe/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 31b34a085e90849bcffd50c5f7b41d93849d494913448a92849c5d621555251d: open /var/lib/docker/containers/31b34a085e90849bcffd50c5f7b41d93849d494913448a92849c5d621555251d/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 3484c3c05947051eea53865c973100d493e7ec495b3d7a5d4fbc1e5096196733: open /var/lib/docker/containers/3484c3c05947051eea53865c973100d493e7ec495b3d7a5d4fbc1e5096196733/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 3ed275efb25789d6c9e18bf36e46f437fbdb21f3ffd7a93cb4525610737441f0: open /var/lib/docker/containers/3ed275efb25789d6c9e18bf36e46f437fbdb21f3ffd7a93cb4525610737441f0/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 4503890c72d2f728e315b6ecae05fbabba85a3f959898a52df800f6fa22ca094: open /var/lib/docker/containers/4503890c72d2f728e315b6ecae05fbabba85a3f959898a52df800f6fa22ca094/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 4db4ed2fbac1b442d770bcd325937508d2b190caa766f4ddef4f21628a953ea2: open /var/lib/docker/containers/4db4ed2fbac1b442d770bcd325937508d2b190caa766f4ddef4f21628a953ea2/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 56f36a11482d6b9c513cbb819d6c67a768bd47dc4c02bd0de62c61365b6ba363: open /var/lib/docker/containers/56f36a11482d6b9c513cbb819d6c67a768bd47dc4c02bd0de62c61365b6ba363/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 5b254c1c2344f61ab6bf6b338b22150cccd6ebd04407f3d00d6b5b3e37810d4f: open /var/lib/docker/containers/5b254c1c2344f61ab6bf6b338b22150cccd6ebd04407f3d00d6b5b3e37810d4f/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 73fd6f0edadcf67fcdad7a397784275db013df085e648230b5818f0e68e3f48c: open /var/lib/docker/containers/73fd6f0edadcf67fcdad7a397784275db013df085e648230b5818f0e68e3f48c/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 7ad8ad079db979b156af858cd36b908434157efd43e8cbc13677fa287bdf4da0: open /var/lib/docker/containers/7ad8ad079db979b156af858cd36b908434157efd43e8cbc13677fa287bdf4da0/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 82a2e14c00dd73e2df1d78c45a0e576d52637c5dc422c32182ded365d6893167: open /var/lib/docker/containers/82a2e14c00dd73e2df1d78c45a0e576d52637c5dc422c32182ded365d6893167/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 86572f605d7990f173875853bcfbdfe3f909c9b26f439378ab73d561e0e735eb: open /var/lib/docker/containers/86572f605d7990f173875853bcfbdfe3f909c9b26f439378ab73d561e0e735eb/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container 9eecdb0bf364f80cc88532832221da9de86c10a31b24660fdce8b68c79504a88: open /var/lib/docker/containers/9eecdb0bf364f80cc88532832221da9de86c10a31b24660fdce8b68c79504a88/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container a0920424bc83bffbc980f39f6dad17c865f84de1c3508a01d3c1298870f608c1: open /var/lib/docker/containers/a0920424bc83bffbc980f39f6dad17c865f84de1c3508a01d3c1298870f608c1/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container be228e991ffba0d423bb8bdca22fda9e318b7084bae53d765212bed6612e6b01: open /var/lib/docker/containers/be228e991ffba0d423bb8bdca22fda9e318b7084bae53d765212bed6612e6b01/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container e01b5a18f7c252f9c06e6d2dc9720baa184bc803e52aa68e9253372854382213: open /var/lib/docker/containers/e01b5a18f7c252f9c06e6d2dc9720baa184bc803e52aa68e9253372854382213/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container f00aff0df390b90f0a739430b4896cd67afc256a1c657271caf989fd3791ac91: open /var/lib/docker/containers/f00aff0df390b90f0a739430b4896cd67afc256a1c657271caf989fd3791ac91/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .[error] daemon.go:321 Failed to load container f271529997f7479b5b6f65ccdd21d6b91445bbb31a3b253e95b1cf6ad6468c37: open /var/lib/docker/containers/f271529997f7479b5b6f65ccdd21d6b91445bbb31a3b253e95b1cf6ad6468c37/config.json: no such file or directory
Nov 14 20:24:47 example.com docker[18225]: .......................[info] : done.
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] +job acceptconnections()
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] -job acceptconnections() = OK (0)
Nov 14 20:24:47 example.com docker[18225]: [info] POST /v1.15/images/create?fromImage=ubuntu%3A14.04
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] +job pull(ubuntu, 14.04)
Nov 14 20:24:47 example.com docker[18225]: [0687b0f9] +job trust_update_base()
Nov 14 20:24:48 example.com docker[18225]: [0687b0f9] -job trust_update_base() = OK (0)
Nov 14 20:24:48 example.com docker[18225]: [0687b0f9] +job trust_key_check(/library/ubuntu)
Nov 14 20:24:48 example.com docker[18225]: [0687b0f9] -job trust_key_check(/library/ubuntu) = OK (0)
Nov 14 20:24:48 example.com kernel: device-mapper: thin: Creation of new snapshot 0 of device 0 failed.
Nov 14 20:24:48 example.com kernel: JBD2: no valid journal superblock found
Nov 14 20:24:48 example.com kernel: EXT4-fs (dm-1): error loading journal
Nov 14 20:24:48 example.com kernel: JBD2: no valid journal superblock found
Nov 14 20:24:48 example.com kernel: EXT4-fs (dm-1): error loading journal
Nov 14 20:24:52 example.com systemd-udevd[298]: error: /dev/dm-1: No such device or address
Nov 14 20:24:53 example.com docker[18225]: [error] pull.go:151 Error from V2 registry: Error mounting '/dev/mapper/docker-8:98-380441-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/docker/devicemapper/mnt/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158': invalid argument
Nov 14 20:24:57 example.com kernel: JBD2: no valid journal superblock found
Nov 14 20:24:57 example.com kernel: EXT4-fs (dm-1): error loading journal
Nov 14 20:24:57 example.com kernel: JBD2: no valid journal superblock found
Nov 14 20:24:57 example.com kernel: EXT4-fs (dm-1): error loading journal
Nov 14 20:25:02 example.com systemd-udevd[298]: error: /dev/dm-1: No such device or address
Nov 14 20:25:03 example.com docker[18225]: Error pulling image (14.04) from ubuntu, Error mounting '/dev/mapper/docker-8:98-380441-511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158' on '/var/lib/docker/devicemapper/mnt/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158': invalid argument
Nov 14 20:25:03 example.com docker[18225]: [0687b0f9] -job pull(ubuntu, 14.04) = ERR (1)
Nov 14 20:25:03 example.com docker[18225]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 14 20:25:03 example.com docker[18225]: [signal 0xb code=0x1 addr=0x20 pc=0x6bb9ac]
Nov 14 20:25:03 example.com docker[18225]: goroutine 58 [running]:
Nov 14 20:25:03 example.com docker[18225]: runtime.panic(0xa6bc20, 0x1087c93)
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
Nov 14 20:25:03 example.com docker[18225]: bufio.(*Writer).flush(0xc2082228c0, 0x0, 0x0)
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/bufio/bufio.go:530 +0xdc
Nov 14 20:25:03 example.com docker[18225]: bufio.(*Writer).Flush(0xc2082228c0, 0x0, 0x0)
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/bufio/bufio.go:519 +0x39
Nov 14 20:25:03 example.com docker[18225]: net/http.(*response).Flush(0xc20816fa40)
...
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/net/http/transport.go:885 +0x38f
Nov 14 20:25:03 example.com docker[18225]: created by net/http.(*Transport).dialConn
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/net/http/transport.go:601 +0x957
Nov 14 20:25:03 example.com docker[18225]: goroutine 118 [runnable]:
Nov 14 20:25:03 example.com docker[18225]: net/http.(*persistConn).readLoop(0xc2080d7130)
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/net/http/transport.go:868 +0x829
Nov 14 20:25:03 example.com docker[18225]: created by net/http.(*Transport).dialConn
Nov 14 20:25:03 example.com docker[18225]: /usr/lib/go/src/pkg/net/http/transport.go:600 +0x93f

(snipping several dozen lines of stack trace in the middle there)

The search results for various permutations of "docker error mount invalid argument dm" are badly washed out so I wasn't able to find any helpful leads; nor was I able to find anything relevant in the docker issue list. I tried re-installing docker (by running pacman -Rns docker and then pacman -S docker hoping that it was a config setting I could nuke & pave but this had no effect. I then ran the following:

$ dmsetup ls
docker-8:98-380441-pool (254:0)

followed by dmsetup remove docker-8:98-380441-pool and attempted the pull again. No difference.

I'm convinced this is not a docker bug but a misconfiguration of my system, but the error provided ("error mounting .... invalid argument") is rather opaque and I'm stuck with what to try next.

Any suggestions would be appreciated.

Last edited by cmtonkinson (2014-11-15 11:41:47)

firecat53

Member

From: Lake Stevens, WA, USA

Registered: 2007-05-14

Posts: 1,542

Website

Re: [solved] docker: Error mounting: invalid argument

As long as you are not trying to rescue any data from any containers/images, stop the docker daemon (.service and .socket), remove /var/lib/docker and then restart docker. It's possible something didn't update well from the 1.2->1.3 update back in mid-October, but that's just a guess

If that doesn't work, post your 'docker info' and 'docker --version'.

Scott

---

Github

cmtonkinson

Member

Registered: 2013-11-15

Posts: 18

Re: [solved] docker: Error mounting: invalid argument

Well, rm -rf /var/lib/docker did it!

I was unable to remove /var/lib/docker/devicemapper ("device or resource busy") but that doesn't seem to matter - as you suggest docker reconfigured itself. Thanks for the tip!

For the record (should have posted this previously):

$ docker info
Containers: 1
Images: 7
Storage Driver: devicemapper
 Pool Name: docker-8:98-380441-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 598.3 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 1.18 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.17.2-1-ARCH
Operating System: Arch Linux
WARNING: No swap limit support
$ docker --version
Docker version 1.3.1, build 4e9bbfa

firecat53

Member

From: Lake Stevens, WA, USA

Registered: 2007-05-14

Posts: 1,542

Website

Re: [solved] docker: Error mounting: invalid argument

Glad it worked. FYI, you'll see substantially better performance (in my experience at least) using the btrfs or aufs storage backends instead of devicemapper.

Scott

---

Github