Arch Linux

Harey

Member

From: Bavaria, Germany

Registered: 2007-03-24

Posts: 374

docker container fails on kernel 4.19.1

There seems to be a kernel problem with docker and 4.19.1

I have 2 docker container I want to test (postgres:latest and mayan-edms:latest )

The process involves the use of 'apt install' inside the container to install some additional tesseract language data files for the german language. This is accomplished by the docker run command:

docker run -d --name mayan-edms --restart=always -p 81:8000 \
-e MAYAN_DATABASE_ENGINE=django.db.backends.postgresql \
-e MAYAN_DATABASE_HOST=172.17.0.1 \
-e MAYAN_DATABASE_NAME=mayan \
-e MAYAN_DATABASE_PASSWORD=mysupersecret \
-e MAYAN_DATABASE_USER=mayan \
-e MAYAN_DATABASE_CONN_MAX_AGE=60 \
-e MAYAN_APT_INSTALLS="tesseract-ocr-deu tesseract-ocr-deu-frak" \
-v /home/docker-volumes/mayan-edms/media:/var/lib/mayan \
-v /srv/scanned-for-EDMS:/srv/watch_folder \
mayanedms/mayanedms:latest

While the process works on the LTS kernel 4.14 and the current 4.18.16 in core it fails with kernel 4.19.1 from testing.

I have already posted on the mayan-edms forum but I think that this is not a problem of the container but of the kernel konfiguration.

The use of vsyscall=emulate as a kernel parameter is not needed to accomplish the installation on 4.14-lts and core/4.18.16 but setting it does not help either on kernel 4.19.1

The conainer shell is bailing out like this:

root@7cfe63f9d46d:/opt/mayan-edms# apt install tesseract-ocr-deu
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. 
root@7cfe63f9d46d:/opt/mayan-edms# dpkg --configure -a
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link

I have read a lot about the use of vsyscall on the different kernels, maybe 4.19.1 introduces an new build parameter that has to be set?

Greetings
Harvey

---

Linux is like a wigwam: No Gates, no Windows and an Apache inside

willy9

Member

Registered: 2009-04-14

Posts: 56

Re: docker container fails on kernel 4.19.1

I can confirm this on testing... Very basic example:

Dockerfile:

FROM    debian:stretch 
RUN      apt-get -y update && apt-get -y install nano

Result:

sudo docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM    debian:stretch
 ---> f2aae6ff5d89
Step 2/2 : RUN     apt-get -y update && apt-get -y install nano
 ---> Running in 64138311bb3e
Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Get:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:2 http://security-cdn.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Get:4 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB]
Get:5 http://security-cdn.debian.org/debian-security stretch/updates/main amd64 Packages [454 kB]
Get:6 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B]
Get:7 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages [5152 B]
Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 Packages [7089 kB]
Fetched 7855 kB in 7s (1086 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  spell
The following NEW packages will be installed:
  nano
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 485 kB of archives.
After this operation, 2092 kB of additional disk space will be used.
Get:1 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 nano amd64 2.7.4-1 [485 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 485 kB in 0s (622 kB/s)
Selecting previously unselected package nano.
(Reading database ... 6498 files and directories currently installed.)
Preparing to unpack .../nano_2.7.4-1_amd64.deb ...
Unpacking nano (2.7.4-1) ...
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)
The command '/bin/sh -c apt-get -y update && apt-get -y install nano' returned a non-zero code: 100

Would appreciate any hint

---

Skrooge, a personal finances manager for the Plasma Desktop
http://skrooge.org

bresilla

Member

Registered: 2018-11-12

Posts: 1

Re: docker container fails on kernel 4.19.1

I can confirm that too.
I was getting this error while building a Ubuntu docker image on Kernel 4.19.1-1 (Manjaro)

 dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link 

I just rolled back to Kernel 4.18.17-1 and now Docker works fine.

jbenoist

Member

Registered: 2018-11-13

Posts: 1

Re: docker container fails on kernel 4.19.1

Confirming the issue, suspecting this is coming from this change:

commit a725356b6659469d182d662f22d770d83d3bc7b5
Author: Amir Goldstein <amir73il@gmail.com>
Date:   Tue Sep 18 16:34:34 2018 +0300

    vfs: swap names of {do,vfs}_clone_file_range()

    Commit 031a072a0b8a ("vfs: call vfs_clone_file_range() under freeze
    protection") created a wrapper do_clone_file_range() around
    vfs_clone_file_range() moving the freeze protection to former, so
    overlayfs could call the latter.

    The more common vfs practice is to call do_xxx helpers from vfs_xxx
    helpers, where freeze protecction is taken in the vfs_xxx helper, so
    this anomality could be a source of confusion.

    It seems that commit 8ede205541ff ("ovl: add reflink/copyfile/dedup
    support") may have fallen a victim to this confusion -
    ovl_clone_file_range() calls the vfs_clone_file_range() helper in the
    hope of getting freeze protection on upper fs, but in fact results in
    overlayfs allowing to bypass upper fs freeze protection.

    Swap the names of the two helpers to conform to common vfs practice
    and call the correct helpers from overlayfs and nfsd.

Harey

Member

From: Bavaria, Germany

Registered: 2007-03-24

Posts: 374

Re: docker container fails on kernel 4.19.1

There already seems to be a bug report on docker github

---

Linux is like a wigwam: No Gates, no Windows and an Apache inside

Harey

Member

From: Bavaria, Germany

Registered: 2007-03-24

Posts: 374

Re: docker container fails on kernel 4.19.1

Still valid for 4.19.2

---

Linux is like a wigwam: No Gates, no Windows and an Apache inside

Netizen29

Member

Registered: 2018-11-14

Posts: 1

Re: docker container fails on kernel 4.19.1

The problem appears only, afaik, if you use overlay2 as storage driver, which is the default for modern docker installations.
If you need to build images locally, as a workaround, you can change the storage driver to "devicemapper", which is not recommended in production. Further information: Docker Archlinux documentation.

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: docker container fails on kernel 4.19.1

Is the issue present in 4.20-rc2 ?

filotek

Member

Registered: 2018-08-15

Posts: 23

Re: docker container fails on kernel 4.19.1

Ran into this issue today after upgrading kernel to 4.19.1.  Besides the change that Netizen29 suggested, I found a mailing list where Ted Ts'o ran into the same issue...

Someone had suggested that...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

was sufficient to stop the error from happening (https://www.spinics.net/lists/linux-uni … 06313.html)

My understanding is that the /metacopy parameter allows only file attribute info to be copied in an overlayfs when its metadata changes (instead of copying the whole file).

Turning this off might not be ideal, but for my use case it's good enough for the short term until a proper fix can be applied.

/EDIT: grammar

Last edited by filotek (2018-11-14 21:46:08)

progandy

Member

Registered: 2012-05-17

Posts: 5,275

Re: docker container fails on kernel 4.19.1

In the default kernel configuration it is disabled according to https://www.spinics.net/lists/linux-uni … 06319.html

The arch kernel changed that in 4.19.arch1-1

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Harey

Member

From: Bavaria, Germany

Registered: 2007-03-24

Posts: 374

Re: docker container fails on kernel 4.19.1

@Netizen29:
But it worked with overlay2 in the 4.18x series... The error has been introduced with the change to 4.19.x.

---

Linux is like a wigwam: No Gates, no Windows and an Apache inside

jetm

Member

From: Costa Rica

Registered: 2010-12-19

Posts: 34

Re: docker container fails on kernel 4.19.1

There is a fix for the kernel: https://www.spinics.net/lists/linux-uni … 06316.html. Hope it's merged in 4.19.3.

---

Code of conduct | dotfiles

yohanaizraeli

Member

Registered: 2016-09-01

Posts: 3

Re: docker container fails on kernel 4.19.1

I also ran into this problem yesterday. I just downgraded the kernel to linux-4.18.16 as explained in the docs here: https://wiki.archlinux.org/index.php/do … the_kernel

I'll stick to this kernel version while I wait for a fix for sometime.

epinephrine

Member

From: Frankfurt

Registered: 2012-10-18

Posts: 92

Re: docker container fails on kernel 4.19.1

Someone had suggested that...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

was sufficient to stop the error from happening (https://www.spinics.net/lists/linux-uni … 06313.html)

I can confirm that this fixes the issue for me! (4.19.1)

modnoob

Member

Registered: 2017-10-17

Posts: 29

Re: docker container fails on kernel 4.19.1

Same Problem for me:
FROM debian:stretch
RUN apt-get update && apt-get install -y postfix dovecot-mysql

Anyone reported it on Arch-Bugtracker?

jocke-l

Member

Registered: 2018-11-17

Posts: 1

Re: docker container fails on kernel 4.19.1

Someone had suggested that...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

was sufficient to stop the error from happening (https://www.spinics.net/lists/linux-uni … 06313.html)

I can confirm that this fixes the issue for me! (4.19.1)

This works for me as well with 4.19.1-arch1-1-ARCH

wopfel

Member

Registered: 2013-11-13

Posts: 16

Re: docker container fails on kernel 4.19.1

Someone had suggested that...

echo N | sudo tee /sys/module/overlay/parameters/metacopy

was sufficient to stop the error from happening (https://www.spinics.net/lists/linux-uni … 06313.html)

I can confirm that this fixes the issue for me! (4.19.1)

This works for me as well with 4.19.1-arch1-1-ARCH

Same here, works for me in 4.19.1. Thank you for this hint, filotek.

bodolsog

Member

Registered: 2016-03-28

Posts: 2

Re: docker container fails on kernel 4.19.1

echo N | sudo tee /sys/module/overlay/parameters/metacopy

I can confirm that this fixes the issue for me! (4.19.1)

This works for me as well with 4.19.1-arch1-1-ARCH

Same here, works for me in 4.19.1. Thank you for this hint, filotek.

Just want to confirm - solution works also for: 4.19.6.

Last edited by bodolsog (2018-12-12 12:05:47)

loqs

Member

Registered: 2014-03-06

Posts: 18,151

Re: docker container fails on kernel 4.19.1

https://git.kernel.org/pub/scm/linux/ke … 229b4162ce
Queued for 4.19.11