[SOLVED] arch-chroot doesn't exit clean if jobs left running in chroot

starfry · 2015-02-05 14:26:40

I've just tracked down an annoying wierd happening on my system.

I have a build script that does this

arch-chroot $buildroot /bin/bash -c 'pacman-key --init && pacman-key --populate archlinux'

it's always been fine but it now leaves mounts around for buildroot and buildroot/dev.

I have tracked the problem down to the latest incarnation of GnuPG which now starts a "gpg-agent" and leaves it running. It never used to do this.
Once the above command stiring has finished, the chroot exits. But, due to the lingering agent, the unmounts normally performed by "arch-chroot" don't happen, but the chroot exits anyway.

$ arch-chroot buildroot
sh-4.3# pgrep gpg-agent
sh-4.3# pacman-key --init
sh-4.3# pgrep gpg-agent  
7107
sh-4.3# exit
exit
umount: /root/.amylum/elements/sodium/dev: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
umount: /root/.amylum/elements/sodium: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
sh-4.3# exit
$ findmnt -R -o TARGET buildroot
TARGET
/path-to-/buildroot
└─/path-to/buildroot/dev

The existing mounts also breaks any further attempt to enter the chroot with "arch-chroot".

I can fix my specific issue by adding "pkill gpg-agent" to the above command string but it would be good if "arch-chroot" was sensitive to this kind of thing and either refused to leave the chroot or killed of any processes running within it.

Also... while evaluating this, I noticed that it mounts the chroot directory onto itself. I am just curious... why does it do that ?

Last edited by starfry (2015-02-06 09:42:44)

falconindy · 2015-02-06 02:35:36

This should have been fixed, and included as part of v14. What version of arch-chroot is this?

edit to answer question:

starfry wrote:

Also... while evaluating this, I noticed that it mounts the chroot directory onto itself. I am just curious... why does it do that ?

Only when the chroot isn't already a mountpoint. This is done to ensure that all mountpoints within the chroot have a parent mount which is accessible from within the chroot (and that there's a corresponding entry for /).

Without bind mount on the chroot:

207 72 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:152 - proc proc rw
212 72 0:14 / /sys ro,nosuid,nodev,noexec,relatime shared:156 - sysfs sys rw
217 72 0:5 / /dev rw,nosuid,relatime shared:160 - devtmpfs udev rw,size=6147156k,nr_inodes=1536789,mode=755
222 217 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:164 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
227 217 0:41 / /dev/shm rw,nosuid,nodev,relatime shared:168 - tmpfs shm rw
232 72 0:42 / /run rw,nosuid,nodev,relatime shared:172 - tmpfs run rw,mode=755
237 72 0:43 / /tmp rw,nosuid,nodev shared:176 - tmpfs tmp rw
242 72 8:2 /etc/resolv.conf /etc/resolv.conf rw,relatime shared:1 - ext4 /dev/sda2 rw,data=ordered

Notice how mounts mostly have a parent ID (the second column) of 72. This mount isn't visible from within the namespace.

With a bind mount on the chroot:

207 72 0:35 /chroots/extra-x86_64/root / rw,nodev,noatime shared:31 - btrfs /dev/sdc1 rw,compress=lzo,space_cache,autodefrag
212 207 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:155 - proc proc rw
222 207 0:14 / /sys ro,nosuid,nodev,noexec,relatime shared:162 - sysfs sys rw
232 207 0:5 / /dev rw,nosuid,relatime shared:169 - devtmpfs udev rw,size=6147156k,nr_inodes=1536789,mode=755
242 232 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:176 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
252 232 0:41 / /dev/shm rw,nosuid,nodev,relatime shared:183 - tmpfs shm rw
262 207 0:42 / /run rw,nosuid,nodev,relatime shared:190 - tmpfs run rw,mode=755
272 207 0:43 / /tmp rw,nosuid,nodev shared:197 - tmpfs tmp rw
282 207 8:2 /etc/resolv.conf /etc/resolv.conf rw,relatime shared:1 - ext4 /dev/sda2 rw,data=ordered

Now we see a parent ID of 207 for those same mounts. This mount is visible in the chroot.

Having a coherent mount tree at least keeps findmnt happy (which is necessary for genfstab). I don't recall if there were other reasons I did this.

Last edited by falconindy (2015-02-06 03:28:38)

starfry · 2015-02-06 09:40:47

Ah, yes, the system I was using has 13-1 which predates the fix. Now I know why it uses "unshare". I've just updated that package to 14-1 and it is indeed fixed!
(I know I should run an update but I can't risk anything breaking on that box until I finish the task I am working on)

Just FYI - one thing I have noticed with the use of "unshare" is that "arch-chroot" (14-1) doesn't work on ARM. I realise that is probably out of scope for you however. For some reason, its "unshare" doesn't like the "--pid" argument. This happens:

$ arch-chroot /mnt
$ unshare: unshare failed: Invalid argument

I have raised this over on ArchLinuxARM but haven't had any response yet.

Thank you for the explanation about the bind mount. The findmnt thing makes sense.

Marked solved.

falconindy · 2015-02-07 15:58:13

starfry wrote:

Just FYI - one thing I have noticed with the use of "unshare" is that "arch-chroot" (14-1) doesn't work on ARM. I realise that is probably out of scope for you however. For some reason, its "unshare" doesn't like the "--pid" argument. This happens:
$ arch-chroot /mnt
$ unshare: unshare failed: Invalid argument
I have raised this over on ArchLinuxARM but haven't had any response yet.

Seems like a bug worth reporting to util-linux folk (particularly if you can distill it down to only: unshare --fork --pid). It's not really out of scope for me, I'm just without any ARM machine to test things on.

edit: does your kernel have CONFIG_PID_NS=y ?

Last edited by falconindy (2015-02-07 16:17:06)

starfry · 2015-02-07 18:55:54

falconindy wrote:

edit: does your kernel have CONFIG_PID_NS=y ?

Just so happens that I have it running here atm, just did a quick check and...

$ zgrep CONFIG_PID_NS /proc/config.gz 
CONFIG_PID_NS=y
$ unshare --fork --pid ls
unshare: unshare failed: Invalid argument
$ unshare --pid ls
unshare: unshare failed: Invalid argument
$  unshare --fork ls
(files listed as expected)
$ uname -a
Linux xxxxx 3.4.103-5-ARCH #1 SMP PREEMPT Fri Jan 16 06:34:00 MST 2015 armv7l GNU/Linux

I know its a stale kernel but it is the current latest in the repos for allwinner/sunxi.

I have another arm board lying around (a raspberry pi) that will have the latest kernel on it. I'll also dig that out and try it there. If it's still the same then I can join the util-linux list and report it upstream.

**edit** just dug out the Pi. Same problem but it's a 3.6 kernel. I haven't got the time to upgrade it right now. I'll also see if some other people can confirm they get the same result.

Last edited by starfry (2015-02-07 20:06:40)

Arch Linux

#1 2015-02-05 14:26:40

[SOLVED] arch-chroot doesn't exit clean if jobs left running in chroot

#2 2015-02-06 02:35:36

Re: [SOLVED] arch-chroot doesn't exit clean if jobs left running in chroot

#3 2015-02-06 09:40:47

Re: [SOLVED] arch-chroot doesn't exit clean if jobs left running in chroot

#4 2015-02-07 15:58:13

Re: [SOLVED] arch-chroot doesn't exit clean if jobs left running in chroot

#5 2015-02-07 18:55:54

Re: [SOLVED] arch-chroot doesn't exit clean if jobs left running in chroot

Board footer