You are not logged in.

#1 2021-04-02 19:38:28

josephg
Member
From: Brazil
Registered: 2009-09-28
Posts: 106

Rootless Podman won't work since version 3.1.0

Running Podman as rootless will give no output, just occupying the terminal. e.g.:

$ podman pull archlinux:latest 
 

Exception is "podman --help", which works just fine.

I had rootless Podman set and working until I installed the podman 3.1.0, in 2021-03-31. There was also a package update for containers-common 0.35.4-2.

Running Podman as root (default), works fine though.

My configuration is:

  • No file in "~/.config/containers/"

  • "~/.local/share/containers/" stores images and other rootless stuff

  • "/etc/containers/storage.conf" unaltered, except for:

    driver = "overlay"
    rootless_storage_path = "$HOME/.local/share/containers/storage"
  • "/etc/containers/registries.conf" unaltered, except for:

    unqualified-search-registries = ["docker.io"]
  • "/etc/sysctl.d/userns.conf" was created with:

    kernel.unprivileged_userns_clone=1
  • "/etc/default/grub" edited to set "systemd.unified_cgroup_hierarchy" to enable cgroup v2:

    GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet systemd.unified_cgroup_hierarchy=1"
  • Ran 'grub-mkconfig' and restarted after editing the file above, and confirmed cgroup v2 is enabled.

Does anyone know what could be happening?  Any suggestion on how to debug this?

Last edited by josephg (2021-04-04 18:13:51)

Offline

#2 2021-04-04 16:46:07

jonathon
Member
Registered: 2016-09-19
Posts: 128

Re: Rootless Podman won't work since version 3.1.0

josephg wrote:

archlinux:latest

It's working for me, so possibly try:

1. `podman pull docker.io/library/archlinux:latest` to qualify the image name;
2. Check permissions on /etc/containers/registries.conf;
3. Accessing docker.io separately, e.g. browser;
4. `strace -f podman pull` and see if it's hanging on anything obvious (e.g. DNS resolution, ...).

Offline

#3 2021-04-04 18:05:10

josephg
Member
From: Brazil
Registered: 2009-09-28
Posts: 106

Re: Rootless Podman won't work since version 3.1.0

jonathon wrote:
josephg wrote:

archlinux:latest

It's working for me, so possibly try:

1. `podman pull docker.io/library/archlinux:latest` to qualify the image name;
2. Check permissions on /etc/containers/registries.conf;
3. Accessing docker.io separately, e.g. browser;
4. `strace -f podman pull` and see if it's hanging on anything obvious (e.g. DNS resolution, ...).

Thanks for the reply.

1. same result (no output, command line occupied)
2. "rw-r--r--", root:root, for /etc/containers/registries.conf
3. docker.io works fine in browser, redirects to docker.com website; but remember that it works when run as root
4. here is the output: http://ix.io/2V1E; any idea?

Offline

#4 2021-04-04 18:16:05

josephg
Member
From: Brazil
Registered: 2009-09-28
Posts: 106

Re: Rootless Podman won't work since version 3.1.0

NOTE: I just added three entries to the bullet list on the original post to mention "kernel.unprivileged_userns_clone=1" and setting grub for cgroup v2.

Offline

#5 2021-04-04 18:21:30

jonathon
Member
Registered: 2016-09-19
Posts: 128

Re: Rootless Podman won't work since version 3.1.0

There's nothing obvious in there that catches my eye, though the final files to be accessed are /etc/containers/containers.conf and /usr/share/containers/containers.conf (both owned by containers-common). Do you have any unmerged pacnew?

Offline

#6 2021-04-04 18:37:39

josephg
Member
From: Brazil
Registered: 2009-09-28
Posts: 106

Re: Rootless Podman won't work since version 3.1.0

I have no pacnew or pacsave related to podman or containers-common.

Offline

#7 2021-04-04 18:58:58

jonathon
Member
Registered: 2016-09-19
Posts: 128

Re: Rootless Podman won't work since version 3.1.0

My final idea: https://man.archlinux.org/man/podman.1# … tless_mode , e.g.:

[storage]
   driver = "overlay"
   [storage.options]
     mount_program = "/usr/bin/fuse-overlayfs"

Offline

#8 2021-04-04 19:29:34

fenrikur
Member
From: Germany
Registered: 2021-04-04
Posts: 3

Re: Rootless Podman won't work since version 3.1.0

I'm currently facing pretty much the same issue (followed the setup guide for podman on the wiki; setup identical to that described by the OP), but for me, `strace -f podman system info` yields the following line amongst its output while cleaning up child processes towards the end:

[pid   330] write(2, "Error: cannot setup namespace us"..., 61Error: cannot setup namespace using newuidmap: exit status 1
 <unfinished ...>

When calling `podman system info` without strace, the command will simply not return at all and instead consume a considerable amount of CPU cycles, seemingly stuck in a loop.

/etc/subuid and /etc/sub/gid are both set up according to the wiki:

my-foo-username:100000:65536

Edit: The system I am encountering the issue on is a fresh install btw.

$ uname -a
Linux v2202104125665148734 5.11.11-arch1-1 #1 SMP PREEMPT Tue, 30 Mar 2021 14:10:17 +0000 x86_64 GNU/Linux

Last edited by fenrikur (2021-04-04 19:46:35)

Offline

#9 2021-04-04 20:08:42

jonathon
Member
Registered: 2016-09-19
Posts: 128

Re: Rootless Podman won't work since version 3.1.0

It still works absolutely fine for me in a freshly installed QEMU VM. No changes made to the VM except the steps contained in the wiki (it has systemd 248). http://ix.io/2V2i

Offline

#10 2021-04-04 20:22:54

fenrikur
Member
From: Germany
Registered: 2021-04-04
Posts: 3

Re: Rootless Podman won't work since version 3.1.0

jonathon wrote:

It still works absolutely fine for me in a freshly installed QEMU VM. No changes made to the VM except the steps contained in the wiki (it has systemd 248). http://ix.io/2V2i

Thanks for the `podman system info`! Just diffed it with my `sudo podman system info` (since rootless doesn't work), but the only differences I can make out seem to be due to my system not running any containers so far and podman being run as root: http://ix.io/2V2l

Offline

#11 2021-04-04 21:37:10

calebstewart
Member
Registered: 2021-04-04
Posts: 5

Re: Rootless Podman won't work since version 3.1.0

I'm also experiencing problems with rootless podman after a full system upgrade. I removed all configuration, and reinstalled podman and containers-common. Added back the storage driver, and rootless storage path, and now any rootless podman command just hangs. Running with "strace -f" shows that it is stuck in a loop calling "newfsstatat" forever on the CWD.

... SNIP ...

[pid 135445] futex(0xc0006a0150, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 135436] newfstatat(AT_FDCWD, ".",  <unfinished ...>
[pid 135445] <... futex resumed>)       = 1
[pid 135444] <... futex resumed>)       = 0
[pid 135436] <... newfstatat resumed>{st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135445] epoll_pwait(4,  <unfinished ...>
[pid 135444] futex(0xc0006a0150, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0
[pid 135436] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0

... SNIP ...

Just after the upgrade, I was getting warnings that no storage driver was set in `/etc/containers/storage.conf`. I don't recall ever editing that, and don't know why this became a problem now. I set the storage driver to `overlay`, and then received permission denied errors. Lastly, I realized I needed to set the rootless storage path. After setting that, I am now stuck with an infinite loop. I'm currently on `podman` 3.1.0 as well.

Offline

#12 2021-04-04 22:14:23

calebstewart
Member
Registered: 2021-04-04
Posts: 5

Re: Rootless Podman won't work since version 3.1.0

I've been doing some more investigation on the output of "strace", and I think I might have found the problem causing the infinite loop. It appears that, contrary to the documentation, environment variables are not being expanded within the "rootless_storage_path" setting. The infinite loop happens while podman is attempting to trace up the directory path for an existing directory, ending up at ".", and then spiraling into nothingness for some reason.

[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local/share/containers/storage/overlay/.has-mount-program", 0xc00001c928, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local/share/containers/storage/overlay", 0xc00001c9f8, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local/share/containers/storage/overlay", 0xc00001cac8, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local/share/containers/storage", 0xc00001cb98, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local/share/containers", 0xc00001cc68, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local/share", 0xc00001cd38, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME/.local", 0xc00001ce08, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, "$HOME", 0xc00001ced8, 0) = -1 ENOENT (No such file or directory)
[pid 232536] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0711, st_size=4096, ...}, 0) = 0

I can confirm that replacing "$HOME" with my personal home directory fixes the hang, and podman appears to behave, partially. I'm passed the infinite loop, but I'm back to getting "operation not permitted" when trying to pull images:

$ podman pull archlinux
Resolving "archlinux" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/library/archlinux:latest...
Getting image source signatures
Copying blob 5bb50848eab8 done
Copying blob 10756994dc19 done
Copying config 3de742be92 done
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): operation not permitted
Error: Error committing the finished image: error adding layer with blob "sha256:5bb50848eab8d3d80a48b3769ef342097f57881b1ef86826e898c43ee4dd2460": Error processing tar file(exit status 1): operation not permitted

Offline

#13 2021-04-05 00:43:21

fenrikur
Member
From: Germany
Registered: 2021-04-04
Posts: 3

Re: Rootless Podman won't work since version 3.1.0

calebstewart wrote:

I've been doing some more investigation on the output of "strace", and I think I might have found the problem causing the infinite loop. It appears that, contrary to the documentation, environment variables are not being expanded within the "rootless_storage_path" setting. The infinite loop happens while podman is attempting to trace up the directory path for an existing directory, ending up at ".", and then spiraling into nothingness for some reason.

[…]

I can confirm that replacing "$HOME" with my personal home directory fixes the hang, and podman appears to behave, partially.

Nice find! Replaced "$HOME" in my storage.conf with "/tmp" and was indeed able to successfully execute both `podman system info` and `podman pull docker.io/alpine`, so I can confirm your finding concerning the env var not being expanded correctly.

The related go code on GitHub for the version of containers/storage that should have been used for podman 3.1.0 (https://github.com/containers/storage/b … ns.go#L102) has not been changed (only moved) recently though and does seem to implement env var expansion correctly via go's `os.ExpandEnv`. So might be that the issue is not necessarily within podman itself (or one of its dependencies), although the value of version.GitCommit for community/podman 3.1.0 has the "-dirty" suffix, signifying the workspace is not identical to what can be pulled from GitHub. But I don't know enough about the packaging process to say what changes might be necessary befor building for arch.

An interesting finding I made was that env variable expansion seemed to work just fine when using "/tmp/$HOME/…" or "/tmp/$USER/…" for the "rootless_storage_path" setting, whereas it would fail when setting the value to "$HOME/…" as per default. The issue with "$HOME" not being correctly expanded also occurs when using something like e.g. "/tmp/../$HOME/…" or "/./$HOME/…", as both get converted to "/$HOME/…" instead of e.g. "/home/username/…".
What makes the issue especially interesting is, that when using e.g. "/tmp/$USER/…", running a rootless podman command will result in both literally "/tmp/$USER/…" as well as the expanded version "/tmp/username/…" paths being created, although I have only seen actual files being written to the expanded paths. So there seems to be different places in containers/storage, some of which use the expanded and some of which use the literal value stored in "rootless_storage_path".

Haven't had the time to delve deeper into the source code to check whether something might have gone awry between containers/storage v1.28.1 (used in podman 3.1.0) and v1.24.6 (used in the previous podman release 3.0.1) or open an issue on GitHub, yet, but maybe somebody else can have a look at it until I'm awake again and tell me I'm not hallucinating due to caffein deprivation or sth.

So long!

Offline

#14 2021-04-05 01:26:52

josephg
Member
From: Brazil
Registered: 2009-09-28
Posts: 106

Re: Rootless Podman won't work since version 3.1.0

I confirm that "rootless_storage_path" disabled or set without home directory path (not $HOME) work around the hanging issue.  I also confirm the "operation not permitted" with trying to pull archlinux image.  Weird enough, I'm able to pull and run alpine and fedora, but not archlinux and ubuntu (these are the images I tested, all from docker.io).

Just now I filed an issue in https://github.com/containers/podman/issues/9936.

Offline

#15 2021-04-05 02:37:11

calebstewart
Member
Registered: 2021-04-04
Posts: 5

Re: Rootless Podman won't work since version 3.1.0

Good catch! I see I can pull alpine as well! I threw my two cents into the issue. Hopefully someone there has an idea.

Offline

Board footer

Powered by FluxBB