Podman in Podman, similar to Docker in Docker?

Question

Is there a way to run Podman inside Podman, similar to the way you can run Docker inside Docker?

Here is a snippet of my Dockerfile which is strongly based on another question:

FROM debian:10.6

RUN apt update && apt upgrade -qqy && \
    apt install -qqy iptables bridge-utils \
                     qemu-kvm libvirt-daemon libvirt-clients virtinst libvirt-daemon-system \
                     cpu-checker kmod && \
    apt -qqy install curl sudo gnupg2 && \
    echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list && \
    curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/Release.key | sudo apt-key add - && \
    apt update && \
    apt -qqy install podman

Now trying some tests:

$ podman run -it my/test bash -c "podman --storage-driver=vfs info"
... (long output; this works fine)

$ podman run -it my/test bash -c "podman --storage-driver=vfs images"
ERRO[0000] unable to write system event: "write unixgram @000ec->/run/systemd/journal/socket: sendmsg: no such file or directory" 
REPOSITORY  TAG     IMAGE ID  CREATED  SIZE

$ podman run -it my/test bash -c "podman --storage-driver=vfs run docker.io/library/hello-world"
ERRO[0000] unable to write system event: "write unixgram @000ef->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done  
Copying config bf756fb1ae done  
Writing manifest to image destination
Storing signatures
ERRO[0003] unable to write pod event: "write unixgram @000ef->/run/systemd/journal/socket: sendmsg: no such file or directory" 
ERRO[0003] Error preparing container 66692b7ff496775499d405d538769a078f2794549955cf2409fcbcbf87f42e94: error creating network namespace for container 66692b7ff496775499d405d538769a078f2794549955cf2409fcbcbf87f42e94: mount --make-rshared /var/run/netns failed: "operation not permitted" 
Error: failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/66692b7ff496775499d405d538769a078f2794549955cf2409fcbcbf87f42e94/userdata/shm": operation not permitted

I've also tried a suggestion from the other question, passing --cgroup-manager=cgroupfs, but without success:

$ podman run -it my/test bash -c "podman --storage-driver=vfs --cgroup-manager=cgroupfs run docker.io/library/hello-world"
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done  
Copying config bf756fb1ae done  
Writing manifest to image destination
Storing signatures
ERRO[0003] unable to write pod event: "write unixgram @000f3->/run/systemd/journal/socket: sendmsg: no such file or directory" 
ERRO[0003] Error preparing container c3fff4d8161903aaebd6f89f3b3c06b55038e11e07b6b561dc6576ca675747a3: error creating network namespace for container c3fff4d8161903aaebd6f89f3b3c06b55038e11e07b6b561dc6576ca675747a3: mount --make-rshared /var/run/netns failed: "operation not permitted" 
Error: failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/c3fff4d8161903aaebd6f89f3b3c06b55038e11e07b6b561dc6576ca675747a3/userdata/shm": operation not permitted

Seems like some network configuration is needed. I found the project below which suggests that some tweaking on network configurations might be necessary, but I don't know what would be the context of that and whether it would apply here or not. https://github.com/joshkunz/qemu-docker

EDIT: I've just discovered /var/run/podman.sock, but also without success:

$ sudo podman run -it -v /run/podman/podman.sock:/run/podman/podman.sock my/test bash -c "podman --storage-driver=vfs --cgroup-manager=cgroupfs run docker.io/library/hello-world"
Trying to pull my/test...
  denied: requested access to the resource is denied
Trying to pull my:test...
  unauthorized: access to the requested resource is not authorized
Error: unable to pull my/text: 2 errors occurred:
        * Error initializing source docker://my/test: Error reading manifest latest in docker.io/my/test: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

        * Error initializing source docker://quay.io/my/test:latest: Error reading manifest latest in quay.io/my/test: unauthorized: access to the requested resource is not authorized

Seems like root cannot see the images I've created under my user.

Any ideas? Thanks.

Answer 1

Assume we would like to run ls / in a docker.io/library/alpine container.

Standard Podman

podman run --rm docker.io/library/alpine ls /

Podman in Podman

Let's run ls / in a docker.io/library/alpine container, but this time we run podman in a quay.io/podman/stable container.

Update June 2021

A GitHub issue comment shows an example of how to run Podman in Podman as a non-root user both on the host and in the outer container. Slightly modified it would look like this:

podman \
  run \
    --rm \
    --security-opt label=disable \
    --user podman \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          docker.io/library/alpine \
            ls / 

Here is a full example:

$ podman --version
podman version 3.2.1
$ cat /etc/fedora-release 
Fedora release 34 (Thirty Four)
$ uname -r
5.12.11-300.fc34.x86_64
$ podman \
  run \
    --rm \
    --security-opt label=disable \
    --user podman \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          docker.io/library/alpine \
            ls / 
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b
Copying config sha256:d4ff818577bc193b309b355b02ebc9220427090057b54a59e73b79bdfe139b83
Writing manifest to image destination
Storing signatures
bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
$ 

To avoid repeatedly downloading the inner container image, create a volume

podman volume create mystorage

and add the command-line option -v mystorage:/home/podman/.local/share/containers:rw to the outer Podman command. In other words

podman \
  run \
    -v mystorage:/home/podman/.local/share/containers:rw \
    --rm \
    --security-opt label=disable \
    --user podman \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          docker.io/library/alpine \
            ls / 

Podman in Podman (outdated answer)

(The old outdated answer from Dec 2020. I'll probably remove this when it's clear that the method described here is outdated)

Let's run ls / in a docker.io/library/alpine container, but this time we run podman in a quay.io/podman/stable container.

The command will look like this:

podman \
  run \
    --privileged \
    --rm \
    --ulimit host \
    -v /dev/fuse:/dev/fuse:rw \
    -v ./mycontainers:/var/lib/containers:rw \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          --user 0 \
          docker.io/library/alpine ls 

(The directory ./mycontainers is here used for container storage)

Here is a full example

$ podman --version
podman version 2.1.1
$ mkdir mycontainers
$ podman run --privileged --rm --ulimit host -v /dev/fuse:/dev/fuse:rw -v ./mycontainers:/var/lib/containers:rw   quay.io/podman/stable podman run --rm --user 0 docker.io/library/alpine ls | head -5
Trying to pull docker.io/library/alpine...
Getting image source signatures
Copying blob sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964
Copying config sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
Writing manifest to image destination
Storing signatures
bin
dev
etc
home
lib
$ podman run --privileged --rm --ulimit host -v /dev/fuse:/dev/fuse:rw -v ./mycontainers:/var/lib/containers:rw  quay.io/podman/stable podman images
REPOSITORY                TAG     IMAGE ID      CREATED     SIZE
docker.io/library/alpine  latest  d6e46aa2470d  4 days ago  5.85 MB

If you would leave out -v ./mycontainers:/var/lib/containers:rw you might see the slightly confusing error message

Error: executable file `ls` not found in $PATH: No such file or directory: OCI runtime command not found error

References:

• How to use Podman inside of a container Red Hat blog post from July 2021.

• discussion.fedoraproject.org (discussion about not found in $PATH)

• github comment (that gives advice about the correct way to run Podman in Podman)