Forwarding ssh agent to container with Podman in Fedora

Question

I am creating a podman image and I'm trying to understand how to allow the container use my host (Fedora 37) ssh-agent so as the container can clone git repositories. I would prefer this than using PAT or other solutions that I can think of.

I have set-up a very basic example:

The application foo:

#!/bin/bash -eux
git clone "$REPO"

Dockerfile:

FROM docker.io/fedora:39@sha256:531209ee9007c2bf909c6548b0456524bde9da0a1ac95ecc67bf3b98e9046969 
ARG HOST
RUN dnf install -yq git
WORKDIR /app
COPY foo foo
ENV PATH="${PATH}:/app"
RUN useradd -m user && mkdir -p /home/user/.ssh && chown -R user:user /home/user/.ssh
WORKDIR /home/user
RUN printf "Host $HOST\n\tStrictHostKeyChecking no\n" >> .ssh/config
USER user
ENTRYPOINT ["foo"]

Then I build the image:

podman build -t foo --build-arg HOST=$HOST .

And try to run it with:

$ podman run -it -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -eREPO=$REPO foo

 + git clone git@*********.git

...
*****: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

If I run the container with --entrypoint /bin/bash and execute ssh-add -L I get:

[user@42e1646bd254 ~]$ ssh-add -L
Could not open a connection to your authentication agent.

I noticed that the "$SSH_AUTH_SOCK" dir in the container is owned by root:

[user@246199390bfe ~]$ stat $SSH_AUTH_SOCK
  File: /run/user/1000/keyring/ssh
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 0,68    Inode: 81          Links: 1
Access: (0755/srwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-19 07:28:45.436018916 +0000
Modify: 2023-03-18 06:20:31.860105581 +0000
Change: 2023-03-19 10:17:03.933082078 +0000
 Birth: 2023-03-18 06:20:31.860105581 +0000

And reading documentation I understood that podman runs as root within the container but gets mapped to my user outside the container (or something like that). So I tried to change my Dockerfile to use root instead of user:

FROM docker.io/fedora:39@sha256:531209ee9007c2bf909c6548b0456524bde9da0a1ac95ecc67bf3b98e9046969 
ARG HOST
RUN dnf install -yq git
WORKDIR /app
COPY foo foo
ENV PATH="${PATH}:/app"
RUN mkdir -p /root/.ssh
WORKDIR /root
RUN printf "Host $HOST\n\tStrictHostKeyChecking no\n" >> .ssh/config
ENTRYPOINT ["foo"]

Re-built and then tried something like:

podman run -it -v $SSH_AUTH_SOCK:/root/keyring:Z -e SSH_AUTH_SOCK=/root/keyring -eREPO=$REPO foo

But I still get a Permission denied (publickey).

I've checked that stat $SSH_AUTH_SOCK in the host contains Context: system_u:object_r:container_file_t:s0:c235,c239 (so I believe the :Z option is working as intended.

Can anyone help me understand what I'm doing wrong?

Using Podman 4.4.2 on Fedora.

UPDATE

I uninstalled podman and installed docker-ce and it worked on first attempt (tried the modified Dockerfile).

I would prefer to stay on podman though.

Answer 1

I struggled with this problem too but I found a solution.

The key is adding --privileged in your run (let's leave security concerns for now)

Here is my Dockerfile:

FROM fedora:37

RUN dnf install -yq git && dnf clean all
RUN mkdir -p .ssh
RUN printf "Host *\n\tStrictHostKeyChecking no\n" >> .ssh/config
ENTRYPOINT ["/bin/bash"]

Build with the following:

podman build -t foo:temp

And ran with the following run:

$ podman run -it -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK --privileged foo:temp -c "ssh -T [email protected]"
The authenticity of host 'github.com (140.82.113.4)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Hi op-jenkins! You've successfully authenticated, but GitHub does not provide shell access.

Idk why StrictHostKeyChecking no does not work, but once you just say yes, it can do ssh transaction.

I also tried with --cap-add ALL but that somehow failed:

$ podman run -it -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK --cap-add ALL foo:temp -c "ssh -T [email protected]"
The authenticity of host 'github.com (140.82.112.4)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
[email protected]: Permission denied (publickey).

When I turned -v option in ssh -vT [email protected], along the debug messages, I did find this error:

debug1: pubkey_prepare: ssh_get_authentication_socket: Permission denied

So adding all linux privileges via cap-add is not enough somehow to get permissions.

I feel that there is some security lift up when doing --privileged but I am no expert. And in my host machine, we have SELinux enabled... so that might also play a factor...? idk.

But hopefully this helped.

Answer 2

Disclaimer: Mileage may vary based on operating system, SELinux setup and whether podman is running using usernamespaces.

When trying to map the ssh agent socket into the container there are essentially three things blocking access:

1. Unix permissions
2. SELinux file labels
3. SELinux socket labels

By default the SSH socket is labeled something like unconfined_u:object_r:user_tmp_t:s0 (Stock Almalinux 9.4)

So predictably if we just mount in the socket, we will get a permission denied:

podman run --rm -it -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK 7b6e95f32d31 ssh-add -L

-> Error connecting to agent: Permission denied

Using audit2allow -w -a we can verify that SELinux has indeed blocked our access attempt to the socket

type=AVC msg=audit(<timestamp>): avc:  denied  { write } for  pid=1234 comm="ssh-add" name="agent.1234"

As you observed, adding :Z will correctly relabel the file with something like system_u:object_r:container_file_t:s0, but the error remains the same.

Checking again with SELinux reveals that the error has changed, instead of being denied a write, now it's connectto which is forbidden.

type=AVC msg=audit(<timestamp>): avc:  denied  { connectto } for  pid=1234 comm="ssh-add" path="/tmp/ssh-XXXX/agent.1234"

So SELinux has polices from what context processes are allowed to connect to sockets not just the files, see e.g https://wiki.gentoo.org/wiki/SELinux/Networking#TCP_and_UDP_socket_support for more information.

Now depending on your system you can either implement a new SELinux policy, or then simply disable labeling the container process, in which case the container runs as an unconfined process and will not be stopped by SELinux.

podman  run --security-opt=label=disable --rm -it -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z 7b6e95f32d31 ssh-add -L

ssh-ed25519 ....

This is what I did as no external access is allowed in the container, but if you plan on running this as a service, it might be prudent to implement new SELinux policies.

Now if the UID of the container process is 0, that's all you need to do. However, if the container you are using is running as another user (e.g the USER directive has been used while building), there is an additional step requires with podman. The following only applies when using usernamespaces, running as root has UID mappings work in a different but more straightforward way.

podman run  --security-opt=label=disable --rm -it -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z bbbfecaf8d3f ssh-add -L
Error connecting to agent: Permission denied

podman run  --security-opt=label=disable --rm -it -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:Z bbbfecaf8d3f id
uid=1000(newuser) gid=1000(newuser) groups=1000(newuser)   

Now based on audit2allow we can see that it's not SELinux blocking us, just standard unix permissions.

We don't have permission to access the socket as the OS checks that the UID we have inside and outside the container don't match (in my case 1002 outside and 1000 inside). Note that here I don't mean actual matches as even if the container was built with the exact ID used outside you still get a permission denied. This is due to the fact that there is no mapping between the internal and external UID. This can be fixed by adding --uidmap=1000:0:1 --uidmap=0:1:1 to the podman run command (assuming that the UID inside the container is 1000)

The format of the argument is --uidmap=[flags]container_uid:intermediate_uid[:amount] The full explanation can be found at https://docs.podman.io/en/latest/markdown/podman-run.1.html#uidmap-flags-container-uid-from-uid-amount, I'm by no means an expert so the following might be slightly incorrect.

In short: Our external UID is already mapped to UID 0 but we also need to tell podman to map the intermediate UID 0 to our final UID 1000 inside the container, the second mapping is a hard requirement from podmans side that UID 0 in the container has to be mapped to some intermediate UID (so here we just map it to 1). If podman is run by a privileged user, no intermediate UID exists and the mapping can be done directly e.g --uidmap=1000:1002:1, mapping UID 0 is also required. I could not directly find in the documentation the requirement for always mapping UID 0 and the error is not very helpful (Error: OCI runtime error: crun: write to /proc/sys/net/ipv4/ping_group_range: Invalid argument), https://github.com/containers/podman/issues/11922#issuecomment-940868998 tells that it is required