Ansible won't let me connect through SSH

Question

I'm trying to connect from one server to another. In fact I'm trying to connect to my host OS (CoreOS) from within a docker container. I have set up a RSA key and it works like a charm when using standard command line to connect to the remote host. It works as expected.

When I'm trying to run ansible customercare -m ping --user=core --connection=ssh --private-key=/home/jenkins/.ssh/id_rsa I'm met with this error

10.45.1.107 | FAILED => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

And the verbose option look like this:

<10.45.1.107> ESTABLISH CONNECTION FOR USER: core
<10.45.1.107> REMOTE_MODULE ping
<10.45.1.107> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/root/.ansible/cp/    ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'IdentityFile=/home/jenkins/.ssh/id_rsa', '-o', 'KbdInteractiveAuthentication=no', '-o',     'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=core', '-o',     'ConnectTimeout=10', '10.45.1.107', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1437988628.37-213828375275223 && chmod a+rx $HOME/.    ansible/tmp/ansible-tmp-1437988628.37-213828375275223 && echo $HOME/.ansible/tmp/ansible-tmp-1437988628.37-213828375275223'"]
10.45.1.107 | FAILED => SSH encountered an unknown error. The output was:
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/root/.ansible/cp/ansible-ssh-10.45.1.107-22-core" does not exist
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.45.1.107 [10.45.1.107] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9985 ms remain after connect
debug1: permanently_set_uid: 0/0
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/jenkins/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/jenkins/.ssh/id_rsa type 1
debug1: identity file /home/jenkins/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7
debug1: match: OpenSSH_6.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "10.45.1.107" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type ED25519 in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],ssh-ed25519
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-    exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],ssh-ed25519,[email protected],ecdsa-sha2-nistp384-cert-    [email protected],[email protected],[email protected],[email protected],ssh-rsa-cert-v00@openssh.    com,[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],chacha20-    [email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],chacha20-    [email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256-    [email protected],[email protected],[email protected],[email protected],[email protected],hmac-    md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,    hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256-    [email protected],[email protected],[email protected],[email protected],[email protected],hmac-    md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,    hmac-md5-96
debug2: kex_parse_kexinit: [email protected],zlib,none
debug2: kex_parse_kexinit: [email protected],zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: [email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],hmac-    [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],hmac-    [email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup [email protected]
debug1: kex: server->client aes128-ctr [email protected] [email protected]
debug2: mac_setup: setup [email protected]
debug1: kex: client->server aes128-ctr [email protected] [email protected]
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ED25519 54:85:33:0a:6f:78:74:a7:13:7d:74:bd:03:f1:9c:ce
debug3: load_hostkeys: loading entries for host "10.45.1.107" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type ED25519 in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host '10.45.1.107' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: ssh_ed25519_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/jenkins/.ssh/id_rsa (0x7f2295d969e0), explicit
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/jenkins/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 53:f8:88:06:5b:c2:a3:0a:05:9f:2c:ed:3b:51:74:47
debug3: sign_and_send_pubkey: RSA 53:f8:88:06:5b:c2:a3:0a:05:9f:2c:ed:3b:51:74:47
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 10.45.1.107 ([10.45.1.107]:22).
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /root/.ansible/cp/ansible-ssh-10.45.1.107-22-core.xNa4LxZkP4s02v2j
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new [/root/.ansible/cp/ansible-ssh-10.45.1.107-22-core]
debug3: muxserver_listen: mux listener channel 0 fd 4
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x08
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 470
Control socket connect(/root/.ansible/cp/ansible-ssh-10.45.1.107-22-core): Connection refused
Failed to connect to new control master
debug1: forking to background
debug1: Entering interactive session.
debug2: set_control_persist_exit_time: schedule exit in 60 seconds

Any clue on what is going on?

[UPDATE] Here's the log from a successful SSH logon:

jenkins@9031c65c8952:~$ ssh [email protected] -vvvv
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 10.45.1.107 [10.45.1.107] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/jenkins/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/jenkins/.ssh/id_rsa type 1
debug1: identity file /home/jenkins/.ssh/id_rsa-cert type -1
debug1: identity file /home/jenkins/.ssh/id_dsa type -1
debug1: identity file /home/jenkins/.ssh/id_dsa-cert type -1
debug1: identity file /home/jenkins/.ssh/id_ecdsa type -1
debug1: identity file /home/jenkins/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/jenkins/.ssh/id_ed25519 type -1
debug1: identity file /home/jenkins/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7
debug1: match: OpenSSH_6.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "10.45.1.107" from file "/home/jenkins/.ssh/known_hosts"
debug3: load_hostkeys: found key type ED25519 in file /home/jenkins/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],ssh-ed25519
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],ssh-ed25519,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,[email protected],[email protected],[email protected],aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-md5,hmac-sha1,[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: [email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup [email protected]
debug1: kex: server->client aes128-ctr [email protected] none
debug2: mac_setup: setup [email protected]
debug1: kex: client->server aes128-ctr [email protected] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ED25519 54:85:33:0a:6f:78:74:a7:13:7d:74:bd:03:f1:9c:ce
debug3: load_hostkeys: loading entries for host "10.45.1.107" from file "/home/jenkins/.ssh/known_hosts"
debug3: load_hostkeys: found key type ED25519 in file /home/jenkins/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host '10.45.1.107' is known and matches the ED25519 host key.
debug1: Found key in /home/jenkins/.ssh/known_hosts:1
debug1: ssh_ed25519_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/jenkins/.ssh/id_rsa (0x7fab14d1cab0),
debug2: key: /home/jenkins/.ssh/id_dsa ((nil)),
debug2: key: /home/jenkins/.ssh/id_ecdsa ((nil)),
debug2: key: /home/jenkins/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/jenkins/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 53:f8:88:06:5b:c2:a3:0a:05:9f:2c:ed:3b:51:74:47
debug3: sign_and_send_pubkey: RSA 53:f8:88:06:5b:c2:a3:0a:05:9f:2c:ed:3b:51:74:47
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 10.45.1.107 ([10.45.1.107]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env TERM
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env LESSOPEN
debug3: Ignored env LESSCLOSE
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Jul 27 09:49:44 2015 from 172.17.0.37
CoreOS stable (717.3.0)
core@localhost ~ $

Answer 1

I had the same problem with ansible running in a docker container on a coreOS host. With the help of this thread, i could follow it up to the -O ControlPath option. Switching off acceleration didn't help for me though. Changing the ssh_args= line to empty string did the trick for me.

Answer 2

I found here that one of the reasons why ssh pipelining fails in docker is a bug when using overlay storage driver. Then a quick fix is to change directory of control path. In /etc/ansible/ansible.cfg add

[ssh_connection] control_path_dir=/dev/shm/ansible_control_path