Why does Fabric display the disconnect from server message for almost 2 minutes?

Question

Fabric displays Disconnecting from username@server... done. for almost 2 minutes prior to showing a new command prompt whenever I issue a fab command.

This problem exists when using Fabric commands issued to both an internal server and a Rackspace cloud server. Below I've included the auth.log from the server, and I didn't see anything in the logs on my MacBook.

Any thoughts as to what the problem is?

Server's SSH auth.log with LogLevel VERBOSE

Apr 21 13:30:52 qsandbox01 sshd[19503]: Accepted password for mrankin from 10.10.100.106 port 52854 ssh2
Apr 21 13:30:52 qsandbox01 sshd[19503]: pam_unix(sshd:session): session opened for user mrankin by (uid=0)
Apr 21 13:30:52 qsandbox01 sudo:  mrankin : TTY=unknown ; PWD=/home/mrankin ; USER=root ; COMMAND=/bin/bash -l -c apache2ctl graceful
Apr 21 13:30:53 qsandbox01 sshd[19503]: pam_unix(sshd:session): session closed for user mrankin

Server Configuration

• OS: Ubuntu 9.10 and Ubuntu 6.10 (tested 4 servers with those OSes)
• OpenSSH: Ubuntu package version 1.5.1p1-6ubuntu2

Client Configuration

• OS: Mac OS X 10.6.3
• Fabric ver 0.9
• Vritualenv ver 1.4.7
• pip ver 0.7

Simple fabfile.py Used for Testing

The problem persists even when I just run fab -H server_ip host_type with the following fabfile.

from fabric.api import run

def host_type():
    run('uname -s')

Thoughts on Cause of the Issue

I'm not certain how long this problem has persisted, but below are some things that have changed since I started to notice the slow server disconnect using Fabric.

1. I recreated my virtualenv's using virtualenv 1.4.7, virtualenvwrapper 2.1, and pip 0.7. Not sure if this is related, but it is a thought since I run my fabfiles from within a virtualenv.
2. I enabled OS X's firewall. I disabled OS X's firewall and the problem persisted, so this is not the issue.

Answer 1

Solution

The problem no longer persists after I issued the following command in my virtualenv:

pip install -U paramiko

This installed paramiko-1.7.6 and pycrypto-2.0.1. Previously, I had paramiko-1.7.4 and pycrypto-2.0.1.

Appears that paramiko was the culprit given that the pycrypto version didn't change. At a minimum there appears to be an interaction between paramiko 1.7.4 and Fabric 0.9 that is fixed by upgrading paramiko to 1.7.6.

Note: I upgraded to paramiko-1.7.6 in one virtualenv and confirmed that the problem went away. I then activated another virtualenv that still had paramiko-1.7.4 and confirmed that the problem still persisted, which it did. Then I upgraded paramiko from 1.7.4 to 1.7.6 and confirmed that the problem went away in that virtualenv as well.

Answer 2

Thanks for keeping track of this here. I just want to note for any readers that Paramiko 1.7.4 has been previously known to be stable with Fabric 0.9, but in the last week or two several users have started exhibiting this or similar problems (disconnect timeouts) so I'm guessing some other component (Python upgrade, or remote server package upgrade, or something) is coming into play that is tipping off a bug in 1.7.4.

I will be checking out the changelogs for Paramiko 1.7.5/1.7.6 and gathering more info about peoples' platforms/Python versions/etc, to try and see if a pattern emerges.

EDIT: Newly created Redmine ticket for this issue is here: http://code.fabfile.org/issues/show/158