Putting command in the background with Fabric does not work on some hosts

Question

For testing purposes, I am running the following command, with plain ssh command line tool:

ssh user@host "nohup sleep 100 >> /tmp/xxx 2>&1 < /dev/null &"

This is working as expected, in all my hosts: a sleep process is created in the background, and the ssh finishes immediately.

I am trying to implement this functionality in python using Fabric. I end up doing a run call. This is what the Fabric logging is reporting:

[user@host] run: nohup sleep 100 >> /tmp/xxx 2>&1 < /dev/null &

Which is exactly what I expect. But if I check the processes which are running in my host, sleep 100 is not one of them. Worse yet: the problem happens only on some of my hosts.

I also added some more info to show what process has been created, by appending a "\necho $!" to the command to be run by Fabric. This is what was reported:

[user@host] run: nohup sleep 100 >> /tmp/xxx 2>&1 < /dev/null &
echo $!
[user@host] out: 30935

I am running out of ideas on how to debug this, since Fabric is reporting that the process has been created, but I see no process running in the other end. The syslog reports that an ssh session is being opened and closed:

Dec  6 09:12:09 host sshd[2835]: Accepted publickey for user from 67.133.172.14 port 37732 ssh2
Dec  6 09:12:09 host sshd[2838]: pam_unix(sshd:session): session opened for user user by (uid=0)
Dec  6 09:12:10 host sshd[2838]: pam_unix(sshd:session): session closed for user user

Could I somehow increase the amount of logging that the ssh daemon is producing, so that I can see at least what command is being requested via ssh?

I know that Fabric has some issues with running commands in the background, but that does not seem to be my problem. Could there be other issues with Fabric / ssh / background processes?

EDIT

I have installed dtach on all my systems. The version packaged in Ubuntu 8.04 is too old, and does not allow calling dtach -n over ssh (problems with terminal), so I had to download and compile the dtach sources. After doing that, I was able to run my command like this, with Fabric:

[user@host] run: dtach -n /tmp/Y sleep 100 >> /tmp/xxx 2>&1

This is working fine in all hosts. But this does not fit my scenario, because:

• dtach creates two processes: one for dtach itself, another for the process being run.
• I can not get the pid of the process being started

Answer 1

You are probably bumping into the infamous Fabric issue #395. The easiet workaround for these problems is to run your task with pty=False.