Follow pids across machines (ssh)

Question

I am basically trying to write a pstree-like command except that it should follow processes across machines.

What I mean is that if I run this :

$ ssh $node sleep 1000

Then the command should display something like this :

ssh $node -- ($node) sleep 1000

And if I'm running :

$ ssh $node ssh $node sleep 1000

ssh $node---($node) ssh $node---($node) sleep 1000

And so on ...

My question is this : How can I map one ssh session on one machine to a spawned process on another machine ?

Local parent-child processes are not a problem, but how can I figure out which ssh command on one node that triggered another process on another node.

1. linux 2.6.18

2. only openSSH for "remote" stuff. Running OpenSSH_4.3p2 currently.

3. SSH access to all nodes of course (key based auth) so ps and netstat are available from all nodes.

4. Linux-only "hacks" are fine, does not need to be portable though that would be an added bonus of course.

5. The user will always be the same and my command/script is running as that user. That user is not root.

6. Does not have to be fast, only accurate.

The spontaneous solution would be to write a pstree clone, that triggers on the command string "ssh", figures out the source-port and then goes to the remote machine in question and figures out which one of sshd's children that was spawned by this particular command.

But maybe there's a more clever way ? :P

Answer 1

Actually, I think your spontaneous solution is the right way to do it: use netstat to get the source-port and look for it on the remote machine. You might have trouble using "netstat -p" without being root - I tried it on two machines, one which was happy to show me my own processes and one which wasn't.

As well as ssh clients, you might extend this to look for other clients that use ssh connections, like rsync or Mercurial. Just be careful not to trace your program's own connection recursively!

A quick experiment with netstat and pstree shows that the idea is sound:

me@mymachine:~$ netstat -p
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 mymachine.example:43681 remote.example.com:ssh ESTABLISHED 27044/ssh
tcp        0      0 mymachine.example:39228 remote.example.com:ssh ESTABLISHED 14499/ssh
tcp        0      0 mymachine.example:45814 remote.example.com:ssh ESTABLISHED 20899/ssh
 me@mymachine:~$ ssh remote netstat -p | grep mymachine.example:43681
tcp        0      0 remote.example.com:ssh mymachine.example:43681 ESTABLISHED 10361/1
me@mymachine:~$ ssh remote pstree -a 10361
sshd
  `-grep -n -e wotsit -i -R /local/home/me/somewhere /dev/null

I'd be interested to see the result, because it would be very useful to me!