Find original owning process of a Linux socket

Question

In Linux and other UNIX-like operating systems, it is possible for two (or more) processes to share an Internet socket. Assuming there is no parent-child relationship between the processes, is there any way to tell what process originally created a socket?

Clarification: I need to determine this from "outside" the processes using the /proc filesystem or similar. I can't modify the code of the processes. I can already tell what processes are sharing sockets by reading /proc/<pid>/fd, but that doesn't tell me what process originally created them.

Answer 1

You can use netstat for this. You should look in the columns 'Local Address' and 'PID/Program name'.

xxx@xxx:~$ netstat -tulpen
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:4005          0.0.0.0:*               LISTEN      1000       68449       7559/sbcl       
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      0          3938        -               
tcp6       0      0 :::6000                 :::*                    LISTEN      0          3937        -               
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          4528        -               

Answer 2

doesn't 'lsof -Ua' help?

Answer 3

You can likely find the shared sockets by parsing /proc/net/tcp (and similar "files" for other protocols). There's some docs on /proc/net/tcp here.

You would need to find the socket (perhaps by its IP addresses/port numbers ?) and parse out the inode number. Once you have the inode, you can search through all of /proc/*/fd/* , calling stat for every link and inspect the st_ino member of struct stat until you find a match.

The inode number should match between the 2 processes, so when you've gone through all /proc/*/fd/* you should have found them both.

If what you do know is the process id and socket fd of the first, you might not need to go through /proc/net/tcp, all you need to do is stat the /proc/<pid>/fd/<fd> and search the rest of /proc/*/fd/* for a matching inode. You'd need /proc/net/tcp if you want to fetch the ip addresses/port number though - which you can find if you know the inode number

Answer 4

For purposes creating a test case, consider a situation where multiple ssh-agent processes are running and have open sockets. I.e. A user runs ssh-agent multiple times and loses the socket/PID information given when the agent started:

$ find /tmp -path "*ssh*agent*" 2>/dev/null
/tmp/ssh-0XemJ4YlRtVI/agent.14405
/tmp/ssh-W1Tl4i8HiftZ/agent.21283
/tmp/ssh-w4fyViMab8wr/agent.10966

Later, the user wants to programmatically determine the PID owner of a particular ssh-agent socket (i.e. /tmp/ssh-W1Tl4i8HiftZ/agent.21283):

$ stat /tmp/ssh-W1Tl4i8HiftZ/agent.21283
  File: '/tmp/ssh-W1Tl4i8HiftZ/agent.21283'
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 805h/2053d      Inode: 113         Links: 1
Access: (0600/srw-------)  Uid: ( 4000/ myname)   Gid: ( 4500/   mygrp)
Access: 2018-03-07 21:23:08.373138728 -0600
Modify: 2018-03-07 20:49:43.638291884 -0600
Change: 2018-03-07 20:49:43.638291884 -0600
Birth: -

In this case, because ssh-agent named its socket nicely as a human onlooker can guess that the socket belongs to PID 21284, because the socket name contains a numeric component that is one-off from a PID identified with ps:

$ ps -ef |  grep ssh-agent
myname   10967     1  0 16:54 ?        00:00:00 ssh-agent
myname   14406     1  0 20:35 ?        00:00:00 ssh-agent
myname   21284     1  0 20:49 ?        00:00:00 ssh-agent

It seems highly unwise to make any assumption that the PIDs will be so reliable as to always only be off by one, but also, one might suppose that not all socket creators will name the sockets so nicely.

@Cypher's answer points to a straightforward solution to the problem of identifying the PID of the socket owner, but is incomplete as lsof actually can only identify this PID with elevated permissions. Without elevated permissions, no results are forthcoming:

$ lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
$

With elevated permissions, however, the PID is identified:

$ sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283
COMMAND     PID    USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
ssh-agent 21284 myname     3u  unix 0xffff971aba04cc00      0t0 1785049 /tmp/ssh-W1Tl4i8HiftZ/agent.21283 type=STREAM

In this case, the owner of the PID (myname) and socket was the one doing the query, so it seemed elevated permissions should not be needed. Furthermore, the task performing the query was not supposed to be able to elevate permissions, so I looked for another answer.

This led me to @whoplisp's answer proposing netstat -tulpen as a solution to the OP's problem. While it may have been effective for the OP, the command line is too restrictive to serve as a general purpose command and was completely ineffective in this case (even with elevated permissions).

$ sudo netstat -tulpen | grep -E -- '(agent.21283|ssh-agent)'
$

netstat, however, can come close if a different command-line is used:

$ netstat -ap | grep -E -- '(agent.21283)'
(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
unix  2      [ ACC ]     STREAM     LISTENING     1785049  -                    /tmp/ssh-W1Tl4i8HiftZ/agent.21283

Sadly, here too, the PID is elusive without elevated permissions:

$ sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)'
unix  2      [ ACC ]     STREAM     LISTENING     1765316  10967/ssh-agent      /tmp/ssh-w4fyViMab8wr/agent.10966
unix  2      [ ACC ]     STREAM     LISTENING     1777450  14406/ssh-agent      /tmp/ssh-0XemJ4YlRtVI/agent.14405
unix  2      [ ACC ]     STREAM     LISTENING     1785049  21284/ssh-agent      /tmp/ssh-W1Tl4i8HiftZ/agent.21283

Of the two solutions, however, lsof clearly wins at the races:

$ time sudo netstat -ap | grep -E -- '(agent.21283|ssh-agent)' >/dev/null

real    0m5.159s
user    0m0.010s
sys     0m0.019s
$ time sudo lsof /tmp/ssh-W1Tl4i8HiftZ/agent.21283 >/dev/null

real    0m0.120s
user    0m0.038s
sys     0m0.066s

Yet another tool exists according to the netstat man page:

$ man netstat | grep -iC1 replace
NOTES
       This program is mostly obsolete.  Replacement for netstat is ss.  Replacement for netstat -r is ip route.  Replacement for netstat -i
       is ip -s link.  Replacement for netstat -g is ip maddr.

Sadly, ss also requires elevated permissions to identify the PID, but, it beats both netstat and lsof execution times:

$ time sudo ss -ap | grep -E "(agent.21283|ssh-agent)"
u_str  LISTEN     0      128    /tmp/ssh-w4fyViMab8wr/agent.10966 1765316               * 0                     users:(("ssh-agent",pid=10967,fd=3))
u_str  LISTEN     0      128    /tmp/ssh-0XemJ4YlRtVI/agent.14405 1777450               * 0                     users:(("ssh-agent",pid=14406,fd=3))
u_str  LISTEN     0      128    /tmp/ssh-W1Tl4i8HiftZ/agent.21283 1785049               * 0                     users:(("ssh-agent",pid=21284,fd=3))

real    0m0.043s
user    0m0.018s
sys     0m0.021s

In conclusion, it might seem that for some PID identification, it appears that elevated permissions are required.

Note: Not all operating systems require elevated permissions. For example, SCO Openserver 5.0.7's lsof seemed to work just fine without elevating permissions.

Caveat: This answer may fail with respect to the OP's qualification for finding "the original creator" of the socket. In the example used, no doubt PID 21283 was the originator of the socket's creation as this PID is identified in the socket name. Neither lsof nor netstat identified PID 21283 as the original creator, though clearly PID 21284 is the current maintainer.