Linux: proc/net/sockstat tcp mem more and more larger

Question

Now, our system find hang and tcp mem more and more larger through /proc/net/sockstat. when hang appear, will print :

"tcp:too many of orphaned sockets"

From sockstat, we know there are few socket, but consumes 1500 page mem, why ?

So I have 2 questions :

1. How to know Which process consumes tcp socket memory?
2. How to avoid "tcp:too many of orphaned sockets"?

(1)

~ # cat /proc/net/sockstat

sockets: used 56

TCP: inuse 6 orphan 0 tw 1 alloc 8 mem 1510

UDP: inuse 8 mem 6

UDPLITE: inuse 0

RAW: inuse 4

FRAG: inuse 0 memory 0

(2)

~ # cat /proc/sys/net/ipv4/tcp_mem

900     1200    1800

~ # cat /proc/sys/net/ipv4/tcp_rmem

4096    87380   87380

~ # cat /proc/sys/net/ipv4/tcp_wmem

4096    16384   65536

Answer 1

For #1, memory consumption for sockets is the sum of

• the socket descriptors
• in-kernel send queues (stuff waiting to be sent out by the NIC)
• in-kernel receive queues (stuff that's been received, but hasn't yet been read by user space yet).

(this post is relevant here)

For your example output from /proc/net/sockstat, the number of sockets is small, so check the size of their send/receive queues. You can do this using commands like netstat -tanp or ss -tp. Keep in mind that send and receive buffer sizes displayed with e.g. ss -m are maximum values (constrained with tcp_rmem and `tcp_wmem), not the currently allocated values.

For #2, this post explains that the "too many orphan socket" is caused by the number of orphans increasing past the value in /proc/sys/net/ipv4/tcp_max_orphans, though some kinds of "bad" sockets are penalized more than others, so you could hit the error even if you're 2x or 4x below the limit.