domain socket "sendto" encounter "errno 111, connection refused"

Question

I am using domain socket to get values from another process, like A to get a value from B, It works well for months. But recently, A is failed during "sendto" message to B with "errno 111, connection refused" occasionally.

I checked the B domain socket bind file, it is exists. I also do some tests in another machine, also works well. So, does anyone encounter this problem before? Can anyone have some clues what might be probably wrong in this scenario? Thanks very much.

Answer 1

When I've seen this error with unix domain sockets, it's usually because the process B isn't running, or there is a mismatch in the connection paths. (If B dies, does it automatically restart? Is it possible that the failures are happening while B has died but not yet restarted?). Another possibility: is it possible that multiple copies of A are running at the same time? You may get the ECONNREFUSED error if B's queue of not-yet-accepted connections is full.

I would suggest running both processes A and B under strace, either:

strace -o A.log A

or, if the process is already running,

strace -o B.log -p <process-id-of-B>

Also,

netstat -na

will give you the status of all unix domain sockets present in the system.

Answer 2

Consider looking under /proc/<pid-B>/fd and see whether B is running out of file descriptors. If so, you have a resource leak and need to clean up. It shouldn't be a problem with UDP programs, but funnier things have been known. lsof might be another tool to use.

Otherwise, you have reasonable suggestions from other people - netstat in particular should help.