Avoid reusing of the same fd number in a multithread socket application

Question

I have an asynchronous application executing several threads doing operations over sockets where operations are scheduled and then executed asynchronously.

I'm trying to avoid a situation when once scheduled a read operation over a socket, the socket gets closed and reopened(by possibly another peer in another operation), before the first operation started execution, which will end up reading the proper file descriptor but the wrong peer.

The problem comes because (accept();close();accept()) returns the same fd in both accepts() which can lead to the above situation.

I can't see a way of avoiding it.

any hint?

Answer 1

Ok, found the answer.

The best way here is to call accept() and get the lowest fd available, duplicate it with a number known by you like dup2(6,1000) and close(6), you have now control of the fd range you use.

Next accept will come again with 6 or similar, and we'll dup2(6,999); and keep decreasing like that and reseting it if it gets too low.

Since the accepting is done always in the same thread and dup2 and close aren't expensive compared to accept which is always done there it's perfect for my needs.

Answer 2

How do you manage the sockets? It sounds like you have multiple threads any of which can:

1. accept an incoming connection
2. close an existing connection
3. make a new outgoing connection

It sounds like you need a way to mediate access to the various sockets floating around. Have you considered associating each socket with mutex which prevents closing the socket while it's still in use, or maybe putting each socket descriptor in a struct with an atomic reference count which will prevent other threads from closing it until all threads are done using it?

Answer 3

a socket is a 5-tuple {local-addr,local-port,remote-addr,remote-port,proto}, so if you are able to use these properties instead of fd for event/handler routing you can avoid the fd clash.

another option would be to serialize all close()/accept() operations (priorities ?) so that they cannot intermix

Answer 4

Great question! I didn't even realize that such a problem could occur.

The only answer that I can think of is that you musn't use close() to signal that a socket is terminated. One solution is to use shutdown() to terminate the connection. You could then close() the socket safely by employing reference counting.