What is the cost of establishing connection using Unix Domain sockets versus TCP sockets?

Question

Oddly I didn't find this info by googling. What is the cost of establishing connection using Unix Domain sockets versus TCP sockets?

Right now I have to do connection pooling with TCP sockets because reconnecting is quite expensive. I wonder if I can simplify my client by simply switching to Unix Domain sockets and getting rid of connection pooling.

Answer 1

If you look into the code, you'll see that Unix Domain sockets execute far less code than TCP sockets.

Messages sent through TCP sockets have to go all the way through the networking stack to the loopback interface (which is a virtual network interface device typically called "lo" on Unix-style systems), and then back up to the receiving socket. The networking stack code tacks on TCP and IP headers, makes routing decisions, forwards a packet to itself through "lo", then does more routing and strips the headers back off. Furthermore, because TCP is a networking protocol, the connection establishment part of it has all kinds of added complexity to deal with dropped packets. Most significantly for you, TCP has to send three messages just to establish the connection (SYN, SYN-ACK, and ACK).

Unix Domain sockets simply look at the virtual file system (or the "abstract namespace") to find the destination socket object (in RAM) and queue the message directly. Furthermore, even if you are using the file system to name your destination socket, if that socket has been accessed recently, its file system structures will be cached in RAM, so you won't have to go to to disk. Establishing a connection, for a Unix Domain socket involves creating a new socket object instance in RAM (i.e., the socket that gets returned by accept(), which is something that has to be done for TCP too) and storing a pointer in each of the two connected socket objects (so they each have a pointer to the other socket later when they need to send). That's pretty much it. No extra packets are needed.

By the way, this paper suggests that Unix Domain sockets are actually faster than even Pipes for data transfers:

http://osnet.cs.binghamton.edu/publications/TR-20070820.pdf

Unfortunately, they didn't do specific measurements of connection establishment costs, but as I have said, I've looked at the Linux source code and it's really quite a lot simpler than the TCP connection establishment code.

Answer 2

Connecting to a server using TCP sockets may involve network traffic, as well as the TCP three-way handshake.

Local sockets (formerly known as Unix domain sockets) are all local, but need to access a physical file on disk.

If you only do local communication then local sockets might be faster as there is less overhead from the protocol. If your application needs to connect remotely then you can't use local sockets.

---

By the way, if you're only communicating locally, and not over a network, a pair named pipes (or anonymous if you're forking) might be even better.