How to write a proxy in go (golang) using tcp connections

Question

I apologize before hand if some of these questions might be obvious for expert network programmers. I have researched and read about coding in networking and it is still not clear to me how to do this.

Assume that I want to write a tcp proxy (in go) with the connection between some TCP client and some TCP server. Something like this:

First assume that these connection are semi-permanent (will be closed after a long long while) and I need the data to arrive in order.

The idea that I want to implement is the following: whenever I get a request from the client, I want to forward that request to the backend server and wait (and do nothing) until the backend server responds to me (the proxy) and then forward that response to the client (assume that both TCP connection will be maintained in the common case).

There is one main problem that I am not sure how to solve. When I forward the request from the proxy to the server, and get the response, how do I know when the server has sent me all the information that I need if I do not know beforehand the format of the data being sent from the server to the proxy (i.e. I don't know if the response from the server is of the form of type-length-value scheme nor do I know if `\r\n\ indicates the end of the message form the server). I was told that I should assume that I get all the data from the server connection whenever my read size from the tcp connection is zero or smaller than the read size that I expected. However, this does not seem correct to me. The reason it might not be correct in general is the following:

Assume that the server for some reason is only writing to its socket one byte at a time but the total length of the response to the "real" client is much much much longer. Therefore, isn't it possible that when the proxy reads the tcp socket connected to the server, that the proxy only reads one byte and if it loops fast enough (to do a read before it receives more data), then read zero and incorrectly concludes that It got all the message that the client intended to receive?

One way to fix this might be to wait after each read from the socket, so that the proxy doesn't loop faster than it gets bytes. The reason that I am worried is, assume there is a network partition and i can't talk to the server anymore. However, it is not disconnected from me long enough to timeout the TCP connection. Thus, isn't it possible that I try to read from the tcp socket to the server again (faster than I get data) and read zero and incorrectly conclude that its all the data and then send it pack to the client? (remember, the promise I want to keep is that I only send whole messages to the client when i write to the client connection. Thus, its illegal to consider correct behaviour if the proxy goes, reads the connection again at a later time after it already wrote to the client, and sends the missing chunk at a later time, maybe during the response of a different request).

The code that I have written is in go-playground.

The analogy that I like to use to explain why I think this method doesn't work is the following:

Say we have a cup and the proxy is drinking half the cup every time it does a read from the server, but the server only puts 1 teaspoon at a time. Thus, if the proxy drinks faster than it gets teaspoons it might reach zero too soon and conclude that its socket is empty and that its ok to move on! Which is wrong if we want to guarantee we are sending full messages every time. Either, this analogy is wrong and some "magic" from TCP makes it work or the algorithm that assumes until the socket is empty is just plain wrong.

A question that deals with a similar problems here suggests to read until EOF. However, I am unsure why that would be correct. Does reading EOF mean that I got the indented message? Is an EOF sent each time someone writes a chunk of bytes to a tcp socket (i.e. I am worried that if the server writes one byte at a time, that it sends 1 EOF per bytes)? However, EOF might be some of the "magic" of how a TCP connection really works? Does sending EOF's close the connection? If it does its not a method that I want to use. Also, I have no control of what the server might be doing (i.e. I do not know how often it wants to write to the socket to send data to the proxy, however, its reasonable to assume it writes to the socket with some "standard/normal writing algorithm to sockets"). I am just not convinced that reading till EOF from the socket from server is correct. Why would it? When can I even read to EOF? Are EOFs part of the data or are they in the TCP header?

Also, the idea that I wrote about putting a wait just epsilon bellow the time-out, would that work in the worst-case or only on average? I was also thinking, I realized that if the Wait() call is longer than the time-out, then if you return to the tcp connection and it doesn't have anything, then its safe to move on. However, if it doesn't have anything and we don't know what happened to the server, then we would time out. So its safe to close the connection (because the timeout would have done that anyway). Thus, I think if the Wait call is at least as long as the timeout, this procedure does work! What do people think?

I am also interested in an answer that can justify maybe why this algorithm work on some cases. For example, I was thinking, even if the server only write a byte at a time, if the scenario of deployment is a tight data centre, then on average, because delays are really small and the wait call is almost certainly enough, then wouldn't this algorithm be fine?

Also, are there any risks of the code I wrote getting into a "deadlock"?

package main

import (
    "fmt"
    "net"
)

type Proxy struct {
    ServerConnection *net.TCPConn
    ClientConnection *net.TCPConn
}

func (p *Proxy) Proxy() {
    fmt.Println("Running proxy...")
    for {
        request := p.receiveRequestClient()
        p.sendClientRequestToServer(request)
        response := p.receiveResponseFromServer() //<--worried about this one.
        p.sendServerResponseToClient(response)
    }
}

func (p *Proxy) receiveRequestClient() (request []byte) {
    //assume this function is a black box and that it works.
    //maybe we know that the messages from the client always end in \r\n or they
    //they are length prefixed.
    return
}

func (p *Proxy) sendClientRequestToServer(request []byte) {
    //do
    bytesSent := 0
    bytesToSend := len(request)
    for bytesSent < bytesToSend {
        n, _ := p.ServerConnection.Write(request)
        bytesSent += n
    }
    return
}

// Intended behaviour: waits until ALL of the response from backend server is obtained.
// What it does though, assumes that if it reads zero, that the server has not yet
// written to the proxy and therefore waits. However, once the first byte has been read,
// keeps writting until it extracts all the data from the server and the socket is "empty".
// (Signaled by reading zero from the second loop)
func (p *Proxy) receiveResponseFromServer() (response []byte) {
    bytesRead, _ := p.ServerConnection.Read(response)
    for bytesRead == 0 {
        bytesRead, _ = p.ServerConnection.Read(response)
    }
    for bytesRead != 0 {
        n, _ := p.ServerConnection.Read(response)
        bytesRead += n
        //Wait(n) could solve it here?
    }
    return
}

func (p *Proxy) sendServerResponseToClient(response []byte) {
    bytesSent := 0
    bytesToSend := len(request)
    for bytesSent < bytesToSend {
        n, _ := p.ServerConnection.Write(request)
        bytesSent += n
    }
    return
}

func main() {
    proxy := &Proxy{}
    proxy.Proxy()
}

Answer 1

Unless you're working with a specific higher-level protocol, there is no "message" to read from the client to relay to the server. TCP is a stream protocol, and all you can do is shuttle bytes back and forth.

The good news is that this is amazingly easy in go, and the core part of this proxy will be:

go io.Copy(server, client)
io.Copy(client, server)

This is obviously missing error handling, and doesn't shut down cleanly, but clearly shows how the core data transfer is handled.