IP_TRANSPARENT usage

Question

I am implementing a transparent TCP/UDP proxy for all ports (1-65535) on a Raspberry Pi on LAN. I am currently testing routing TCP packets with destination port 80 to the Raspberry Pi. The idea is that one interface (cf "proxy ip") captures incoming traffic and the other (cf "server ip") sends it to the internet and processes it before the original one sends the response to the client. The necessary routing on the router is done via

iptables -t mangle -A PREROUTING -p tcp -s SERVER_IP -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -s SOME_TEST_CLIENT_IP --dport 80 -j MARK --set-mark 3 
ip rule add fwmark 3 table 2 
ip route add default via PROXY_IP dev br0 table 2

inspired by this page. This architecture implies a one-to-one port mapping between external IP addresses and the Raspberry PI's proxy interface. The packets arrive with the correct port and destination on the Raspberry Pi (verified with tcpdump), however the proxy doesn't accept the connections: no SYN-ACK is sent for the incoming SYN's. The proxy listening sockets are mainly configured with

const char PROXY_IP_ADDR[] = "192.168.1....";
const char SERVER_IP_ADDR[] = "192.168.1....";
...
struct sockaddr_in saProxy = {0};
saProxy.sin_family = AF_INET;
saProxy.sin_port = htons(80);
inet_pton(AF_INET, PROXY_IP_ADDR, &(saProxy.sin_addr.s_addr));
int enable = 1;
int sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if(-1 == setsockopt(sockfd, SOL_IP, IP_TRANSPARENT, (const char*)&enable, sizeof(enable)) /*error processing*/;
if(-1 == bind(sockfd, (sockaddr*)&saProxy, sizeof(saProxy))) /* error processing*/;
if(-1 == listen(sockfd, 1)) /* error processing*/;

Followed by epoll_ctl() and epoll_wait(). The proxy has been tested with sending HTTP requests and NBNS traffic directly to PROXY_IP without the aforementioned routing in place and it is receiving and processing these connections properly.

Unfortunately, I have found very little documentation or examples related to IP_TRANSPARENT. My original Windows-related question before I could do any testing on Linux. Kernel version is 4.1.13-v7+. How can I achieve this type of proxying?

Edit: I believe I may be missing some routing settings on the Raspberry Pi such as perhaps described here, but I have very little experience with iptables so I don't quite understand the rules described there, although I have read that non-local traffic is rejected by the kernel unless some specific routing is set up since it doesn't know about sockets.

I have also tested binding directly to an external IP address and attempted to listen for packets with this destination address, but the symptoms remain unchanged.

Answer 1

The solution was pretty simple actually. In order to use IP_TRANSPARENT for this purpose, you need to have a single listening socket bound to some port X. Then you need to setup the following rules, assuming you want to redirect ALL traffic going through any (I believe) interface, excluding traffic generated for/by the proxy itself. Here the proxy's IP is 192.168.1.100 and we redirect TCP to port 82 and UDP to port 83.

iptables -t mangle -A PREROUTING ! -d 192.168.1.100 -p tcp -j TPROXY --on-port 82 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
iptables -t mangle -A PREROUTING ! -d 192.168.1.100 -p udp -j TPROXY --on-port 83 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Linux has a special mechanism called tproxy for this.

For TCP

From here on, the socket returned by accept is automatically bound to the original destination and connected to the source, so using it for transparent proxying requires no more work on this side of the proxy.

In order to get the original destination of the socket as a sockaddr_in structure, call getsockname() on the socket returned by accept() as usual.

For UDP

To be able to get the original destination, on the UDP socket, set this option before binding:

int enable = 1;
setsockopt(sockfd, SOL_IP, IP_RECVORIGDSTADDR, (const char*)&enable, sizeof(enable));

Then, to receive the data and get the original destination

char cmbuf[100];
unsigned char bytes[16*1024];
sockaddr_in srcIpAddr, dstIpAddr;
int dstPort;
iovec iov;
iov.iov_base = bytes;
iov.iov_len = sizeof(bytes)-1;
msghdr mh;
mh.msg_name = &srcIpAddr;
mh.msg_namelen = sizeof(sockaddr_in);
mh.msg_control = cmbuf;
mh.msg_controllen = 100;
mh.msg_iovlen = 1;
mh.msg_iov = &iov;
int res = recvmsg(sock, &mh, 0);
sem_post(&udpSem); //I use a semaphore to indicate when incoming data is read and socket is ready for new datagram to be processed

for(cmsghdr *cmsg = CMSG_FIRSTHDR(&mh); cmsg != NULL; cmsg = CMSG_NXTHDR(&mh, cmsg))
{
    if(cmsg->cmsg_level != SOL_IP || cmsg->cmsg_type != IP_ORIGDSTADDR) continue; //normally we use IP_PKTINFO if not using tproxy, but this would yield 192.168.1.100:83 in the example
    std::memcpy(&dstIpAddr, CMSG_DATA(cmsg), sizeof(sockaddr_in));
    dstPort = ntohs(dstIpAddr.sin_port);
}

Then, if we want to reply to the datagram, we need to make a new UDP socket (as UDP is connectionless) and bind it to the original destination of the datagram, stored in dstIpAddr. I had some trouble here as I first tried using IP_FREEBIND, but this option does not seem to work for sending data through UDP, I think it is only intended for TCP listening sockets, so we use IP_TRANSPARENT again before binding to be able to bind to a non-local address.