How to know if sendto() with TCP Fast Open actually used Fast Open?

Question

I write a TCP client on a Linux 3.15 machine, which is able to use TCP Fast Open:

        status = sendto(sd, (const void *) data, data_len, MSG_FASTOPEN,
                        (const struct sockaddr *) hostref->ai_addr,
                        sizeof(struct sockaddr_in));
        if (status < 0) {
            fprintf(stderr, "sendto: %s\n", strerror(errno));
            exit(EXIT_FAILURE);
        }
        fprintf(stdout, "TFO connection successful to %s\n",
                    text_of(hostref->ai_addr));

Using tcpdump, I can check the sending of the TCP Fast Open option and that it does bypass the 3-way handshake (tested with Google's servers).

However, with servers which does not accept TCP fast open, sendto still succeeds, and the message "TFO connection successful" is displayed. Apparently, the Linux kernel code falls back to regular TCP if the server does not support TCP Fast Open (again, checked with tcpdump).

How to find out if my connection used TCP Fast Open or not?

Answer 1

By looking at the patch set that added TCP fast open in the linux kernel, you notice that it wasn't added any external indication that fast open was used.

You can indirectly notice certain cases where fast open was not used and certain cases where fast open was definitely used.

The case that you are sure fast open was not used is when the value of the TCPFastOpenActive counter was not incremented in /proc/net/netstat after a successful sendto() connection:

+   if (tcp_transmit_skb(sk, syn_data, 0, sk->sk_allocation) == 0) {
+       NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPFASTOPENACTIVE);
+       goto done;
+   }

The case that you are sure fast open was used is when you have a non-blocking socket, you already have a fast open cookie and sendto() does not return EINPROGRESS:

For non-blocking socket, it returns the number of bytes queued (and transmitted in the SYN-data packet) if cookie is available. If cookie is not available, it transmits a data-less SYN packet with Fast Open cookie request option and returns -EINPROGRESS like connect().

For the remaining case, that is, you don't have a cookie, but you were able to connect and TCPFastOpenActive was incremented, you can't say if fast open was used (TCPFastOpenActive increment was caused by your fast open) or if fast open was not used (TCPFastOpenActive increment was not caused by your fast open).

http://kernelnewbies.org/Linux_3.6#head-ac78950a7b57d92d5835642926f0e147c680b99c

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=cf60af03ca4e71134206809ea892e49b92a88896

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/diff/net/ipv4/tcp_output.c?id=783237e8daf13481ee234997cbbbb823872ac388