What may cause a TCP/IP reset (RST) flag to NOT be sent?

Question

I've two linux servers (let's name them A and B), connected to same (unmanaged) switch. I've disabled firewall on both servers (no rules in all tables, and all default policies set to ACCEPT). So, nothing should prevent one server to send any TCP/IP packets and another server to receive them as is.

Now, on A we run TCP server application, which listen/accept incoming connections, and then send a lot of data in a loop to connected client(s). It doesn't try to read from client, and expect to get EPIPE error while doing write() to socket if/when client disconnects.

Next, on B I run nc (netcat) as client application, connects to server application on A, begin receiving data, and few seconds later I press Ctrl-C to interrupt this connection.

What I see, is server application on A just hangs in write(), it doesn't got EPIPE or any other error.

I've traced TCP/IP packets using tcpdump, and here is what I see:

• after interrupting netcat on B, B send FIN to A, which correctly reply with ACK to that FIN - so, now we've fair half-open TCP connection, which is ok
• next, A tries to send next data to client with usual ACK and PSH,ACK packets, which is also expected and correct
• BUT, B doesn't reply in any way to these packets (while I expect it to reply with RST packet because it receiving packets to already-closed/non-existing TCP connection)
• A doesn't got ACK, so it stop sending new data and start resending old packets (and at this point next call to write() hangs)

I've also tried to run netcat on A (so both client and server applications runs on same physical server), and this way everything works as expected - server application got EPIPE immediately after I interrupt netcat with Ctrl-C. And tcpdump show there is RST packet sent as expected.

So, what may cause to not sending RST in this case?

I'm using Hardened Gentoo Linux, up-to-date, kernel 2.6.39-hardened-r8, without any specific sysctl network-related configuration.

It may or may not be important to note there is significant network activity on these servers, about 5000 tcp connections listed by netstat -alnp at any moment, and I think about 1000 connections opens and closes every second in average. It's usual to see in kernel log something like this (but port number is different from used by server application discussed above):

    TCP: Possible SYN flooding on port XXXXX. Sending cookies.
    net_ratelimit: 19 callbacks suppressed

Here is how TCP session usually looks like: http://i54.tinypic.com/1zz10mx.jpg

Answer 1

This behavior is result of enabled feature in my hardened kernel:

Security options  --->
    Grsecurity  --->
        Network Protections  --->
        [*] TCP/UDP blackhole and LAST_ACK DoS prevention

CONFIG_GRKERNSEC_BLACKHOLE:

If you say Y here, neither TCP resets nor ICMP destination-unreachable packets will be sent in response to packets sent to ports for which no associated listening process exists. This feature supports both IPV4 and IPV6 and exempts the loopback interface from blackholing. Enabling this feature makes a host more resilient to DoS attacks and reduces network visibility against scanners. The blackhole feature as-implemented is equivalent to the FreeBSD blackhole feature, as it prevents RST responses to all packets, not just SYNs.