telnet client connection stops receiveing data, server is still sending

Question

I'm Working in an embedded linux environment.

it launches a telnet daemon on startup which watches on a particular port and launches a program when a connection is received.

i.e.

telnetd -l /usr/local/bin/PROGA -p 1234

PROGA - will output some data at irregular intervals. When it is not outputting data, every X period of time it sends out a 'heartbeat' type string to let the client know that we are still active i.e. "heartbeat\r\n"

After a random amount of time, the client (use a linux version of telnet, launched by: telnet xxx.xxx.xxx.xxx 1234) will fail to receive the 'heartbeat\r\n'

The data the client sees:

heartbeat  
heartbeat  
heartbeat  
...
heartbeat
[nothing, should have received heartbeat]
[nothing forever]

heartbeat is sent:

result = printf("%s", heartbeat);

checking result, it is always the length of heartbeat. Logging to syslog shows us that the printf() is executing with success at the proper intervals

I've since added in a tcdrain and fflush which both return success, but do not seem to help the situation.

Any help would be appreciated.

**UDPATE: got a wireshark capture from the server side. Very Clearly the heartbeat is being sent continuously. No Hicups, no delays. Found something interesting on the client though. The client in this test case (telnet on Ubuntu 9.04) seems to suddenly stop receiving heartbeat (as describes above). Wireshark confirms this, big pause in packets. Well, once the client had stopped receiving the heartbeat, pressing any keystroke (on the client) seems to trigger a spew of data from the client's buffer (all heartbeats). Wireshark on the client also shows this massive amount of data all in one packet.

Unfortunately I don't really know what this means. It this a line mode on/off thing? Line endings (\r\n) are very clearly coming through.

**Update 2: running netcat instead of telnetd, the problem is not reproducible.

Answer 1

The first thing I would do is get out Wireshark and try to find out if the server is truly sending the message. It would be instructive to run Wireshark at the server as well as third party PC. Is there anything different about the last heartbeat?

---

Edit. Well, that was an interesting find on your client.

It seems like there's some sort of terminal thing in the way. You may want to use the netcat program rather than telnetd. netcat is designed for sending arbitrary data over a TCP session in raw mode, without any special formatting, and it has the ability to hook up an arbitrary process to a socket. On a Windows machine you can use PuTTY in raw mode to accomplish the same thing.

It may still be worth examining traffic with a third party between your client and server. The kernel may be optimizing away writes to the network and internally buffering data. That's the only way to ensure that what see is what's really happening on the wire.