QT socket does no read all data

Question

I want to read the data through socket in Qt. I am using QBytearray to store the data. Actually server sends 4095 bytes in a single stretch, but in the QT client side I am receiving in different chunks because of my application design.

void Dialog::on_pushButton_clicked()
{
socket=new QTcpSocket(this);
socket->connectToHost("172.17.0.1",5000);
if(socket->waitForConnected(-1))
qDebug()<<"Connected";
Read_data();
}

void Dialog::Read_data()
{
QString filename(QString("%1/%2.bin").arg(path,device));
qDebug()<<"filename"<<filename;
QFile fileobj(filename);

int cmd,file_size,percentage_completed;
if(!fileobj.open(QFile::WriteOnly | QFile::Text))
{
    qDebug()<<"Cannot open file for writting";
    return;
}
QTextStream out(&fileobj);
while(1)
{

socket->waitForReadyRead(-1);
byteArray=socket->read(4);
qDebug()<<"size of bytearray"<<byteArray.size();

length=0xffff & ((byteArray[3]<<8)|(0x00ff & byteArray[2]));
int rem;
byteArray=socket->read(length);
while(byteArray.size()!=length)
{    

rem=length-byteArray.size();

byteArray.append( socket->read(rem));

}
fileobj.write(byteArray);
fileobj.flush();
byteArray.clear();
}
}

server code:

#include<stdio.h>
#include<stdlib.h>
#include<fcntl.h>
#include<sys/ioctl.h>
#include<mtd/mtd-user.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <arpa/inet.h>
#include<math.h>
#include <netinet/tcp.h>
static int msb,lsb,size,listenfd = 0, connfd = 0,len;
main()
{

struct sockaddr_in serv_addr; 
serverlen=sizeof(serv_addr);
listenfd = socket(AF_INET, SOCK_STREAM, 0);
memset(&serv_addr, '0', sizeof(serv_addr)); 
serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
serv_addr.sin_port = htons(5000);

if(bind(listenfd,(struct sockaddr*)&serv_addr, sizeof(serv_addr))<0)
{
  perror("\n Error in binding"); 
  exit(1);
}

size=100000;
listen(listenfd, 1);

fd=fopen(new.bin,"r");  
len=4089;

while(1)
{

 buff[0]=25;
 buff[1]=2;
 buff[2]=60;
 buff[3]=47;
 n=fread(buff+4,1,length, fd);
 buff[len+4]=5;
 buff[len+5]='\n';
 if(n>0)
 sent_bytes=send(connfd,buff,n+6,0);
 size =size-len;
 if(size==0)
 break;
 }
 }

If I execute the code in localhost(127.0.0.1) I can receive the data fully. The problem arises only when I connect to different host IP. Kindly help me in this regard

EDIT 1: The problem is when bytesAvailable() returns the maximum bytes I am waiting for waitForReadyRead() times out. It works fine if the bytesAvailable() is less than as expected. Does bytesAvailable() allocate any buffer annoyed by this behaviour.

while(1)
{
while(socket->bytesAvailable()<4)
{
    if (!socket->waitForReadyRead())
    {
        qDebug() << "waitForReadyRead() timed out";
        return;
    }
}
byteArray=socket->read(4);
length=0xffff & ((byteArray[3]<<8)|(0x00ff & byteArray[2]));
int rem_bytes=length+2;
qDebug()<<"bytes available"<<socket->bytesAvailable();
while(socket->bytesAvailable()<=rem_bytes)
{
    qDebug()<<"reading";
    if (!socket->waitForReadyRead(10000))//times out here if bytesAvailable() == rem_bytes but executes well in other cases
    {
        qDebug() << "waitForReadyRead() timed out";
        return;
    }
    qDebug()<<"ready";
    byteArray.append(socket->read(rem_bytes));
    qDebug()<<"size of bytearray"<<byteArray.size();
    if(byteArray.size()==length+2)
    {
        for(int j=0;j<length;j++)
            newarray.append(byteArray[j]);
        fileobj.write(newarray);
        fileobj.flush();
        newarray.clear();
        byteArray.clear();
        break;                  
    }
    else
    {
        rem_bytes -=byteArray.size();
    }

}
Send();
}

I have tried by sending different data sizes cannot figure it out why?. Please provide me a solution pointing where I have gone wrong

Answer 1

Your problem stems from your misunderstanding of how TCP works.

When data is transmitted from a sender, it is broken into packets and then each packet is transmitted one by one until all the data has finished sending. If packets go missing, they are re-transmitted until either they reach their destination, or a timeout is reached.

As an added complication, each packet might follow various routes before arriving at the destination. The receiver has the task of acknowledging to the sender that packets have been received and then making sure that the packets are joined back together in the correct order.

For this reason, the longer the network route, the greater the chance of getting a delay in getting the data re-assembled. This is what you've been experiencing with your localhost versus networked-computer tests.

The IP stack on your computer does not wait for the complete data to arrive before passing it to your application but it will pause if it's missing a packet in sequence.

e.g. If you have 10 packets and packet 4 arrives last, the IP stack will pass the data to your application in two sets: 1-2-3, [[wait for 4 to arrive]], 4-5-6-7-8-9-10.

For this reason, when waitForReadyRead() returns true, you cannot expect that all your data has arrived, you must always check how many bytes have been actually received.

There are two places in your code where you wait for data. The first thing you wait for is a four-byte number to tell you how much data has been sent. Even though it's highly likely that you will have received all four bytes, it's good practice to check.

while(socket.bytesAvailable() < 4){
    if (!socket.waitForReadyRead()) { // timeout after 30 second, by default
        qDebug() << "waitForReadyRead() timed out";
        return;
    }
}
byteArray=socket->read(4);
qDebug()<<"size of bytearray"<<byteArray.size();

length=0xffff & ((byteArray[3]<<8)|(0x00ff & byteArray[2]));

The next thing you need to do is keep cycling through a wait-read-wait-read loop until all your data has arrived, each time keeping track of how many bytes you still expect to receive.

int bytesRemaining = length;

while(socket->bytesAvailable() < bytesRemaining){
    if (!socket->waitForReadyRead()){
        qDebug() "waitForReadyRead() timed out";
        return;
    }

    // calling read() with the bytesRemaining argument will not guarantee
    // that you will receive all the data. It only means that you will 
    // receive AT MOST bytesRemaining bytes.
    byteArray = socket->read(bytesRemaining);

    bytesRemaining -= byteArray.size();

    fileobj.write(byteArray);
    fileobj.flush();
}

All this said, you should not use the blocking API in your main thread or your GUI could freeze up. I suggest either using the asynchronous API, or create a worker thread to handle the downloading (and use the blocking API in the worker thread).

To see examples of how to use the two different APIs, looking in the documentation for the Fortune Client Example and the Blocking Fortune Client Example.

EDIT: My apologies, there's a bug in the code above that doesn't take an number of possibilities into account, most importantly, if all data has already been received, and the end game once all data has finally arrived.

The following one-line change should clear that up:

Change

while(socket->bytesAvailable() < bytesRemaining){

To

while (bytesRemaining > 0) {