Missing one-pixel row while transfering image with TCP socket

Question

I'm facing strange error right now, I have python script, that is sending/receiving data using TCP socket, everything works fine, but when I'm trying to download image with this script, it will download it, but there is a missing one-pixel row. Any ideas on how to fix it?

Server download script:

    def download(self, cmd):
        try:
            self.c.send(str.encode(cmd))
            command,filename=cmd.split(' ')
            nFile = open(filename, 'wb')
            i = self.c.recv(1024)
            while not ('complete' in str(i)):   
                nFile.write(i)
                i = self.c.recv(1024)
            nFile.close()
            self.reset = True
            print('\nGot that file')
        except Exception as e:
            print(e)

Client upload script:

   def upload(self, filename):
    try:
        fd = open(filename, 'rb')
        data = fd.read(1024)
        while (data):
            self.s.sendall(data)
            data = fd.read(1024)
        self.s.send(str.encode('complete'))
        fd.close()
    except Exception as e:
        print(e)

EXAMPLE - You can see, that last row of pixels is missing:

SOLUTION(1): It's not a solution, just workaround, use the other one!

What happens if you remove the complete part of the payload before writing the last chunk of data to nFile? – mtrw

The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.

Client upload script:

   def upload(self, filename):
try:
    fd = open(filename, 'rb')
    data = fd.read(1024)
    while (data):
        self.s.sendall(data)
        data = fd.read(1024)
    sleep(0.2)
    self.s.send(str.encode('complete'))
    fd.close()
except Exception as e:
    print(e)

SOLUTION(2):

TCP is a stream protocol with no message boundaries. This means that multiple sends can be received in one recv call, or one send can be received in multiple recv calls.

The delay work-around may not work reliably. You need to delimit messages in the stream.

– Maxim Egorushkin

Server download script:

try:
    msg_header = self.c.recv(4)
    while len(msg_header) != 4:
        msg_header += self.c.recv(4 - len(msg_header))
    file_len = struct.unpack('<I', msg_header)[0]
    nFile = open(filename, 'wb')
    data = self.c.recv(file_len)
    while len(data) != file_len:
        data += self.c.recv(file_len - len(data))
    nFile.write(data)
    nFile.close()
    print('\nGot that file')
except Exception as e:
    print(e)

Client upload script:

try:
    file_len = os.stat(filename).st_size
    msg_header = struct.pack('<I', file_len)
    self.s.sendall(msg_header)
    fd = open(filename, 'rb')
    data = fd.read(file_len)
    while (data):
        self.s.sendall(data)
        data = fd.read(file_len)
    fd.close()
except Exception as e:
    print(e)

Answer 1

The problem was with sending 'complete' string to the server, because the script had not enough time to get all bytes from the image. So one way to fix this is to add sleep(0.2) to the script.

TCP is a stream protocol with no message boundaries. This means that multiple sends can be received in one recv call, or one send can be received in multiple recv calls.

The delay work-around may not work reliably. You need to delimit messages in the stream.

There are 2 common ways of delimiting messages in a stream:

1. Prefix messages with a header.
2. End messages with a suffix.

---

Since you are sending binary data any suffix can naturally be present in the payload. Unless the suffix is longer than the payload, which isn't practical.

Hence, what you may like to do here is prefix a fixed-size header to your payload. In this particular case, a header with a 4-byte binary file length would suffice. E.g.:

file_len = os.stat(filename).st_size
msg_header = struct.pack('<I', file_len)
self.s.sendall(msg_header)

The receiver needs to read the header first:

msg_header = self.s.recv(4)
while len(msg_header) != 4:
    msg_header += self.s.recv(4 - len(msg_header))
file_len = struct.unpack('<I', msg_header)

And then read exactly file_len from the socket.

Knowing the size of the file being received also allows you to preallocate the buffer to avoid memory reallocations and/or preallocate the entire file to minimize disk fragmentation or avoid out of disk space error after the file transfer has started.