paramiko sftp server connection dropped

Question

I am trying to copy big files (~650mb) files using paramiko sftp. I am able to copy small files, but I see the following error when I try to copy big files. I am able to copy the file by using sftp directly form terminal.

Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp_file.py", line 453, in _prefetch_thread
    self.sftp._async_request(self, CMD_READ, self.handle, long(offset), int(length))
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 656, in _async_request
    self._send_packet(t, str(msg))
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp.py", line 172, in _send_packet
    self._write_all(out)
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp.py", line 136, in _write_all
    n = self.sock.send(out)
  File "/usr/lib/python2.7/dist-packages/paramiko/channel.py", line 701, in send
    self.transport._send_user_message(m)
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1426, in _send_user_message
    self._send_message(data)
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1406, in _send_message
    self.packetizer.send_message(data)
  File "/usr/lib/python2.7/dist-packages/paramiko/packet.py", line 310, in send_message
    self.write_all(out)
  File "/usr/lib/python2.7/dist-packages/paramiko/packet.py", line 260, in write_all
    raise EOFError()
EOFError

Traceback (most recent call last):
  File "./extract.py", line 107, in <module>
    main()
  File "./extract.py", line 81, in main
    getFiles()
  File "./extract.py", line 58, in getFiles
    'ABC.zip')
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 614, in get
    data = fr.read(32768)
  File "/usr/lib/python2.7/dist-packages/paramiko/file.py", line 153, in read
    new_data = self._read(read_size)
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp_file.py", line 152, in _read
    data = self._read_prefetch(size)
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp_file.py", line 132, in _read_prefetch
    self.sftp._read_response()
  File "/usr/lib/python2.7/dist-packages/paramiko/sftp_client.py", line 668, in _read_response
    raise SSHException('Server connection dropped: %s' % (str(e),))
paramiko.SSHException: Server connection dropped: 

Also this is thrown only after reading certain number of bytes. I introduced a print statement inside sftp_client.py and this is the output (bytes)

reading bytes data_size: 11
reading bytes data_size: 28
reading bytes data_size: 28
reading bytes data_size: 32776
reading bytes data_size: 32776
reading bytes data_size: 32776
reading bytes data_size: 32776
reading bytes data_size: 32776
reading bytes data_size: 32776
reading bytes data_size: 32776

It is not clear to me why the connection gets dropped in the middle of the file. Any suggestions on how to handle this?

Thanks!

Answer 1

I had the same issue. Here's my solution for paramiko version 3.3.0 and above.

The sftp_client.get() method now has the parameter max_concurrent_prefetch_requests. The default argument is None (unlimited).

Passing 64 as a limit worked for me. As described in the docstring, this is the same as the value in the OpenSSH standard.

Answer 2

Sorry, not enough reputation to upvote or comment, but I wanted to confirm that yes the Paramiko version 3.3.0 now includes an excellent solution to this issue. This new parameter quickly solved this age old problem I was having which numerous people had made work around solution.

Here is the Paramiko documentation which covers this new parameter max_concurrent_prefetch_requests: https://docs.paramiko.org/en/latest/api/sftp.html?highlight=max_concurrent_prefetch_requests#paramiko.sftp_client.SFTPClient.get

Here is the github page discussing the issue which eventually got solved, in case anyone wants more info on it: https://github.com/paramiko/paramiko/pull/2058