Getting a OSError when trying to LIST ftp directories in Python

Question

Using Python 3.6.3.

I am connecting to an FTP to using ftplib via FTP_TLS.

ftps = ftplib.FTP_TLS('example.com', timeout=5)

# TLS is more secure than SSL
ftps.ssl_version = ssl.PROTOCOL_TLS

# login before securing control channel
ftps.login('[email protected]', 'password')

# switch to secure data connection
ftps.prot_p()

# Explicitly set Passive mode
ftps.set_pasv(True)

This all works. I can send ftps.mkd('mydir') (make directory) and ftps.cwd('mydir') (change working dir), and both work fine.

But if I send any of these (they're all basically synonyms as far as I can tell):

        ftps.dir()
        ftps.nlst()
        ftps.retrlines('LIST')
        ftps.retrlines('MLSD')

Then I get back an exception (below also includes all ftplib debug info as well; generally matches up with what FileZilla shows too):

*cmd* 'AUTH TLS'
*put* 'AUTH TLS\r\n'
*get* '234 AUTH TLS OK.\n'
*resp* '234 AUTH TLS OK.'
*cmd* 'USER [email protected]'
*put* 'USER [email protected]\r\n'
*get* '331 User [email protected] OK. Password required\n'
*resp* '331 User [email protected] OK. Password required'
*cmd* 'PASS ******************************'
*put* 'PASS ******************************\r\n'
*get* '230 OK. Current restricted directory is /\n'
*resp* '230 OK. Current restricted directory is /'
*cmd* 'PBSZ 0'
*put* 'PBSZ 0\r\n'
*get* '200 PBSZ=0\n'
*resp* '200 PBSZ=0'
*cmd* 'PROT P'
*put* 'PROT P\r\n'
*get* '200 Data protection level set to "private"\n'
*resp* '200 Data protection level set to "private"'
*cmd* 'MKD mydir'
*put* 'MKD mydir\r\n'
*get* '257 "mydir" : The directory was successfully created\n'
*resp* '257 "mydir" : The directory was successfully created'
*cmd* 'CWD mydir'
*put* 'CWD mydir\r\n'
*get* '250 OK. Current directory is /mydir\n'
*resp* '250 OK. Current directory is /mydir'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 TYPE is now ASCII\n'
*resp* '200 TYPE is now ASCII'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (8,8,8,8,8,8)\n'
*resp* '227 Entering Passive Mode (8,8,8,8,8,8)'
*cmd* 'MLSD'
*put* 'MLSD\r\n'
*get* '150 Accepted data connection\n'
*resp* '150 Accepted data connection'
Traceback (most recent call last):
  File "c:\my_script.py", line 384, in run_ftps
    ftps.retrlines('MLSD')
  File "c:\libs\Python36\lib\ftplib.py", line 485, in retrlines
    conn.unwrap()
  File "C:\libs\Python36\lib\ssl.py", line 1051, in unwrap
    s = self._sslobj.unwrap()
  File "C:\libs\Python36\lib\ssl.py", line 698, in unwrap
    return self._sslobj.shutdown()
OSError: [Errno 0] Error

The same FTP command (LIST) works fine via filezilla.

The closest thing I can find with googling is this: https://bugs.python.org/msg253161 - and I'm not sure if it's related or relevant.

Short version: What does "OSError: [Errno 0] Error" actually mean, and how do I list my directory contents?

Edit: The issue only seems to happen with FTP_TLS. It works fine via a plain FTP connection, but I need FTP_TLS.

Answer 1

The problem is probably that the FTP server requires that the TLS session in the new data channel is the same as the control channel. This has not been fixed in Python 3.7. Subclass ftplib.FTP_TLS as in the solution found here https://stackoverflow.com/a/43301750 with a small fix by me:

import ftplib
from ssl import SSLSocket


class ReusedSslSocket(SSLSocket):
    def unwrap(self):
        pass


class MyFTP_TLS(ftplib.FTP_TLS):
    """Explicit FTPS, with shared TLS session"""
    def ntransfercmd(self, cmd, rest=None):
        conn, size = ftplib.FTP.ntransfercmd(self, cmd, rest)
        if self._prot_p:
            conn = self.context.wrap_socket(conn,
                                            server_hostname=self.host,
                                            session=self.sock.session)  # reuses TLS session            
            conn.__class__ = ReusedSslSocket  # we should not close reused ssl socket when file transfers finish
        return conn, size

Use it like this:

ftps = MyFTP_TLS('example.com', timeout=5)

Answer 2

    s = self._sslobj.unwrap()
  File "E:\Software\_libs\Python36\lib\ssl.py", line 698, in unwrap
    return self._sslobj.shutdown()
OSError: [Errno 0] Error

This error happens if the server just closes the TCP connection without having done a proper TLS shutdown. OSError: [Errno 0] means that there was actually no error at all at the underlying TCP socket, i.e. the server did a normal shutdown of the TCP socket. Only, the server did not do the required TLS level shutdown before that.

This looks for me like a bug in the server implementation but maybe this is also a configuration option causing this. And, python ftplib seems to behave less tolerant in the face of such incorrect protocol implementation then other clients. But you can find similar reports with other clients by searching for "Server did not properly shut down TLS connection".

A fix might be to put the conn.unwrap() within retrlines and retrbinary in ftplib.py into a try...except statement and just ignore the error.

Answer 3

I ran into this problem today and wanted to add a bit to this solution.

I needed to have makepasv return the host instead of the internal IP address of the FTP server that is behind a firewall. This passv fix together with the TLS session fixes solved my issue.

class ReusedSslSocket(SSLSocket):
    def unwrap(self):
        pass
class FTP_TLS_IgnoreHost(FTP_TLS):
    def makepasv(self):
        _, port = super().makepasv()
        return self.host, port
    """Explicit FTPS, with shared TLS session"""
    def ntransfercmd(self, cmd, rest=None):
        conn, size = FTP.ntransfercmd(self, cmd, rest)
        if self._prot_p:
            conn = self.context.wrap_socket(conn,
                      server_hostname=self.host,
                      session=self.sock.session)  # reuses TLS session            
            conn.__class__ = ReusedSslSocket  # we should not close reused ssl socket when file transfers finish
        return conn, size

ftp = FTP_TLS_IgnoreHost(host, timeout=5)
ftp.ssl_version = ssl.PROTOCOL_TLS
ftp.auth()
ftp.login(user,passwd)
ftp.prot_p()
ftp.set_pasv(True)
pwd = ftp.pwd()
print(pwd)
print(ftp.nlst())
ftp.close()