Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using Python to Know When a File Has Completely Been Received From an FTP Source

Tags:

python

ftp

I am using Python to develop an application that does the following:

  • Monitors a particular directory and watches for file to be transferred to it. Once the file has finished its transfer, run some external program on the file.

The main issue I have developing this application is knowing when the file has finished transferring. From what I know the file will be transferred via SFTP to a particular directory. How will Python know when the file has finished transferring? I know that I can use the st_size attribute from the object returned by os.stat(fileName) method. Are there more tools that I need to use to accomplish these goals?

like image 392
FearlessFuture Avatar asked Aug 29 '13 00:08

FearlessFuture


People also ask

How can I tell if a file is completed on a FTP server?

There is no way to be absolutely certain a file being written by the FTP daemon is complete. It's even possible that the file transfer failed and then gets restarted and completed. You must poll the file's size and set a time limit, say 5 minutes.

How do I transfer files using FTP in Python?

FTP(File Transfer Protocol) To transfer a file, 2 TCP connections are used by FTP in parallel: control connection and data connection. For uploading and downloading the file, we will use ftplib Module in Python. It is an in-built module in Python.

How do I download from SFTP to local in Python?

Here, we first move to the targeted directory using the conn.cd() method. Then, we just pass the file name to the conn. get() method. This leads to the downloading of that file into the local directory of our client-side local machine.


1 Answers

I ended up using a combination of watchdog and waited until I can open the file for writing

 #If there is no error when trying to read the file, then it has completely loaded
    try:
        with io.FileIO(fileName, "r+") as fileObj:

            '''
            Deal with case where FTP client uses extensions such as ".part" and '.filepart" for part of the incomplete downloaded file.
            To do this, make sure file exists before adding it to list of completedFiles.
            '''
            if(os.path.isfile(fileName)):
                completedFiles.append(fileName)
                print "File=" + fileName + " has completely loaded."
    except IOError as ioe:
        print str(ioe)
like image 160
FearlessFuture Avatar answered Oct 02 '22 19:10

FearlessFuture