Python memory issue uploading multiple files to API

Question

I'm running a script to upload 20k+ XML files to an API. About 18k in, I get a memory error. I was looking into it and found the memory is just continually climbing until it reaches the limit and errors out (seemingly on the post call). Anyone know why this is happening or a fix? Thanks. I have tried the streaming uploads found here. The empty strings are due to sensitive data.

def upload(self, oauth_token, full_file_path):
        file_name = os.path.basename(full_file_path)
        upload_endpoint = {'':''}
        params = {'': '','': ''}
        headers = {'': '', '': ''}
        handler = None
        try:
            handler = open(full_file_path, 'rb')
            response = requests.post(url=upload_endpoint[''], params=params, data=handler, headers=headers, auth=oauth_token, verify=False, allow_redirects=False, timeout=600)
            status_code = response.status_code
            # status checking
            return status_code
        finally:
            if handler:
                handler.close()

    def push_data(self):
        oauth_token = self.get_oauth_token()
        files = os.listdir(f_dir)
        for file in files:
            status = self.upload(oauth_token, file_to_upload)

Answer 1

What version of Python are you using? It looks like there is a bug in Python 3.4 causing memory leaks related to network requests. See here for a similar issue: https://github.com/psf/requests/issues/5215

It may help to update Python.