python requests: post and big content

Question

I am sending a CSV file to a server using a POST request.

I am using a file-like object with requests.post

Will there be a problem if the CSV file is quite big and I have limited memory or the fact that I use a file-like object will never load the whole file in memory? I am not sure about that.

I know there is the stream option but it sounds like it's more for getting the response and not sending data.

headers = {
    'content-type': 'text/csv',
}
csvfile = '/path/file.csv'
with open(csvfile) as f:
    r = requests.post(url, data=f, headers=headers)

Answer 1

Using an open file object as the data parameter ensures that requests will stream the data for you.

If a file size can be determined (via the OS filesystem), the file object is streamed using a 8kb buffer. If no filesize can be determined, a Transfer-Encoding: chunked request is sent sending the data per line instead (the object is used as an iterable).

If you were to use the files= parameter for a multipart POST, on the other hand, the file would be loaded into memory before sending. Use the requests-toolbelt package to stream multi-part uploads:

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

csvfile = '/path/file.csv'
with open(csvfile) as f:
    m = MultipartEncoder(fields={'csv_field_name': ('file.csv', f, 'text/csv')})
    headers = {'Content-Type': m.content_type}
    r = requests.post(url, data=m, headers=headers)

Answer 2

This will not load the entire file into memory, it will be split into chunks and transmitted a little at a time. You can see this in the source code here.