HTTP PUT with base64 encoded attachment using requests

Question

I'm using requests to perform an HTTP PUT on a file, but for some reason, it is uploading the raw ASCII and not base64-encoding it first.

files = {'file': ('mydata.csv', open('mydata.csv', 'rb'))}  
...  
try:
    logging.info("Upload URL: " + insert_upload_url)
    headers = {'Content-type': 'multipart/form-data'}
    upload_res = requests.put(insert_upload_url, files=files,
                              data={'insertUpload': data}, headers=headers)
    logging.info("Status: " + str(upload_res.status_code))
    if upload_res.status_code != requests.codes.ok:
        logging.info("Reason: " + upload_res.reason)
    else:
        logging.info("Response: " + upload_res.text)
except Exception as e:
    logging.info("Error: " + e.message)

When I dump the raw HTTP request, I see that the data is not being encoded as application/octet-stream and not in base64:

PUT /apibatchmember/services/rest/batchmemberservice/G9X7CsNn3HisxFdwAu4W76mBewUkQcKD_limGK1MDZ-eW3-olsn8HW0l5oePvEU_ZwHaPbEz2_c1YonjauDs7Jhk9DGvGNSTLMbgSSY9TGVuk00I_-tKPw8mjoXzC63YsFzBIYIeYXHKf34dYxmmhz4iSeDw/batchmember/insertUpload HTTP/1.1
Host: api.example.com
Content-Length: 5062
Content-type: multipart/form-data
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.2.3 CPython/2.7.5 Darwin/12.5.0

--5f934d42eabb4d7abe8bdef2cea94b6b
Content-Disposition: form-data; name="insertUpload"

<?xml version="1.0" encoding="UTF-8"?>
<insertUpload>
    <criteria>LOWER(EMAIL)</criteria>
    <fileName>email.csv</fileName>
    <separator>,</separator>
    <fileEncoding>UTF-8</fileEncoding>
    <skipFirstLine>false</skipFirstLine>
    <dateFormat>mm/dd/YYYY</dateFormat>
    <mapping>
        <column>
            <colNum>0</colNum>
            <fieldName>CUSTNUM</colNum>
        <column>
        <column>
            <colNum>1</colNum>
            <fieldName>FIRSTNAME</colNum>
        <column>
        <column>
            <colNum>2</colNum>
            <fieldName>LASTNAME</colNum>
        <column>
        <column>
            <colNum>3</colNum>
            <fieldName>EMAIL</colNum>
        <column>
    </mapping>
</insertUpload>
--5f934d42eabb4d7abe8bdef2cea94b6b
Content-Disposition: form-data; name="file"; filename="email.csv"
Content-Type: text/csv

\xef\xbb\xbf1045,Janice,Waddell,[email protected]
1156,Scott,Sheldon,[email protected]
1267,Adrianus,Lengkeek,[email protected]
1295,EDWIN,ODIFE,[email protected]
1345,Albert,Stephenson,[email protected]
...

--5f934d42eabb4d7abe8bdef2cea94b6b--

How can I get that text/csv part to be a base64-encoded stream?

Thanks!

UPDATE

I am getting the correct attachment header now using the following:

with open('/Users/mark.richman/email.csv', 'rb') as fd:
     b64data = base64.b64encode(fd.read())

files = {'file': ('email.csv', b64data, 'application/octet-stream')}

However, I still don't get Content-Transfer-Encoding: base64 in the file attachment header. Any idea?

UPDATE 2

I had to hack in the header as such: files = {'file': ('email.csv', b64data, 'application/octet-stream\r\nContent-Transfer-Encoding: base64')}

I see the header in the HTTP Dump, but I am still getting back HTTP 415 Unsupported Media Type.

UPDATE 3

It looks like requests needs an API update to support what I'm trying to do here. The first part of the multipart (the XML data) needs to have Content-Type: text/xml set, and the API currently doesn't support this. Discussion over here.

Answer 1

You have to do

import base64


response = requests.put(url, data={'insertUpload': base64.b64encode(data)}, files=files, ...)

You don't define data though so if you're attempting to use a file object then this will not work.

If data is a string then this will work fine.

---

I misunderstood your question initially.

So if you're not trying to avoid loading the whole file into memory (which it seems like you are), you could just do

with open('mydata.csv', 'rb') as fd:
     b64data = base64.b64encode(fd.read())

files = {'file': ('mydata.csv', b64data, 'application/octet-stream')}
requests.put(...)  # everything here is the same as what you did

On the other hand if you're looking to encode the data without loading it entirely into memory then there is no existing method for this. For the entire contents to be base64 encoded correctly it needs everything in the file first. Consider (for example) the following:

print(base64.b64encode('line\n'))
# => bGluZQo=
print(base64.b64encode('line\nline\n'))
# => bGluZQpsaW5lCg==

They are related but they're not capable of simply being concatenated. You might have better luck scoping the issue at hand to just using making a File-like object that will base64 encode text on the fly and such. Good luck!