python: HTTP PUT with unencoded binary data

Question

I cannot for the life of me figure out how to perform an HTTP PUT request with verbatim binary data in Python 2.7 with the standard Python libraries.

I thought I could do it with urllib2, but that fails because urllib2.Request expects its data in application/x-www-form-urlencoded format. I do not want to encode the binary data, I just want to transmit it verbatim, after the headers that include

Content-Type: application/octet-stream
Content-Length: (whatever my binary data length is)

This seems so simple, but I keep going round in circles and can't seem to figure out how.

How can I do this? (aside from open up a raw binary socket and write to it)

Answer 1

I found out my problem. It seems there is some obscure behavior in urllib2.Request / urllib2.urlopen() (at least in Python 2.7)

The urllib2.Request(url, data, headers) constructor seems to expect the same type of string in its url and data parameters.

I was giving the data parameter raw data from a file read() call (which in Python 2.7 returns it in the form of a 'plain' string), but my url was accidentally Unicode because I concatenated a portion of the URL from the result of another function which returned Unicode strings.

Rather than trying to "downcast" url from Unicode -> plain strings, it tried to "upcast" the data parameter to Unicode, and it gave me a codec error. (oddly enough, this happens on the urllib2.urlopen() function call, not the urllib2.Request constructor)

When I changed my function call to

# headers contains `{'Content-Type': 'application/octet-stream'}`
r = urllib2.Request(url.encode('utf-8'), data, headers)

it worked fine.

Answer 2

You're misreading the documentation: urllib2.Request expects the data already encoded, and for POST that usually means the application/x-www-form-urlencoded format. You are free to associate any other, binary data, like this:

import urllib2

data = b'binary-data'
r = urllib2.Request('http://example.net/put', data,
                    {'Content-Type': 'application/octet-stream'})
r.get_method = lambda: 'PUT'
urllib2.urlopen(r)

This will produce the request you want:

PUT /put HTTP/1.1
Accept-Encoding: identity
Content-Length: 11
Host: example.net
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7

binary-data

Answer 3

Have you considered/tried using httplib?

HTTPConnection.request(method, url[, body[, headers]])

This will send a request to the server using the HTTP request method method and the selector url. If the body argument is present, it should be a string of data to send after the headers are finished. Alternatively, it may be an open file object, in which case the contents of the file is sent; this file object should support fileno() and read() methods. The header Content-Length is automatically set to the correct value. The headers argument should be a mapping of extra HTTP headers to send with the request.