python requests drops "data" when use of "header" in request

Question

I am working on REST API of a site that requires this request type when I want to upload a file:

1. 'Authorization' and multi-part content type in header
2. File as binary string in form (body)
3. File Type in request URL

So I did this code:

import requests

url = 'http://httpbin.org/post'

parameters = {
        'format': 'pdf',
        }

headers = {
  'Content-Type': 'multipart/form-data',
  'Accept': 'application/json',
  'Authorization' : 'Some authorization code'
        }

data = {'file': open('1.pdf', 'rb')}

r = requests.post(url, params=parameters, headers=headers, data=data)

print(r.text)

BUT it seems to requests is dropping data :

{
  "args": {
    "format": "pdf"
  },
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json",
    "Accept-Encoding": "gzip, deflate",
    "Authorization": "Some authorization code",
    "Connection": "close",
    "Content-Length": "30",
    "Content-Type": "multipart/form-data",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.18.1"
  },
  "json": null,
  "origin": "x.x.x.x",
  "url": "http://httpbin.org/post?format=pdf"
}

it works when I remove 'headers' part in request:

r = requests.post(url, params=parameters, data=data)

Because response is :

{
  "args": {
    "format": "pdf"
  },
  "data": "",
  "files": {},
  "form": {
    "fax_file": "some samplae texts\n"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "close",
    "Content-Length": "30",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.18.1"
  },
  "json": null,
  "origin": "x.x.x.x",
  "url": "http://httpbin.org/post?format=pdf"
}

I have tried prepared request too and result is same.

Answer 1

You are trying to post file data, so use the files option:

r = requests.post(url, params=parameters, files=data, headers=headers)

You should really not set the Content-Type header, however; it is set for you when you use the files option. The header in this case includes the field boundary, to really want the library to take care of this for you:

headers = {
  'Accept': 'application/json',
  'Authorization' : 'Some authorization code'
}

If you leave the Content-Type header in, you’d have to generate the content body up front to be able to supply the required boundary info to the receiving server.

You could experiment with dropping the Accept header too; by default requests will add Accept: */* if you don't specify that header, signalling that anything is acceptable.

When using only the data argument, the parameters are encoding to application/x-www-form-urlencoded form, which doesn't support large file data, and your Content-Type header doesn't match the actual POST body content.

See Post a Multipart-Encoded File in the requests documentation and application/x-www-form-urlencoded or multipart/form-data? here on Stack Overflow.

Demo:

>>> import requests
>>> url = 'http://httpbin.org/post'
>>> parameters = {'format': 'pdf'}
>>> headers = {
...   'Accept': 'application/json',
...   'Authorization' : 'Some authorization code',
... }
>>> data = {'file': open('1.pdf', 'rb')}
>>> r = requests.post(url, params=parameters, files=data, headers=headers)
>>> print(r.text)
{
  "args": {
    "format": "pdf"
  },
  "data": "",
  "files": {
    "file": "<file data as base64>"
  },
  "form": {},
  "headers": {
    "Accept": "application/json",
    "Accept-Encoding": "gzip, deflate",
    "Authorization": "Some authorization code",
    "Cache-Control": "max-age=0",
    "Connection": "close",
    "Content-Length": "374751",
    "Content-Type": "multipart/form-data; boundary=d4b84f8bfd464e3f97e3de584d7315fc",
    "Host": "httpbin.org",
    "O2Gw-Id": "03",
    "User-Agent": "python-requests/2.18.4",
    "X-Gateway": "wap.london.02.net"
  },
  "json": null,
  "origin": "10.120.6.78, 82.132.221.209",
  "url": "http://httpbin.org/post?format=pdf"
}

Note the multipart/form-data; boundary=d4b84f8bfd464e3f97e3de584d7315fc value for the Content-Type header!