Post request with multipart/form-data in appengine python not working

Question

I'm attempting to send a multipart post request from an appengine app to an external (django) api hosted on dotcloud. The request includes some text and a file (pdf) and is sent using the following code

from google.appengine.api import urlfetch
from poster.encode import multipart_encode
from libs.poster.streaminghttp import register_openers

register_openers()
file_data = self.request.POST['file_to_upload']
the_file = file_data
send_url = "http://127.0.0.1:8000/"
values = {
          'user_id' : '12341234',
          'the_file' : the_file
          }

data, headers = multipart_encode(values)
headers['User-Agent'] = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
data = str().join(data)
result = urlfetch.fetch(url=send_url, payload=data, method=urlfetch.POST, headers=headers)
logging.info(result.content)

When this method runs Appengine gives the following warning (I'm not sure if it's related to my issue)

Stripped prohibited headers from URLFetch request: ['Content-Length']

And Django sends through the following error

<class 'django.utils.datastructures.MultiValueDictKeyError'>"Key 'the_file' not found in <MultiValueDict: {}>"

The django code is pretty simple and works when I use the postman chrome extension to send a file.

@csrf_exempt
def index(request):
    try:
        user_id = request.POST["user_id"]
        the_file = request.FILES["the_file"]
        return HttpResponse("OK")
    except:
        return HttpResponse(sys.exc_info())

If I add

print request.POST.keys()

I get a dictionary containing user_id and the_file indicating that the file is not being sent as a file. if I do the same for FILES i.e.

print request.FILES.keys()    

I get en empty list [].

EDIT 1:

I've changed my question to implement the suggestion of someone1 however this still fails. I also included the headers addition recommended by the link Glenn sent, but no joy.

EDIT 2:

I've also tried sending the_file as variations of

the_file = file_data.file
the_file = file_data.file.read()

But I get the same error.

EDIT 3:

I've also tried editing my django app to

the_file = request.POST["the_file"]

However when I try to save the file locally with

path = default_storage.save(file_location, ContentFile(the_file.read()))

it fails with

<type 'exceptions.AttributeError'>'unicode' object has no attribute 'read'<traceback object at 0x101f10098>

similarly if I try access the_file.file (as I can access in my appengine app) it tells me

<type 'exceptions.AttributeError'>'unicode' object has no attribute 'file'<traceback object at 0x101f06d40>

Answer 1

Here is some code I tested locally that should do the trick (I used a different handler than webapp2 but tried to modify it to webapp2. You'll also need the poster lib found here http://atlee.ca/software/poster/):

In your POST handler on GAE:

from google.appengine.api import urlfetch
from poster.encode import multipart_encode
payload = {}
payload['test_file'] = self.request.POST['test_file']
payload['user_id'] = self.request.POST['user_id']
to_post = multipart_encode(payload)
send_url = "http://127.0.0.1:8000/"
result = urlfetch.fetch(url=send_url, payload="".join(to_post[0]), method=urlfetch.POST, headers=to_post[1])
logging.info(result.content)

Make sure your HTML form contains method="POST" enctype="multipart/form-data". Hope this helps!

EDIT: I tried using the webapp2 handler and realized the way files are served are different than how the framework I used to test with works (KAY). Here is updated code that should do the trick (tested on production):

import webapp2
from google.appengine.api import urlfetch
from poster.encode import multipart_encode, MultipartParam

class UploadTest(webapp2.RequestHandler):
  def post(self): 
    payload = {}
    file_data = self.request.POST['test_file']
    payload['test_file'] = MultipartParam('test_file', filename=file_data.filename,
                                          filetype=file_data.type,
                                          fileobj=file_data.file)
    payload['name'] = self.request.POST['name']
    data,headers= multipart_encode(payload)
    send_url = "http://127.0.0.1:8000/"
    t = urlfetch.fetch(url=send_url, payload="".join(data), method=urlfetch.POST, headers=headers)
    self.response.headers['Content-Type'] = 'text/plain'
    self.response.out.write(t.content)
  def get(self):
    self.response.out.write("""
    <html>
        <head>
            <title>File Upload Test</title>
        </head>
        <body>
            <form action="" method="POST" enctype="multipart/form-data">
                <input type="text" name="name" />
                <input type="file" name="test_file" />
                <input type="submit" value="Submit" />
            </form>
        </body>
    </html>""")

Answer 2

You're urlencoding the data where you should be multipart_encoding it. Have a look at this: Trying to post multipart form data in python, won't post