Google App Engine urlfetch to POST files

Question

I am trying to send a file to torrage.com from an app in GAE. the file is stored in memory after being received from a user upload.

I would like to be able to post this file using the API available here: http://torrage.com/automation.php but i am having some problems undestanding how the body of the post should be encoded, the most i got from the API is a "file empty" message.

Answer 1

I find torrage's API docs on the POST interface (as opposed to the SOAP one) pretty confusing and conflicting with the sample C code they also supply. It seems to me that in their online example of PHP post they are not sending the file's contents (just like @kender's answer above is not sending it) while they ARE sending it in the SOAP examples and in the example C code.

The relevant part of the C sample (how they compute the headers that you'd be passing to urlfetch.fetch) is:

  snprintf(formdata_header, sizeof(formdata_header) - 1,
    "Content-Disposition: form-data; name=\"torrent\"; filename=\"%s\"\n"
    "Content-Type: " HTTP_UPLOAD_CONTENT_TYPE "\n"
    "\n",
    torrent_file);
  http_content_len = 2 + strlen(content_boundary) + 1 + strlen(formdata_header) + st.st_size + 1 + 2 + strlen(content_boundary) + 3;
  LTdebug("http content len %u\n", http_content_len);
  snprintf(http_req, sizeof(http_req) - 1, 
    "POST /%s HTTP/1.1\n"
    "Host: %s\n"
    "User-Agent: libtorrage/" LTVERSION "\n"
    "Connection: close\n"
    "Content-Type: multipart/form-data; boundary=%s\n"
    "Content-Length: %u\n"
    "\n",
    cache_uri, cache_host, content_boundary, http_content_len);

"application/x-bittorrent" is the HTTP_UPLOAD_CONTENT_TYPE. st.st_size is the number of bytes in the memory buffer with all the file's data (the C sample reads that data from file, but it doesn't matter how you got it into memory, as long as you know its size). content_boundary is a string that's NOT present in the file's contents, they build it as "---------------------------%u%uLT" with each %u substituted by a random number (repeating until that string hits upon two random numbers that make it not present in the file). Finally, the post body (after opening the HTTP socket and sending the other headers) they write as follows:

  if (write_error == 0) if (write(sock, "--", 2) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, content_boundary, strlen(content_boundary)) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, "\n", 1) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, formdata_header, strlen(formdata_header)) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, filebuf, st.st_size) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, "\n--", 3) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, content_boundary, strlen(content_boundary)) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, "--\n", 3) <= 0) write_error = 1;

where filebuf is the buffer with the file's contents.

Hardly sharp and simple, but I hope there's enough info here to work out a way to build the arguments for a urlfetch.fetch (building them for a urllib.urlopen would be just as hard, since the problem is the scarcity of documentation about exactly what headers and what content and how encoded you need -- and that not-well-documented info needs to be reverse engineered from what I'm presenting here, I think).

Alternatively, it may be possible to hack a SOAP request via urlfetch; see here for Carson's long post of his attempts, difficulties and success in the matter. And, good luck!