Upload a file over 2.15 GB in R

Question

I've got a manual process where I'm uploading 5-6 GB file to a web server via curl:

curl -X POST --data-binary @myfile.csv http://myserver::port/path/to/api

This process works fine, but I'd love to automate it using R. The problem is, I either don't know what I'm doing, or the R libraries for curl don't know how to handle files bigger than ~2GB:

library(RCurl)
postForm(
     "http://myserver::port/path/to/api",
      file = fileUpload(
        filename = path.expand("myfile.csv"),
        contentType = "text/csv"
      ),.encoding="utf-8")

Yeilds Error: Internal Server Error

httr doesn't work either:

library(httr)
POST(
      url = "http://myserver:port/path/to/api",
      body = upload_file(
        path =  path.expand("myfile.csv"),
        type = 'text/csv'),
      verbose()
    )

Which yields:

Response [http://myserver:port/path/to/api]
  Date: 2015-06-30 11:11
  Status: 400
  Content-Type: <unknown>
<EMPTY BODY>

httr is a little more informative with the verbose() option, telling me:

-> POST http://myserver:port/path/to/api
-> User-Agent: libcurl/7.35.0 r-curl/0.9 httr/1.0.0
-> Host: http://myserver::port
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: text/csv
-> Content-Length: -2147483648
-> Expect: 100-continue
-> 
<- HTTP/1.1 400 Bad Request
<- Server: Apache-Coyote/1.1
<- Transfer-Encoding: chunked
<- Date: Tue, 30 Jun 2015 11:11:11 GMT
<- Connection: close
<- 

The Content-Length: -2147483648 looks suspiciously like a 32 bit integer overflow, so I think this is a bug in httr. I suspect RCurl is experiencing a similar failure.

I'd really love a minimal wrapper around curl -X POST --data-binary, but barring that, what are my options for uploading fairly large files from R?

Answer 1

This bug is fixed in the dev version of httr/curl:

devtools::install_github("jeroenooms/curl")
devtools::install_github("hadley/httr")

This is a bug in the httr and curl packages for R. The bug has been fixed on GitHub as of July 2, 2015, and the change will roll out to CRAN soon.

It is also possible I was calling RCurl incorrectly in the above command, but I could never figure out the correct invocation.