Android HttpUrlConnection: Post Multipart

Question

I'm trying to post a test File to a spring rest servlet deployed on tomcat using Android. I'm developing on Android 4.1.2, but I have verified same problem on 4.0.3.

The problem is that the file upload requires a very long time (about 70 seconds for a 4MB file), also in local network. The time is equiparable using a 3g connection. I've excluded that it could be a server problem: executing the same call with curl it takes 1 / 2 seconds, and using apache as backend results are the same.

Using HttpClient works fine.

I'm using Spring Android RestClient 1.0.1.RELEASE and, given Android version and the fact that I'm not overriding default behaviour, it uses HttpUrlConnection instead of HttpClient to make http requests.

I have also implemented my custom ClientHttpRequestFactory in order to manipulate some details of SSL connection and I have defined my own implementation of ClientHttpRequestInterceptor in order to modify authentication header.

I have also set setBufferRequestBody(false) in order to avoid OutOfMemoryException on big files. But this property have no effects on time required.

MyClientHttpRequestFactory:

public class MyClientHttpRequestFactory extends SimpleClientHttpRequestFactory{

    @Override
    protected void prepareConnection(HttpURLConnection connection,  String httpMethod) throws IOException {
        super.prepareConnection(connection, httpMethod);
        connection.setConnectTimeout(240 * 1000);
        connection.setReadTimeout(240 * 1000);


        if ("post".equals(httpMethod.toLowerCase())) {
            setBufferRequestBody(false);
        }else {
            setBufferRequestBody(true);
        }
    }

@Override
protected HttpURLConnection openConnection(URL url, Proxy proxy) throws IOException {
    final HttpURLConnection httpUrlConnection = super.openConnection(url, proxy);

    if (url.getProtocol().toLowerCase().equals("https")
        &&
        settings.selfSignedCert().get())
    {
        try {
            ((HttpsURLConnection)httpUrlConnection).setSSLSocketFactory(getSSLSocketFactory());
            ((HttpsURLConnection)httpUrlConnection).setHostnameVerifier(new NullHostnameVerifier());
        } catch (Exception e) {
            MyLog.e(LOG_TAG, "OpenConnection", e);
        } 
    } 

    return httpUrlConnection;
}

MyClientHttpRequestInterceptor:

public class MyClientHttpRequestInterceptor implements ClientHttpRequestInterceptor {

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
        final HttpHeaders headers = request.getHeaders();

        headers.setAuthorization(new HttpBasicAuthentication( settings.username().get(), settings.password().get()));

        if (settings.enable_gzip().get()) {
            headers.setAcceptEncoding(ContentCodingType.GZIP);
        }

        return execution.execute(request, body);
    }
}

And here my Rest call:

List<ClientHttpRequestInterceptor> interceptors = Arrays.asList((ClientHttpRequestInterceptor)myClientHttpRequestInterceptor);

RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new FormHttpMessageConverter());
restTemplate.getMessageConverters().add(new StringHttpMessageConverter());
restTemplate.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
restTemplate.setInterceptors(interceptors);

MultiValueMap<String, Object> parts = new LinkedMultiValueMap<String, Object>();
parts.add("file", new FileSystemResource("/sdcard/test/4MB_file"));

HttpEntity<MultiValueMap> requestEntity = new HttpEntity<MultiValueMap>(parts);
restTemplate.exchange(myUrl, HttpMethod.POST, requestEntity, Integer.class).getBody();

}

Looking at Spring Android source code, the next lines of code my request is passing through are:

public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
    HttpURLConnection connection = openConnection(uri.toURL(), this.proxy);
    prepareConnection(connection, httpMethod.name());
    if (this.bufferRequestBody) {
        return new SimpleBufferingClientHttpRequest(connection);
    } else {
        return new SimpleStreamingClientHttpRequest(connection, this.chunkSize);
    }
}

Because of this.bufferRequestBody is false, return new SimpleStreamingClientHttpRequest(connection, this.chunkSize); is executed (with chunkSize = 0)

SimpleStreamingClientHttpRequest(HttpURLConnection connection, int chunkSize) {
    this.connection = connection;
    this.chunkSize = chunkSize;

    // Bugs with reusing connections in Android versions older than Froyo (2.2)
    if (olderThanFroyo) {
        System.setProperty("http.keepAlive", "false");
    }
}

and then:

ClientHttpRequest delegate = requestFactory.createRequest(request.getURI(), request.getMethod());

delegate.getHeaders().putAll(request.getHeaders());

if (body.length > 0) {
    FileCopyUtils.copy(body, delegate.getBody());
}
return delegate.execute();

From here is all android subsystem I think..

I have dumped tcp traffic and analyzed it:

POST /urlWherePost HTTP/1.1
Content-Type: multipart/form-data;boundary=nKwsP85ZyyzSDuAqozCTuZOSxwF1jLAtd0FECUPF
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxx=
Accept-Encoding: gzip
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.1.2; sdk Build/MASTER)
Host: 192.168.168.225:8080
Connection: Keep-Alive
Content-Length: 4096225

--nKwsP85ZyyzSDuAqozCTuZOSxwF1jLAtd0FECUPF
Content-Disposition: form-data; name="file"; filename="4MB_file"
Content-Type: application/octet-stream
Content-Length: 4096000

I've tryed to re-create similar request with curl:

curl --verbose 
    -H "Connection: Keep-Alive" 
    -H "Content-Type: multipart/form-data" 
    -H "Accept-Encoding: gzip" 
    -H "Content-Disposition: form-data; name=\"file\"; filename=\"4MB_file\"" 
    -H "Content-Type: application/octet-stream" 
    --user xxx:xxx 
    -X POST 
    --form file=@4MB_file 
    http://192.168.168.225:8080/urlWherePost

but with curl the post is ok.

Posting json data is not a problem (maybe small body size). But when I try to send "big" files the time increase.

Looking in DDMS shell, on Network Statistics I've also found that the network throughput is never over 250kb in TX. There seems to be a bootleneck, but how to investigate it? Where I can look, which parameter can I change?

Thank you for any suggestion!

Answer 1

Have you tried using the MultipartEntity method? I had the same problem when downloading big amounts of JSON data from the server, but I switched to this method and caught all the data that the server provided me.

HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://myurl.com");

try {
    MultipartEntity entity = new MultipartEntity();

    entity.addPart("type", new StringBody("json"));
    entity.addPart("data", new JSONObject(data));
    httppost.setEntity(entity);
    HttpResponse response = httpclient.execute(httppost);
} catch (ClientProtocolException e) {
} catch (IOException e) {
} catch (JSONException e){
}