Why does RestTemplate consume excessive amounts of memory?

Question

Question

Why does Spring's RestTemplate use an excessive amount of heap (particularly the G1 Old Generation) when sending a file.

Context

We observed the RestTemplate to consume excessive amounts of memory when sending files via POST requests. We used Spring's WebClient as comparison and it behaves completely sane.

We created a demo project on github which contains the full code. The important parts are the following snippets:

private void sendFileAsOctetStream(File file) {
    final RequestEntity<FileSystemResource> request = RequestEntity.post(URI.create("http://localhost:8080/file"))
            .contentType(MediaType.APPLICATION_OCTET_STREAM)
            .body(new FileSystemResource(file));
    restTemplate.exchange(request, void.class);
}

and

private void sendFileAsOctetStream(File file) {
    webClient.post()
            .uri("/file")
            .body(BodyInserters.fromResource(new FileSystemResource(file)))
            .exchange()
            .block();
}

We observed the memory usage with jconsole when sending a 550MB file with both the implementations (left is WebClient, right is RestTemplate. The WebClient cosumes a couple of MegaBytes while the RestTemplate requires 2.7 GigaByte:

1. An initial manual GC to clean the old generation
2. The request
3. A manual GC (only for the RestTemplate)

Answer 1

This is due to the default RestTemplate which simply uses an unconfigured SimpleClientHttpRequestFactory for the creation of the requests.

The mentioned requst factory has a flag bufferRequestBody which by default is set to true, which leads to very high memory consumption when sending large requests.

From the javadoc of SimpleClientHttpRequestFactory#setBufferRequestBody():

Indicate whether this request factory should buffer the request body internally. Default is true. When sending large amounts of data via POST or PUT, it is recommended to change this property to false, so as not to run out of memory. This will result in a ClientHttpRequest that either streams directly to the underlying HttpURLConnection (if the Content-Length is known in advance), or that will use "Chunked transfer encoding" (if the Content-Length is not known in advance).

You can provide your own request factory, when creating the RestTemplate by using one of the other overloaded constructors, and setting mentioned flag to false on the request factory:

@Bean
public RestTemplate restTemplate() {
    SimpleClientHttpRequestFactory rf = new SimpleClientHttpRequestFactory();
    rf.setBufferRequestBody(false);
    return new RestTemplate(rf);
}