POST InputStream with RestTemplate

Question

I have a client that needs to POST a large number of large json files to a server. I've been able to get it working by reading each of the files into memory and posting the entire file with RestTemplate. However, the client quickly runs out of memory dealing with the large json files. I want to switch to a streaming approach but can't figure out how to use a FileInputStream with the RestTemplate properly. I found this question and used the code given in the accepted answer but I'm still seeing memory usage and OutOfMemory exceptions that lead me to believe that it is not streaming the files but still reading them into memory entirely. What am I doing wrong? Here is what I have currently:

final InputStream fis = ApplicationStore.class.getResourceAsStream(path);

final RequestCallback requestCallback = new RequestCallback() {
    @Override
    public void doWithRequest(final ClientHttpRequest request) throws IOException {
        request.getHeaders().add("Content-type", "application/json");
        IOUtils.copy(fis, request.getBody());
    }
};

final RestTemplate restTemplate = new RestTemplate();
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);     
restTemplate.setRequestFactory(requestFactory);     
final HttpMessageConverterExtractor<String> responseExtractor =
         new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());

restTemplate.execute("http://" + host + ":8080/upads-data-fabric" + "/ruleset", httpMethod, requestCallback, responseExtractor);

Answer 1

Don't. Use a Resource in combination with an appropriate RestTemplate#exchange method.

Create an HttpEntity with the Resource as the body. There's ClassPathResource to represent class path resources. The RestTemplate, by default, registers a ResourceHttpMessageConverter.

Internally, the ResourceHttpMessageConverter streams the request content to the opposite end of the connection with StreamUtils#copy(InputStream, OutputStream) with a buffer size that's currently set to 4096.

Answer 2

In addition to the @sotirios-delimanolis answer you also need to specify this setting to your RestTemplate so that internally your org.springframework.http.HttpOutputMessage is recognized as org.springframework.http.StreamingHttpOutputMessage because otherwise it just copies the entire stream to its internal stream so you just load it into memory. This way it uses chunks of your original stream and sends them.

HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);

I say that because there is only one implementation of StreamingHttpOutputMessage and HttpComponentsClientHttpRequestFactory is the only place where it is created.

Reproducible example:

MultiValueMap<String, Object> bodyMap = new LinkedMultiValueMap<>();
UrlResource urlResource = new UrlResource(MY_EXTERNAL_FILE_URL) { //uses URL#inputStream
    @Override
    public String getFilename() {
        return FILE_NAME;
    }
};
bodyMap.add("file", urlResource); //other service uses -- @RequestParam("file") MultipartFile -- in its controller
RequestEntity<MultiValueMap<String, Object>> request =
    RequestEntity.post(URI.create("http://localhost:6666/api/file"))
        .contentType(MediaType.MULTIPART_FORM_DATA)
        .body(bodyMap);

//should be a @Bean
RestTemplate restTemplate = new RestTemplate ();
HttpComponentsClientHttpRequestFactory requestFactory = new 
HttpComponentsClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);

System.out.println(restTemplate.exchange(request, FileMetadata.class));