How to get large JSON Using REST template Spring MVC without memory issues in java

Question

When i get large JSON from REST service i receive java.lang.OutOfMemoryError: Java heap space...

This is occurs after ~1000 calls the REST service

    HttpEntity<ProductRequest> requestUpdate = new HttpEntity<>(request, httpHeaders);

    ResponseEntity<String> messageEntity = restTemplate.exchange(getEndPointAddress(), HttpMethod.POST, requestUpdate, String.class);

    Map<String, Object> map = gson.fromJson(Helper.cleanJson(messageEntity.getBody()), Map.class);

<pre>
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[?:1.7.0_181]
    at java.lang.String.<init>(String.java:203) ~[?:1.7.0_181]
    at java.lang.StringBuilder.toString(StringBuilder.java:405) ~[?:1.7.0_181]
    at org.springframework.util.StreamUtils.copyToString(StreamUtils.java:80) ~[stormjar.jar:4.0.3]
    at org.springframework.http.converter.StringHttpMessageConverter.readInternal(StringHttpMessageConverter.java:86) ~[stonjar.jar:4.0.3]
    at org.springframework.http.converter.StringHttpMessageConverter.readInternal(StringHttpMessageConverter.java:41) ~[stonjar.jar:4.0.3]
    at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:193) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:104) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:884) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:868) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:622) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) ~[stonjar.jar:4.0.3]
    at com.myorg.wcf.PMiddle.retrieveP(PMiddle.java:95) ~[ssar.jar:4.0.3]
    at com.myorg.RetrieveP.getPJsonFromService(RetrieveP.java:188) ~[ssar.jar:4.0.3]
    at com.myorg.RetrieveP.execute(RetrieveP.java:107) ~[ssar.jar:4.0.3]
    at org.apache.storm.daemon.executor$fn__5044$tuple_action_fn__5046.invoke(executor.clj:727) ~[storm-core-1.1.0.jar:1.1.0]
    at org.apache.storm.daemon.executor$mk_task_receiver$fn__4965.invoke(executor.clj:459) ~[storm-core-1.1.0.jar:1.1.0]
    at org.apache.storm.disruptor$clojure_handler$reify__4480.onEvent(disruptor.clj:40) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.daemon.executor$fn__5044$fn__5057$fn__5110.invoke(executor.clj:846) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) [ss-core-1.1.0.jar:1.1.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.7.0_181]</pre>

Answer 1

RestTemplate and underlying Spring beans are not designed for stream processing, they build the entire ResponsEntity in the memory. Based on your example you are not using RestTemplate as intended. You are reading String instead of delegating response object Map<String, Object> deserialization to underlying converter bean.

If the JSON response is really too big to fit into your heap you can replace RestTemplate with URLConnection or any other HTTP client library that will let you access the response InputStream directly. This will allow you to manually read and parse the response e.g. by using com.google.gson.stream.JsonReader:

URLConnection conn = // create and open
JsonReader reader = new JsonReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
// read manually using reader

You will be able to save memory that way by avoiding the one big String object that holds the entire JSON body.