Is there an event-driven JSON REST client API for Java?

Question

I have a Java application which uses Spring's RestTemplate API to write concise, readable consumers of JSON REST services:

In essence:

 RestTemplate rest = new RestTemplate(clientHttpRequestFactory);  ResponseEntity<ItemList> response = rest.exchange(url,             HttpMethod.GET,                  requestEntity,             ItemList.class);   for(Item item : response.getBody().getItems()) {         handler.onItem(item);  } 

The JSON response contains a list of items, and as you can see, I have have an event-driven design in my own code to handle each item in turn. However, the entire list is in memory as part of response, which RestTemplate.exchange() produces.

I would like the application to be able to handle responses containing large numbers of items - say 50,000, and in this case there are two issues with the implementation as it stands:

1. Not a single item is handled until the entire HTTP response has been transferred - adding unwanted latency.
2. The huge response object sits in memory and can't be GC'd until the last item has been handled.

Is there a reasonably mature Java JSON/REST client API out there that consumes responses in an event-driven manner?

I imagine it would let you do something like:

 RestStreamer rest = new RestStreamer(clientHttpRequestFactory);   // Tell the RestStreamer "when, while parsing a response, you encounter a JSON  // element matching JSONPath "$.items[*]" pass it to "handler" for processing.  rest.onJsonPath("$.items[*]").handle(handler);   // Tell the RestStreamer to make an HTTP request, parse it as a stream.  // We expect "handler" to get passed an object each time the parser encounters  // an item.  rest.execute(url, HttpMethod.GET, requestEntity); 

I appreciate I could roll my own implementation of this behaviour with streaming JSON APIs from Jackson, GSON etc. -- but I'd love to be told there was something out there that does it reliably with a concise, expressive API, integrated with the HTTP aspect.

Answer 1

A couple of months later; back to answer my own question.

I didn't find an expressive API to do what I want, but I was able to achieve the desired behaviour by getting the HTTP body as a stream, and consuming it with a Jackson JsonParser:

  ClientHttpRequest request =          clientHttpRequestFactory.createRequest(uri, HttpMethod.GET);   ClientHttpResponse response = request.execute();    return handleJsonStream(response.getBody(), handler); 

... with handleJsonStream designed to handle JSON that looks like this:

 { items: [        { field: value; ... },        { field: value, ... },       ... thousands more ...   ] } 

... it validates the tokens leading up to the start of the array; it creates an Item object each time it encounters an array element, and gives it to the handler.

 // important that the JsonFactory comes from an ObjectMapper, or it won't be  // able to do readValueAs()  static JsonFactory jsonFactory = new ObjectMapper().getFactory();   public static int handleJsonStream(InputStream stream, ItemHandler handler) throws IOException {       JsonParser parser = jsonFactory.createJsonParser(stream);       verify(parser.nextToken(), START_OBJECT, parser);      verify(parser.nextToken(), FIELD_NAME, parser);      verify(parser.getCurrentName(), "items", parser);      verify(parser.nextToken(), START_ARRAY, parser);      int count = 0;      while(parser.nextToken() != END_ARRAY) {         verify(parser.getCurrentToken(), START_OBJECT, parser);         Item item = parser.readValueAs(Item.class);         handler.onItem(item);         count++;      }      parser.close(); // hope it's OK to ignore remaining closing tokens.      return count;  } 

verify() is just a private static method which throws an exception if the first two arguments aren't equal.

The key thing about this method is that no matter how many items there are in the stream, this method only every has a reference to one Item.