Jackson multiple objects and huge json files

Question

I get the feeling that the answer might be a duplicate of this: Jackson - Json to POJO With Multiple Entries but I think that potentially the question is different enough. Also I'm using raw data binding rather than full data binding.

So like the asker of that question, I have multiple objects in a file and I'm trying to turn them into POJOs and stuff them into a database of my design so I can access the data quickly rather than slowly.

The files here are in the order of tens of GB, with up to millions of objects in each file. Anyway here is what I have so far:

ObjectMapper mapper = new ObjectMapper();
Map<String,Object> data = mapper.readValue(new File("foo.json"), Map.class);
System.out.println(data.get("bar"));

And this works great for printing the bar element of the first object in foo, but I need a way to iterate through every element in a way that won't eat up all my memory.

Thanks.

Answer 1

You don't have to choose between Streaming (JsonParser) and ObjectMapper, do both! Traverse a bit with parser, but then call JsonParser.readValueAs(MyType.class) to bind individual JSON Object.

Or, call ObjectMapper's readValue() method passing JsonParser at appropriate points. Or use ObjectMapper.reader(Type.class).readValues() and iterate that way.

Answer 2

Use this code sample to see the basic idea.

final InputStream in = new FileInputStream("json.json");
try {
  for (Iterator it = new ObjectMapper().readValues(
      new JsonFactory().createJsonParser(in), Map.class); it.hasNext();)
    System.out.println(it.next());
}
finally { in.close();} }