iPad - Parsing an extremely huge json - File (between 50 and 100 mb)

Question

I'm trying to parse an extremely big json-File on an iPad. The filesize will vary between 50 and 100 mb (there is an initial file and there will be one new full set of data every month, which will be downloaded, parsed and saved into coredata)

I'm building this app for a company as an Enterprise solution - the json file contains sensitive customerdata and it needs to be saved locally on the ipad so it will work even offline. It worked when the file was below 20mb, but now the set of data became bigger and I really need to parse it. I'm receiving memory warnings during parsing and after the third warning it just crashes. I have several different Core Data entities and I'm just setting all the values coming from the json file (when app is launched for the first time) and after everything is done, I'm doing the [context save].

I was hoping somebody could give me some advice on how to handle such huge files. I was thinking about splitting the json file up to several smaller json files and maybe parsing them in multiple threads, but I don't know if that's the right approach. I guess one big problem is that the whole file is being held in the memory - maybe there's some way of "streaming" it into the memory or something like that?

I'm using JSONKit (https://github.com/johnezang/JSONKit) for parsing the file, since I have read that it's the fastest one (maybe there's a slower one which goes easier on memory?).

Thanks in advance.

Answer 1

1) Write your data to a file, then use NSData's dataWithContentsOfFile:options:error: and specify the NSDataReadingMappedAlways and NSDataReadingUncached flags. This will tell the system to use mmap() to reduce the memory footprint, and not to burden the file system cache with blocks of memory (that makes it slower, but much less of a burden to iOS).

2) You can use the YAJL SAX style JSON parser to get objects as they decode.

Note: I have not done 2) but have used the techniques embodied in 1).

3) I ended up needed such a thing myself, and wrote SAX-JSON-Parser-ForStreamingData that can be tied to any asynchronous downloader (including my own).

Answer 2

Given the current memory constraints on a mobile device, it's likely impossible to parse 100 MB JSON text and then create a representation of a Foundation object which itself will take roughly 10 times the amount of RAM than the size of source JSON text.

That is, your JSON result would take about 1 GByte RAM in order to allocate the space required for the foundation objects.

So, there is likely no way to create one gigantic JSON representation - no matter how you get and read and parse the input. You need to split it into many smaller ones. This may require a modification on the server side, though.

Another solution is this, but much more elaborated:

Use a SAX style parser, which takes the huge JSON as input via a streaming API and outputs several smaller JSON texts (the inner parts). The SAX style parser may use a Blocks API (dispatch lib) to pass its results - the smaller JSONs asynchronously to another JSON parser. That is, the smaller JSONs are fed a usual JSON parser which produces the JSON representations, which in turn are fed your CoreData Model Generator.

You can even make it possible to download the huge JSON and parse it simultaneously with the SAX style parser, while simultaneously creating smaller JSONs and simultaneously storing them into Core Data.

What you need is a JSON parser with a SAX style API that can parse chunks of input text, performs fast, and can create a representation of Foundation objects.

I know only one JSON library which has this feature set, and there are even examples given which can partly show how you can accomplish exactly this: JPJson on GitHub. The parser is also very fast - on ARM it's faster than JSONKit. Caveat: its implementation is in C++ and requires a few steps to install it on a developer machine. It has a well documented Objective-C API, though.

Would like to add that I'm the author ;) An update is soon available, which utilizes latest C++11 compiler and C++11 library features resulting in even faster code (25% faster on ARM than JSONKit and twice as fast as NSJSONSerialization).

To give you same of the facts of the speed: The parser is able to download (over WiFi) and parse 25 MByte data containing 1000 JSONs (25 kByte each) in 7 seconds on Wifi 802.11g, and 4 seconds on Wifi 802.11n, including creating and releasing the 1000 representations on an iPad 2.