What's the most efficient way of converting a 10 MB JSON response into an NSDictionary?

Question

Our app must display a big chunk of data with minimal remote http requests, so we have added an endpoint to our backend that provides all the necessary data as a single json response. This results in ~1.5MB (compressed) or roughly 8 MBs of uncompressed json-formatted text.

Not much of a problem, it downloads in 10 - 30 seconds and we're using ASIHTTPRequest to write the whole response to disk.

Now comes the fun part - after reading the uncompressed file into a memory mapped string, we use stig's json-framework to convert it into an NSDictionary. This has worked very well for the rest of our app and the typical 2 KB json response for the rest of our API endpoints. However, deserializing these 8 MBs of data takes from a couple of seconds (simulator) to minutes (3G and 2nd gen iPod Touch).

I'm researching the best approach to read in all this data.

I would love to use binary plists served straight from the backend, but we are using Java and I haven't found a proper library that fits our requirements, and with such a tight deadline, writing our own might not be the best idea.

If it helps in any way, the json string we are parsing is mostly an array of X items, like so:

{
    "items": [ { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" },
               { "key1": "value1", "key2": "value2" }
             ]
}

What is the most efficient method to read in this 8 MB json formatted string into a NSDictionary in memory?

Answer 1

Take a look at JSONKit. It's very fast, and if your JSON has a lot of keys and values that repeat themselves (as is common), its "recently instantiated object cache" will dramatically cut down on the amount of memory used for the "final" NSDictionary.

Answer 2

While JSONKit is faster than YAJL, YAJL supports parsing of the stream. You should be able to wire it up with ASIHTTPRequest to parse the JSON while it's downloading it. Check out the section labeled "Document style parsing as data becomes available" here:

http://gabriel.github.com/yajl-objc/