Max recommended size of external JSON object in JavaScript

Question

I have a large amount of data to sort and query, and I can't rely on an internet connection. Ideally, I'd like to store my entire data-set as a JSON object (currently around 17MB, but could get much larger) and use something like jLinq or SQLite to query it, as opposed to having to output numerous smaller files.

I'm interested in finding what the largest recommended filesize is for an external getJSON call using JavaScript (jQuery, specifically). 1MB, 20MB, 100MB? Information on the subject is scarce. Information on querying large data-sets client-side is scarce all around.

Answer 1

Your biggest problem will probably be loading time since it will have to convert it from a string of JSON into an actual JavaScript object. The other big problem will be that the entire data set will be in memory for the page. I'm not familiar with any page using 100MB+ of data.

I made a jsfiddle to test loading performance of large JSON strings. It looks like it takes ~500ms just to parse a ~20MB string of JSON (on a Core i7 machine), and in Chrome it uses 80MB more memory than if the JSON string basically empty. So 100MB could take a few seconds to load and use 400MB+ of memory.

This won't solve either of those issues, but have you considered using SQL.js? It is a JavaScript implementation of SQLite. It should make querying that data much easier.