What is the most efficient way in JavaScript to parse huge amounts of data from a file

Question

What is the most efficient way in JavaScript to parse huge amounts of data from a file?

Currently I use JSON parse to serialize an uncompressed 250MB file, which is really slow. Is there a simple and fast way to read a lot of data in JavaScript from a file without looping through every character? The data stored in the file are only a few floating point arrays?

UPDATE: The file contains a 3d mesh, 6 buffers (vert, uv etc). Also the buffers need to be presented as typed arrays. streaming is not a option because the file has to be fully loaded before the graphics engine can continue. Maybe a better question is how to transfer huge typed arrays from a file to javascript in the most efficient way.

Answer 1

I would recommend a SAX based parser for these kind of JavaScript or a stream parser.

DOM parsing would load the whole thing in memory and this is not the way to go by for large files like you mentioned.

For Javascript based SAX Parsing (in XML) you might refer to https://code.google.com/p/jssaxparser/

and

for JSON you might write your own, the following link demonstrates how to write a basic SAX based parser in Javascript http://ajaxian.com/archives/javascript-sax-based-parser

Answer 2

Have you tried encoding it to a binary and transferring it as a blob?

https://developer.mozilla.org/en-US/docs/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data

http://www.htmlgoodies.com/html5/tutorials/working-with-binary-files-using-the-javascript-filereader-.html#fbid=LLhCrL0KEb6

Answer 3

There isn't a really good way of doing that, because the whole file is going to be loaded into memory and we all know that all of them have big memory leaks. Can you not instead add some paging for viewing the contents of that file?

Check if there are any plugins that allow you to read the file as a stream, that will improve this greatly.

UPDATE

http://www.html5rocks.com/en/tutorials/file/dndfiles/

You might want to read about the new HTML5 API's to read local files. You will have the issue with downloading 250mb of data still tho.