Memory mapped equivalent for FirefoxOS

Question

How would you emulate a memory mapped file in FirefoxOS, Tizen or any other mobile pure-JS solution?

The use case is for a mobile browser and you need lots of data which does not fit in the RAM or you don't want to waste RAM for it yet and prefer to lazy load it.

The only thing I found is IndexedDB or what can I do about it? Any better tricks or APIs?

Hmmh it looks like Web SQL Database could be also a solution on Android, Tizen or iOS. But Firefox does not support it (?)

Update: I'm asking because of some experiments

Answer 1

First thing first, Web SQL won't be ever standardised as explained in the specification, so it should be considered only for WebKit/Blink-based browsers.

There is an awesome overview of offline storage options in this queston, even though the map tiles are considered in that question, I think it is still relevant to your use case.

I believe you are on the right track with IndexedDB for the graph data. On a high level it is a key-value asynchronous object store (see the Basic Concepts document). For your use case, you could index graph nodes in an object store. There is, for example, LevelGraph library which stores graph data in IndexedDB, though it is built for Semantic Web triples. HeliosJS is also worth mentioning, though it is an in-memory graph database.

Edit: Current API for IndexedDB is asynchronous. There is synchronous API drafted in the spec, which could be used only in web workers. Unfortunately no engine currently implements this feature. There is a pending patch for Gecko, but I did not find any plans for Blink or WebKit, so it is not a meaningful option now.

It is possible to access raw files through Web APIs. You could use XHR2 to load a (local) file as a binary Blob. Unfortunately XHR2 is mostly designed for streaming files and not for a random access, though you could split-up the data into multiple files and request them on demand, but that may be slow. The direct access to files is currently quite limited, FileList and createObjectURL are primarily used for direct file user input (through Drag and Drop or file input field), FileSystem API was recently killed, and the DeviceStorage is non-standard and privileged (Firefox OS-specific). You can also store files in IndexedDB, which is described for FileHandle API. However, once you manage to get access to the raw File object, you can use the Blob.slice method to load chunks of the file – there is a great example of reading file chunks via upload form. You may also want to look at jDataView library & friends, which eases handling of binary data through the more efficient ArrayBuffer.

Edit: As for the synchronous API, localStorage (aka DOM Storage) could be considered too. It is also a key-value storage, but much much simpler and more limited than IndexedDB:

• Size of the storage is limited, usually to 5 MB
• Only one localStorage per domain/application (you can have multiple named object stores in IndexedDB).
• Only strings can be stored.

In general, localStorage is useful cookies replacement, but it is not really useful for storing large offline data.

---

So to sum it up:

• IndexedDB is the easiest and widely available option, though it may be slow, inefficient or hit memory limits with very large data; also, only asynchronous API is currenlty possible.
• Raw file access is hard to obtain without user interaction and the APIs are unstable and non-standard.

In the end, you can combine both approaches, two options come in mind:

• Use XHR2 to parse the large file in chunks and store the parsed nodes into IndexedDB
• Store the large file into IndexedDB (via XHR), use FileHandle.getFile to load the File object and Blob.slice to read its content.

In all cases, you can (should) use Web Workers to handle data manipulation and calculations in the background.

Anyway, GraphHopper looks great, we really lack such non-trivial offline applications for Firefox OS, so good luck!