How to send and receive large JSON data

Question

I'm relatively new to full-stack development, and currently trying to figure out an effective way to send and fetch large data between my front-end (React) and back-end (Express) while minimizing memory usage. Specifically, I'm building a mapping app which requires me to play around with large JSON files (10-100mb).

My current setup works for smaller JSON files:

Backend:

const data = require('../data/data.json');

router.get('/', function(req, res, next) {
  res.json(data);
});

Frontend:

componentDidMount() {
      fetch('/')
      .then(res => res.json())
      .then(data => this.setState({data: data}));
  }

However, if data is bigger than ~40mb, the backend would crash if I test on local due to running out of memory. Also, holding onto the data with require() takes quite a bit of memory as well.

I've done some research and have a general understanding of JSON parsing, stringifying, streaming, and I think the answer lies somewhere with using chunked json stream to send the data bit by bit, but am pretty much at a loss on its implementation, especially using a single fetch() to do so (is this even possible?).

Definitely appreciate any suggestions on how to approach this.

Answer 1

First off, 40mb is huge and can be inconsiderate to your users especially if there' s a high probability of mobile use.

If possible, it would be best to collect this data on the backend, probably put it onto disk, and then provide only the necessary data to the frontend as it's needed. As the map needs more data, you would make further calls to the backend.

If this isn't possible, you could load this data with the client-side bundle. If the data doesn't update too frequently, you can even cache it on the frontend. This would at least prevent the user from needing to fetch it repeatedly.

Alternatively, you can read the JSON via a stream on the server and stream the data to the client and use something like JSONStream to parse the data on the client.

Here's an example of how to stream JSON from your server via sockets: how to stream JSON from your server via sockets