DC and crossfilter with large datasets

Question

I have been working on dc and crossfilter js and I currently have a large dataset with 550,000 rows and size 60mb csv and am facing a lot of issues with it like browser crashes etc

So , I'm trying to understand how dc and crossfilter deals with large datasets. http://dc-js.github.io/dc.js/

The example on their main site runs very smoothly and after seeing timelines->memory (in console) it goes to a max of 34 mb and slowly reduces with time

My project is taking up memory in the range of 300-500mb per dropdown selection, when it loads a json file and renders the entire visualization

So, 2 questions

• What is the backend for the dc site example? Is it possible to find out the exact backend file?
• How can I reduce the data overload on my RAM from my application, which is running very slowly and eventually crashing?

Answer 1

Hi you can try running loading the data, and filtering it on the server. I faced a similar problem when the size of my dataset was being too big for the browser to handle. I posted a question a few weeks back as to implementing the same. Using dc.js on the clientside with crossfilter on the server

Here is an overview of going about it.

On the client side, you'd want to create fake dimensions and fake groups that have basic functionality that dc.js expects(https://github.com/dc-js/dc.js/wiki/FAQ#filter-the-data-before-its-charted). You create your dc.js charts on the client side and plug in the fake dimensions and groups wherever required.

Now on the server side you have crossfilter running(https://www.npmjs.org/package/crossfilter). You create your actual dimensions and groups here.

The fakedimensions have a .filter() function that basically sends an ajax request to the server to perform the actual filtering. The filtering information could be encoded in the form of a query string. You'd also need a .all() function on your fake group to return the results of the filtering.