Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

d3 — Progressively draw a large dataset

I'm using d3.js to plot the contents of an 80,000 row .tsv onto a chart.

The problem I'm having is that since there is so much data, the page becomes unresponsive for aprox 5 seconds while the entire dataset is churned through at once.

Is there an easy way to process the data progressively if it's spread over a longer period of time? Ideally the page would remain responsive, and the data would be plotted as it became available, instead of in one big hit at the end

like image 488
swanny Avatar asked Oct 06 '13 08:10

swanny


People also ask

How much data can D3 handle?

D3 charts are most often rendered using SVG, a retained mode graphics model, which is easy to use, but performance is limited. SVG charts can typically handle around 1,000 datapoints.

Is d3js slow?

D3. js really slow for data as much as 50,000 rows and even slow for the force directed graphs for data of 2000 rows.

Is D3 good for data visualization?

D3 is often preferred over other data visualization tools as it is a very flexible tool that can provide dynamic properties to most of its functions. With D3, there are no standard visualization formats.


1 Answers

I think you'll have to chunk your data and display it in groups using setInterval or setTimeout. This will give the UI some breathing room to jump in the middle.

The basic approach is: 1) chunk the data set 2) render each chunk separately 3) keep track of each rendered group

Here's an example:

var dataPool = chunkArray(data,100);
function updateVisualization() {
    group = canvas.append("g").selectAll("circle")
    .data(dataPool[poolPosition])
    .enter()
    .append("circle")
    /* ... presentation stuff .... */

}

iterator = setInterval(updateVisualization,100);

You can see a demo fiddle of this -- done before I had coffee -- here:

http://jsfiddle.net/thudfactor/R42uQ/

Note that I'm making a new group, with its own data join, for each array chunk. If you keep adding to the same data join over time ( data(oldData.concat(nextChunk) ), the entire data set still gets processed and compared even if you're only using the enter() selection, so it doesn't take long for things to start crawling.

like image 155
John Williams Avatar answered Sep 24 '22 12:09

John Williams