ProcessingJS performance with large data

Question

My goal is to create an interactive web visualization of data from motion tracking experiments.

The trajectories of the moving objects are rendered as points connected by lines. The visualization allows the user to pan and zoom the data.

My current prototype uses Processing.js because I am familiar with Processing, but I have run into performance problems when drawing data with greater than 10,000 vertices or lines. I pursued a couple of strategies for implementing the pan and zoom, but the current implementation, which I think is the best, is to save the data as an svg image and use the PShape data type in Processing.js to load, draw, scale and translate the data. A cleaned version of the code:

/* @pjs preload="nanoparticle_trajs.svg"; */
PShape trajs;

void setup() 
{
  size(900, 600);

  trajs = loadShape("nanoparticle_trajs.svg");

}

//function that repeats and draws elements to the canvas
void draw() 
{

  shape(trajs,centerX,centerY,imgW,imgH);

}

//...additional functions that get mouse events

Perhaps I should not expect snappy performance with so many data points, but are there general strategies for optimizing the display of complex svg elements with Processing.js? What would I do if I wanted to display 100,000 vertices and lines? Should I abandon Processing all together?

Thanks

EDIT:

Upon reading the following answer, I thought an image would help convey the essence of the visualization:

It is essentially a scatter plot with >10,000 points and connecting lines. The user can pan and zoom the data and the scale bar in the upper-left dynamically updates according to the current zoom level.

Answer 1

Here's my pitch:

Zoom level grouping and break down data as your users focuses/zooms-in

I would suggest you group together some data and present it as simple node

On zooming in to a particular node you can break down the node and release the group thus showing it's details.

This way you limit the amount of data you need to show on zoomed-out views (where all the nodes would be shown) and you add details as the user zooms-in to a region - in which case not all nodes would be showing since zooming in only focuses on one area of your graph

Viewport limit

Detect what is in the current view area and draw just that. Avoid drawing the whole node graph structure if your user cannot see it in his viewport - Show only what is necessary. Although I suspect that this is already done by Processing.js anyway, I don't know if your zooming functionality takes advantage of this.

Consider bitmap caching if your nodes are interactive/clickable

If your elements are clickable/interactive you might want to consider grouping data and showing them as bitmaps(large groups of data showed as a single image), until the user clicks on a bitmap in which case the bitmap is removed and the original shape is re-drawed in that bitmaps place. This minimizes the amount of points/lines the engine has to draw on each redraw cycle.

For bitmap caching see this link,(this is Fabric.js - a canvas library and SVG but the concept/idea is the same) and also this answer I posted to one of my questions for interactive vector/bitmap caching

---

As a side note:

Do you really need to use Processing?

If there's no interaction or animation happening and you just want to blit pixels(just draw it once) on a Canvas, consider abandoning a vector based library altogether. Plain-old canvas just blits pixels on a canvas and that's all. The initial startup drawing of data might have some delay, but since there's not any internal reference to the points/shapes/lines after they were drawn - there's nothing eating up your resources/clogging your memory.

So if this is the case - consider making the switch to plain Canvas. However data visualisations are all about animations and interactivity so I doubt you'll want to give them up.