Speed up ListLinePlot

Question

I am trying to use Mathematica to analyse some raw data. I'd like to be able to dynamically display the range of data I'm interested in using Manipulate and ListLinePlot, but the plot rendering is extremely slow. How can I speed it up?

Here are some additional details. An external text file stores the raw data: the first column is a timestamp, the second, third and fourth columns are data readings, for example:

1309555993069, -2.369941, 6.129157, 6.823794
1309555993122, -2.260978, 6.170018, 7.014479
1309555993183, -2.070293, 6.129157, 6.823794
1309555993242, -1.988571, 6.238119, 7.123442

A single data file contains up to 2·10⁶ lines. To display, for example, the second column, I use:

x = Import["path/to/datafile"];
ListLinePlot[x[[All, {1, 2}]]]

The execution time of this operation is unbearably long. To display a variable range of data I tried to use Manipulate:

Manipulate[ListLinePlot[Take[x, numrows][[All, {1, 2}]]], {numrows, 1, Length[x]}]

This instruction works, but it quickly crawls when I try to display more than few thousand lines. How can I speed it up?

Some additional details:

• MATLAB displays the same amount of data on the same computer almost instantaneously, thus the raw data size shouldn't be an issue.
• I already tried to turn off graphics antialiasing, but it didn't impact rendering speed at all.
• Using DataRange to avoid Take doesn't help.
• Using MaxPlotPoints distorts too much the plot to be useful.
• Not using Take in Manipulate doesn't help.
• The rendering seems to take huge amount of time. Running Timing[ListLinePlot[Take[x,100000][[All, {1, 2}]]]] returns 0.33: this means that the evaluation of Take by itself is almost instantaneous, is the plot rendering that slows everything down.
• I am running Mathematica on Ubuntu Linux 11.10 using the fglrx drivers. Forcing Mathematica to use mesa drivers didn't help.

Any hint?

Answer 1

If your goal is to just visualize your data quickly but properly, you can use the following trick, which I am constantly using.

I partition the data into a number of blocks corresponding roughly to the resolution of my screen (usually 1000 or less), more detail cannot be displayed anyway. Then I determine the Min and Max of each block, and draw a zig-zag line from min to max to min to max... The result will look exactly like the original data. You can however not "zoom in", as you would then see the zig-zag line (e.g. when exporting to high-res pdf). Then you need to use a larger number of blocks.

rv = RandomVariate[ExponentialDistribution[2], 100000];

ListLinePlot[rv, PlotRange -> All] (* original, slow *)
ListLinePlot[rv, PlotRange -> All, MaxPlotPoints -> 1000] (* fast but distorted *)

numberOfBlocks = 1000;

ListLinePlot[Riffle @@ Through[{Min /@ # &, Max /@ # &}[
   Partition[rv,Floor[Length[rv]/numberOfBlocks]]]], PlotRange -> All]

You can add the DataRange->{...} option to label the x-axis appropriately.

Hope this helps!

EDIT: See also this similar question on Mathematica Stackexchange: https://mathematica.stackexchange.com/q/140/58