d3.js time series infinite scroll

Question

I am working on a time series line chart that lets the user scroll back from the present. I can find tutorials on real-time d3.js charts, I can find tutorials on zooming and panning, and I can find tutorials on using external data sources. I'm having trouble putting all this knowledge together.

Here is the behavior that I am looking for:

• The chart can pan backward in time (meaning that the lines, data points, and axes move with dragging of the mouse or finger)
• Panning should only effect the x-axis, and no zooming should occur.
• As the user pans the chart, more data loads in, giving an experience of infinite scrolling
• I plan on buffering in at least one extra "page" worth of data for the user to scroll into (already got this part figured out)
• I don't think I need transitions, because the panning of the chart will already smoothly translate it

This is what I have working so far:

  // set up a zoom handler only for panning
  // by limiting the scaleExtent    
  var zoom = d3.behavior.zoom()
  .x(x)
  .y(y)
  .scaleExtent([1, 1])
  .on("zoom", pan);

  var loadedPage = 1; // begin with one page of data loaded
  var nextPage = 2; // next page will be page 2
  var panX = 0;

  function pan() 
  {
     if (d3.event) 
     {
        panX = d3.event ? d3.event.translate[0] : 0;

        // is there a better way to determine when
        // to load the next page?
        nextPage = panX / (width + margin.left + margin.right) + 2;
        nextPage = Math.floor(nextPage);

        // if we haven't loaded in the next page's data
        // load it in so that the user can scroll into it
        if (nextPage > loadedPage) {

          console.log("Load a new page");
          loadedPage += 1;

          // load more data
          Chart.query( /*params will be here*/ ).then(
            function(response) {

              // append the new data onto the front of the array
              data = data.concat(response);
              console.log(data.length);

              // I need to add the new data into the line chart
              // but how do I make that work with the pan
              // logic from zoom?

         }
       );
     }
        // is this where I update the axes and scroll the chart?
        // What's the best way to do that?

      }
    }

In this code, I can know when to pull more data from the server, but I'm not sure how to insert the data into the chart in a way that works with the pan offset. Do I use transform translate, or can I update the d value of the path of my line?

Any suggestions would be welcome... also, if anyone knows of any demos which already show panning infinitely through time series data, that would be much appreciated.

Answer 1

As mentioned in the other answer, I know this is a very old post but hopefully the following will help someone...

I made a pen that I think hits all the requirements mentioned. As I didn't have a real API to use, I created some data using a json-generator (great tool), included it, and sorted it in descending order. Then I use the built in slice and concat methods to take bits of the array, data, and add to the chart_data variable (similarly to how one might use an api).

Important Sections:

Once you've created your scales, axes, and points (lines, bars, etc.), you need to create the zoom behavior. As mentioned in the question, keeping the scaleExtent limited to the same number on both sides prevents zooming:

var pan = d3.behavior.zoom()     .x(x_scale)     .scale(scale)     .size([width, height])     .scaleExtent([scale, scale])     .on('zoom', function(e) { ... }); 

Now that we've created the behavior, we need to call it. I'm also calculating what the x translation will be for this moment in time, now, and programmatically panning there:

// Apply the behavior viz.call(pan);  // Now that we've scaled in, find the farthest point that // we'll allow users to pan forward in time (to the right) max_translate_x = width - x_scale(new Date(now)); viz.call(pan.translate([max_translate_x, 0]).event); 

Both preventing the user from scrolling past now and loading more data is all done in the zoom event handler:

... .scaleExtent([scale, scale]) .on('zoom', function(e) {      var current_domain = x_scale.domain(),          current_max = current_domain[1].getTime();       // If we go past the max (i.e. now), reset translate to the max      if (current_max > now)         pan.translate([max_translate_x, 0]);       // Update the data & points once user hits the point where current data ends      if (pan.translate()[0] > min_translate_x) {         updateData();         addNewPoints();      }       // Redraw any components defined by the x axis      x_axis.call(x_axis_generator);      circles.attr('cx', function(d) {          return x_scale(new Date(d.registered));      }); }); 

The other functions are pretty straightforward and can be found at the bottom of the pen. I'm not aware of any built in D3 function to prevent panning past the present but I'm definitely open to feedback if I've missed an easier way to do some of this.

Let me know if you have trouble viewing the pen or need clarification on something. If I have time I'll update this with another version demoing an infinite scrolling line chart.

P.S. In the pen, I'm consoling out the selection and data as they update. I suggest opening the console to see exactly what's happening.

Answer 2

This is too late, but answering just in case somebody needs again. I was having most of the code ready for my scatterplot so uploading that. Hope it helps you. The code is created as a trial when I was learning this features. So please check before you use.

Note: D3js panning implemented with zoom behavior, zooming disabled with scaleExtent, Y panning restricted. Data loaded when x extremes are reached. Please check the Plunkr link

// Code goes here

window.chartBuilder = {};
(function(ns) {

  function getMargin() {
    var margin = {
      top: 20,
      right: 15,
      bottom: 60,
      left: 60
    };
    var width = 960 - margin.left - margin.right;
    var height = 500 - margin.top - margin.bottom;
    return {
      margin: margin,
      width: width,
      height: height
    };
  }

  function getData() {
    var data = [
      [5, 3],
      [10, 17],
      [15, 4],
      [2, 8]
    ];
    return data;
  }

  //function defineScales(data, width, height) {
  //    var x = d3.scale.linear()
  //        .domain([0, d3.max(data, function (d) {
  //            return d[0];
  //        })])
  //        .range([0, width]);
  //
  //    var y = d3.scale.linear()
  //        .domain([0, d3.max(data, function (d) {
  //            return d[1];
  //        })])
  //        .range([height, 0]);
  //    return {x: x, y: y};
  //}
  function defineYScale(data, domain, range) {
    var domainArr = domain;
    if (!domain || domain.length == 0) {
      domainArr = [0, d3.max(data, function(d) {
        return d[1];
      })];
    }
    var y = d3.scale.linear()
      .domain(domainArr)
      .range(range);

    return y;
  }

  function defineXScale(data, domain, range) {
    var domainArr = domain;
    if (!domain || domain.length == 0) {
      domainArr = [d3.min(data, function(d) {
        return d[0];
      }), d3.max(data, function(d) {
        return d[0];
      })];
    }

    var x = d3.scale.linear()
      .domain(domainArr)
      .range(range);
    return x;
  }

  function getSvg(width, margin, height) {
    var chart = d3.select('body')
      .append('svg:svg')
      .attr('width', width + margin.right + margin.left)
      .attr('height', height + margin.top + margin.bottom)
      .attr('class', 'chart');
    return chart;
  }

  function getContainerGroup(chart, margin, width, height) {
    var main = chart.append('g')
      .attr('transform', 'translate(' + margin.left + ',' + margin.top + ')')
      .attr('width', width)
      .attr('height', height)
      .attr('class', 'main');
    return main;
  }

  function renderXAxis(x, main, height) {
    var xAxis = d3.svg.axis()
      .scale(x)

    .orient('bottom');
    var xAxisElement = main.select('.x.axis');
    if (xAxisElement.empty()) {
      xAxisElement = main.append('g')
        .attr('transform', 'translate(0,' + height + ')')
        .attr('class', 'x axis')
    }
    xAxisElement.call(xAxis);

    return xAxis;
  }

  function renderYAxis(y, main) {
    var yAxis = d3.svg.axis()
      .scale(y)
      .orient('left');
    var yAxisElement = main.select('.y.axis');
    if (yAxisElement.empty()) {

      yAxisElement = main.append('g')
        .attr('transform', 'translate(0,0)')
        .attr('class', 'y axis');
    }
    yAxisElement.call(yAxis);
    return yAxis;
  }

  function renderScatterplot(main, data, scales) {
    var g = main.append("svg:g");
    var divTooltip = d3.select('.tooltip1');
    if (divTooltip.empty()) {
      divTooltip = d3.select('body').append('div')
        .attr('class', 'tooltip1')
        .style('opacity', 0);
    }

    g.selectAll("scatter-dots")
      .data(data, function(d, i) {
        return i;
      })
      .enter().append("svg:circle")
      .attr("cx", function(d, i) {
        return scales.x(d[0]);
      })
      .attr("cy", function(d) {
        return scales.y(d[1]);
      })
      .on('click', function(d) {

        // log(d.toString());


      })

    .attr("r", 8);
  }

  function addZoomRect(main, scales, zoom) {
    var zoomRect = main.append('rect')
      .attr('width', function() {
        return scales.x(d3.max(scales.x.domain()));
      })
      .attr('height', function() {
        return scales.y(d3.min(scales.y.domain()));
      })
      .attr('x', 0)
      .attr('y', 0)
      .attr('fill', 'transparent')
      .attr('stroke', 'red');
    if (zoom) {
      zoomRect.call(zoom);
    }
    return zoomRect;
  }

  function restrictYPanning(zoom) {
    var zoomTranslate = this.translate();
    this.translate([zoomTranslate[0], 0]);
  }

  function addXScrollEndEvent(scales, direction, data) {
    var zoomTranslate = this.translate();
    var condition;
    var currentDomainMax = d3.max(scales.x.domain());
    var dataMax = d3.max(data, function(d) {
      return d[0];
    });
    var currentDomainMin = d3.min(scales.x.domain());
    var dataMin =
      d3.min(data, function(d) {
        return d[0];
      });
    if (currentDomainMax > dataMax && direction === 'right') {
      //log('currentDomainMax ', currentDomainMax);
      //log('dataMax ', dataMax);
      //log('----------------');
      condition = true;
    }

    if (dataMin > currentDomainMin && direction === 'left') {
      //log('currentDomainMin ', currentDomainMin);
      //log('dataMin ', dataMin);
      //log('----------------');
      condition = true;
    }
    //var xRightLimit, xTranslate;
    //if (direction === 'right') {
    //    xRightLimit = scales.x(d3.max(scales.x.domain())) - (getMargin().width + 60);
    //
    //    xTranslate = 0 - zoomTranslate[0];// + scales.x(d3.min(scales.x.domain()));
    //
    //    condition = xTranslate > xRightLimit;
    //} else {
    //    xRightLimit = scales.x(d3.min(scales.x.domain()));
    //
    //    xTranslate = zoomTranslate[0];// + scales.x(d3.min(scales.x.domain()));
    //
    //    condition = xTranslate > xRightLimit;
    //}
    return condition;
  }

  function onZoom(zoom, main, xAxis, yAxis, scales, data) {
    //var xAxis = d3.svg.axis()
    //    .scale(scales.x)
    //    .orient('bottom');
    //var yAxis = d3.svg.axis()
    //    .scale(scales.y)
    //    .orient('left');
    //alert(data);
    var translate = zoom.translate();
    var direction = '';

    if (translate[0] < ns.lastTranslate[0]) {
      direction = 'right';
    } else {
      direction = 'left';
    }
    ns.lastTranslate = translate; //d3.transform(main.attr('transform')).translate  ;
    // log('zoom translate', ns.lastTranslate);
    // log('d3 Event translate', d3.event.translate);
    window.scales = scales;
    window.data = data;


    // ns.lastTranslate = translate;

    var divTooltip = d3.select('.tooltip1');
    if (divTooltip.empty()) {
      divTooltip = d3.select('body').append('div')
        .attr('class', 'tooltip1')
        .style('opacity', 0);
    }


    restrictYPanning.call(zoom);
    var xScrollEndCondition = addXScrollEndEvent.call(zoom, scales, direction, data);
    if (xScrollEndCondition) {
      if (zoom.onXScrollEnd) {

        zoom.onXScrollEnd.call(this, {
          'translate': translate,
          'direction': direction

        });
      }
    }


    main.select(".x.axis").call(xAxis);
    main.select(".y.axis").call(yAxis);
    var dataElements = main.selectAll("circle")
      .data(data, function(d, i) {
        return i;
      });

    dataElements.attr("cx", function(d, i) {
        return scales.x(d[0]);
      })
      .attr("cy", function(d) {
        return scales.y(d[1]);
      }).attr("r", 8);

    dataElements.enter().append("svg:circle")
      .attr("cx", function(d, i) {
        return scales.x(d[0]);
      })
      .attr("cy", function(d) {
        return scales.y(d[1]);
      }).on('click', function(d) {

        // log(d.toString());


      })

    .attr("r", 8);
    // log(direction);



  }

  //var xRangeMax;
  //var xRangeMin;
  ns.lastTranslate = [0, 0];

  /**
   * Created by Lenovo on 7/4/2015.
   */
  function log(titlee, msgg) {
    var msg = msgg;

    var title;
    if (titlee) {
      title = titlee + ':-->';
    }

    if (!msgg) {
      msg = titlee;
      title = '';
    } else {
      if (Array.isArray(msgg)) {
        msg = msgg.toString();
      }
      if ((typeof msg === "object") && (msg !== null)) {
        msg = JSON.stringify(msg);
      }
    }

    var tooltip = d3.select('.tooltip1');
    var earlierMsg = tooltip.html();
    var num = tooltip.attr('data-serial') || 0;
    num = parseInt(num) + 1;

    msg = '<div style="border-bottom:solid 1px green"><span style="color:white">' + num + ')</span><strong>' + title + '</strong> ' + decodeURIComponent(msg) + ' </div>';
    tooltip.html('<br>' + msg + '<br>' + earlierMsg).style({
        'color': 'lightGray',
        'background': 'darkGray',
        'font-family': 'courier',
        'opacity': 1,
        'max-height': '200px',
        'overflow': 'auto'
      })
      .attr('data-serial', num);
  }

  function addLoggerDiv() {
    var divTooltip = d3.select('.tooltip1');
    if (divTooltip.empty()) {
      divTooltip = d3.select('body').append('div')
        .attr('class', 'tooltip1')
        .style({
          'opacity': 0,
          'position': 'relative'
        });

      d3.select('body').append('div')
        .text('close')
        .style({
          'top': 0,
          'right': 0,
          'position': 'absolute',
          'background': 'red',
          'color': 'white',
          'cursor': 'pointer'
        })
        .on('click', function() {
          var thisItem = divTooltip;
          var txt = thisItem.text();
          var display = 'none';
          if (txt === 'close') {
            thisItem.text('open');
            display = 'none';
          } else {
            thisItem.text('close');
            display = 'block';
          }
          devTooltip.style('display', display);

        });

      d3.select('body').append('div')
        .text('clear')
        .style({
          'top': 0,
          'right': 20,
          'position': 'absolute',
          'background': 'red',
          'color': 'white',
          'cursor': 'pointer'
        })
        .on('click', function() {
          divTooltip.html('');
          divTooltip.attr('data-serial', '0');
        });
    }
  }



  $(document).ready(function() {
    var data = getData();
    var __ret = getMargin();
    var margin = __ret.margin;
    var width = __ret.width;
    var height = __ret.height;
    var scales = {};
    var xRangeMax = width;
    scales.x = defineXScale(data, [], [0, xRangeMax]);
    scales.y = defineYScale(data, [], [height, 0]);
    addLoggerDiv();
    var svg = getSvg(width, margin, height);
    var main = getContainerGroup(svg, margin, width, height);
    // draw the x axis
    var xAxis = renderXAxis(scales.x, main, height);
    // draw the y axis
    var yAxis = renderYAxis(scales.y, main);

    var thisobj = this;
    var zoom = d3.behavior.zoom().x(scales.x).y(scales.y).scaleExtent([1, 1]).on('zoom', function() {
      onZoom.call(null, zoom, main, xAxis, yAxis, scales, data);
    });
    zoom.onXScrollEnd = function(e) {
      var maxX = d3.max(data, function(d) {
        return d[0];
      });
      var minX = d3.min(data, function(d) {
        return d[0];
      });
      var incrementX = Math.floor((Math.random() * 3) + 1);
      var maxY = d3.max(data, function(d) {
        return d[1];
      })
      var minY = d3.min(data, function(d) {
        return d[1];
      })
      var incrementY = Math.floor((Math.random() * 1) + 16);
      var xRangeMin1, xRangeMax1, dataPoint;
      if (e.direction === 'left') {
        incrementX = incrementX * -1;
        dataPoint = minX + incrementX;
        // log('dataPoint ', dataPoint);

        //xRangeMin1 = d3.min(scales.x.range()) - Math.abs(scales.x(minX) - scales.x(dataPoint));
        xRangeMin1 = scales.x(dataPoint);
        xRangeMax1 = d3.max(scales.x.range());
      } else {
        dataPoint = maxX + incrementX;
        // log('dataPoint ', dataPoint);

        //xRangeMax1 = d3.max(scales.x.range()) + (scales.x(dataPoint) - scales.x(maxX));
        xRangeMax1 = d3.max(scales.x.range()) + 20; //scales.x(dataPoint);
        xRangeMin1 = d3.min(scales.x.range()) //e.translate[0];

      }
      data.push([dataPoint, incrementY]);

      //scales = defineScales(data, width + incrementX, height );
      //             scales.x = defineXScale(data, [], [xRangeMin1, xRangeMax1]);
      //             scales.y = defineYScale(data, [], [height, 0]);

      scales.x.domain(d3.extent(data, function(d) {
        return d[0];
      }));
      x = scales.x;
      y = scales.y;
      xAxis = renderXAxis(scales.x, main, height);
      // draw the y axis
      yAxis = renderYAxis(scales.y, main);
      zoom.x(scales.x).y(scales.y);


    }
    var zoomRect = addZoomRect(main, scales, zoom);


    renderScatterplot(main, data, scales);

  });
})(window.chartBuilder);

/* Styles go here */

.chart {
    font-family: Arial, sans-serif;
    font-size: 10px;
}

.axis path, .axis line {
    fill: none;
    stroke: #000;
    shape-rendering: crispEdges;
}

.bar {
    fill: steelblue;
}

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.4.11/d3.min.js"></script>

I have created zoom.onXScrollEnd function to add new points to data.

Hope it helps.