Window in Spark Streaming?

Question

In spark streaming, the DStreams we receive is a batch of RDDs. So how does windowing helps further.

As per my understanding it also batches the RDDs.

Correct me if I am wrong (new to Spark Streaming).

Answer 1

The number of records in one batch is determined by the batch interval. A window will keep the number of batches as fit within the size of a window, that's why the window size must be a multiple of the batch interval. Your operations will then run on multiple batches, and with each new batch the window will move forward, discarding older batches.

The point is that in streaming, data that belongs together doesn't necessarily arrive at the same time, especially at low batch intervals. With windows you are essentially looking back in time.

But note that your job still runs at the specified batch interval, so it will produce output at the same pace as before but look at more data at once. You will also look at the same data multiple times!

There a nice blog post by Michael Noll which explains this in more detail: http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/.

Update:

You can increase your batch interval, but then your job is processing slower as well, i.e. only creating output every 10 seconds instead of 2. You can also put a window on one part of the computation, whereas the batch interval affects everything. Check out reduceByKeyAndWindow for example.