How do you setup multiple Spark Streaming jobs with different batch durations?

Question

We are in the beginning phases of transforming the current data architecture of a large enterprise and I am currently building a Spark Streaming ETL framework in which we would connect all of our sources to destinations (source/destinations could be Kafka topics, Flume, HDFS, etc.) through transformations. This would look something like:

SparkStreamingEtlManager.addEtl(Source, Transformation*, Destination) SparkStreamingEtlManager.streamEtl() streamingContext.start()

The assumptions is that, since we should only have one SparkContext, we would deploy all of the ETL pipelines in one application/jar.

The problem with this is that the batchDuration is an attribute of the context itself and not of the ReceiverInputDStream (Why is this?). Do we need to therefore have multiple Spark Clusters, or, allow for multiple SparkContexts and deploy multiple applications? Is there any other way to control the batch duration per receiver?

Please let me know if any of my assumptions are naive or need to be rephrased. Thanks!

Answer 1

In my experience, different streams have different tuning requirements. Throughput, latency, capacity of the receiving side, SLAs to be respected, etc.

To cater for that multiplicity, we require to configure each Spark Streaming job to address said specificity. So, not only batch interval but also resources like memory and cpu, data partitioning, # of executing nodes (when the loads are network bound).

It follows that each Spark Streaming job becomes a separate job deployment on a Spark Cluster. That will also allow for monitoring and management of separate pipelines independently of each other and help in the further fine-tuning of the processes.

In our case, we use Mesos + Marathon to manage our set of Spark Streaming jobs running 3600x24x7.