ImDarrenG questions

ImDarrenG answers

• Spark & Scala: Generate DataSet (or Dataframe) with given size
• Airflow: When to use CeleryExecutor and when to use MesosExecutor
• In Spark Streaming, is there a way to detect when a batch has finished?
• How to create bins on a column in a data frame
• Apache Spark reads for S3: can't pickle thread.lock objects
• Spark DStream periodically call saveAsObjectFile using transform does not work as expected
• Difference between QuantileDiscretizer and Bucketizer in Spark
• java.lang.RuntimeException: java.lang.String is not a valid external type for schema of bigint or int
• Airflow - Python file NOT in the same DAG folder