Michael Heil questions

Michael Heil answers

• Usage of Broadcast Variables when using only Spark-SQL API
• What is the difference between Kafka partitions and Kafka replicas?
• Data still remains in Kafka topic even after retention time/size
• Kafka - Setting high linger.ms and batch.size not helping
• Duplicate message consumption in Kafka due to auto-downscaling/deletion of pods
• How to add partition to Kafka topic and keep same-key message in same partition？
• Is Consumer Offset managed at Consumer group level or at the individual consumer inside that consumer group?
• Spark DataFrame orderBy and DataFrameWriter sortBy, is there a difference?
• How does spark structured streaming job handle stream - static DataFrame join?
• How is a Spark Dataframe partitioned by default?