Michael Heil questions

Michael Heil answers

• Kafka - Setting high linger.ms and batch.size not helping
• Duplicate message consumption in Kafka due to auto-downscaling/deletion of pods
• How to add partition to Kafka topic and keep same-key message in same partition？
• Is Consumer Offset managed at Consumer group level or at the individual consumer inside that consumer group?
• Spark DataFrame orderBy and DataFrameWriter sortBy, is there a difference?
• How does spark structured streaming job handle stream - static DataFrame join?
• How is a Spark Dataframe partitioned by default?
• Read from Kafka topic process the data and write back to Kafka topic using scala and spark
• How to get kafka consumer lag for spark structured streaming application
• Does kafka distinguish between consumed offset and commited offset?