Douglas M questions

Douglas M answers

• Last Access Time Update in Hive metastore
• PySpark unzip files: Which is a good approach for unzipping files and storing the csv files into a Delta Table?
• Spark schema management at single place
• What is the optimal way to read from multiple Kafka topics and write to different sinks using Spark Structured Streaming?
• Deserialization of large numpy arrays using pickle is order of magnitude slower than using numpy
• Spark dataframe to arrow
• Predicate pushdown on non partitioned parquet data