Will writing to a Delta table from a Spark structured streaming job create a version for every micro batch of data written?
From the docs:
As you write into a Delta table or directory, every operation is automatically versioned.
So, yes you are correct.
Getting the data:
Using a timestamp
Using version number
Reference: https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With