Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Delta table versioning while writing from a Spark structured streaming job

Will writing to a Delta table from a Spark structured streaming job create a version for every micro batch of data written?

like image 363
Somasundaram Sekar Avatar asked Nov 19 '25 23:11

Somasundaram Sekar


1 Answers

From the docs:

As you write into a Delta table or directory, every operation is automatically versioned.

So, yes you are correct.

Getting the data:

  • Using a timestamp

  • Using version number

Reference: https://databricks.com/blog/2019/02/04/introducing-delta-time-travel-for-large-scale-data-lakes.html

like image 107
thebluephantom Avatar answered Nov 22 '25 00:11

thebluephantom



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!