Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to update a large broadcast variable in a streaming use case?

Tags:

apache-spark

I have a use case where I have a streaming job running getting input data from kafka queue. And I have a reference data of 1 million rows which gets updated every hour. I load the reference data in the driver and then broadcast it to the workers. I would like to update this broadcast variable (in the driver) and resend it to workers.

What would be the best way to do this within spark, without introducing hbase/redis/cassandra etc?

And how reliable is this?

Do let me know if more information is needed. Thank you in advance. =)

like image 695
Subba Rao Avatar asked Oct 20 '22 01:10

Subba Rao


1 Answers

Answer to the similar question was given later here: How can I update a broadcast variable in spark streaming?

In short, you will need to: "unpersist" broadcast variable, update, and rebroadcast it.

PS. formally this question is not a duplicate, because it was posted earlier.

like image 108
Timofey Chernousov Avatar answered Nov 15 '22 08:11

Timofey Chernousov