Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there anyway to share stateful variables in dataflow pipeline?


I'm making dataflow pipeline with python.
I want to share global variables across pipeline transform and across worker nodes like global variables (across multiple workers).
Is there any way to support this? thanx in advance

like image 883
Terry Cho Avatar asked Dec 12 '25 04:12

Terry Cho


1 Answers

Stateful processing may be of use for sharing state between workers of a specific node (would not be able to share between transforms though): https://beam.apache.org/blog/2017/02/13/stateful-processing.html

like image 103
Chris Staikos Avatar answered Dec 13 '25 23:12

Chris Staikos