I've a existing system where data is published to Pub/Sub topic, read by a cloud functions subscriber & pushed to Big Query to store it (No additional transformation done in subscriber CF).
Is it a good idea to change my subscriber CF to a Dataflow streaming job using pub/sub-BQ template? What are the pros/cons of using them?
All depends of your use case and your data rate.
You also have to take into account the easiness of deployment of a function (at the opposite of Dataflow where you have to drain your pipeline, stop it and relaunch it) and the capability to do much more (and over a longer period of time) with Dataflow (you are limited in processing capability with function, and the processing duration of each message can't go above 9 minutes).
According with your project perspective, one solution or the other can be better.
In bonus, I have a third alternative: Cloud Run. Cloud Run is almost as easy as function do update and deploy, the processing duration is a little bit longer (15 minutes per message) and you can process several message on the same instance, and thus, the pricing can be far more interesting than with function because of this factorization. If you are interested, have a look on this article that I wrote
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With