Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Dataflow pipeline and pubsub emulator

I'm trying to setup my development environment. Instead of using google cloud pubsub in production, I've been using the pubsub emulator for development and testing. To achieve this I set the following environment variable:

export PUBSUB_EMULATOR_HOST=localhost:8586

This worked for the python google pubsub library but when I switched to using java apache beam for google dataflow, the pipeline still points to production google pubsub. Is there a setting, environment variable or method on the pipeline that I need to set so that the pipeline reads for the local pubsub emulator?

like image 716
rezn8 Avatar asked Jan 05 '23 04:01

rezn8


1 Answers

I found the solution in the PubsubOptions interface, and extending it for my own PipelineOptions implementation. Then using setPubsubRootUrl() to set it to localhost:port of the emulator.

like image 184
rezn8 Avatar answered Jan 06 '23 17:01

rezn8