Truststore and Google Cloud Dataflow

Question

I need to use a trust store to make an SSL Kafka connection in Google Cloud Dataflow. Can I supply this from a bucket or is there a way to store this on the "local file system"?

Answer 1

You can use KafkaIO.Read.withConsumerFactoryFn to supply a factory function that will be invoked to create the Kafka consumer. In that function, you're free to do anything you like, e.g. you can download the trust store file from a GCS bucket (I would recommend using GcsUtil for that) and save it to a temporary file on local disk - AFAIK Kafka itself only supports having this file on local disk. Then manually create a KafkaConsumer and point it at the file.