Staging files on google dataflow worker

Question

Is there anything in the Dataflow SDK that would allow me to stage resource files on a worker? I have specific static file resources that I need to make available on the file system for a custom DoFn that is performing NLP. My goal would is to get a zip file resource from the classloader and unzip it on the worker file system only once as the worker is being initialized rather than trying to do this in the custom DoFn.

Answer 1

You can specify --filesToStage to specify files that should be staged. There are several issues to be aware of:

1. By default, the Dataflow SDK sets --filesToStage to all of the files in your classpath, which ensures that the code needed to run your pipeline is available to the worker. If you override this option you'll need to make sure that it includes your code.
2. The files on the worker (which will be in the classpath) will have a MD5 hash appended to them. So if you specified --filesToStage=foo.zip, the file name would be foo-<someHash>.zip. You would need to iterate over all the files in the classpath to find the appropriate one.

See the documentation on --filesToStage in https://cloud.google.com/dataflow/pipelines/executing-your-pipeline for some more info.