Python support for SparkRunner in Apache Beam

Question

Is there support for running python programs using Apache beam and SparkRunner?

The documentation doesn't seem to have it: https://beam.apache.org/get-started/wordcount-example/#apache-spark-runner

And when I look at the API reference https://beam.apache.org/documentation/sdks/pydoc/0.6.0/apache_beam.runners.html I don't find any mention of SparkRunner in there.

There is mention and support for Java I believe, but I'm wondering if the python support is there?

Answer 1

There's no support for running pipelines built with Apache Beam's Python SDK on the Apache Spark at the moment. However, this work is in progress, embodied in the Apache Beam portability framework.

Stay tuned -- this is something that should be available relatively soon!