Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I restart a cancelled Cloud Dataflow streaming job?

I've created a standard PubSub to BigQuery dataflow. However, in order to ensure I wasn't going to run up a huge bill while offline, I cancelled the dataflow. From the GCP console, there doesn't seem to be an option to restart it - is this possible, either through the console, or through the shell (and if so, how)?

like image 701
Paul Michaels Avatar asked Jan 03 '18 18:01

Paul Michaels


People also ask

How do I cancel my dataflow job?

Go to the Dataflow Jobs page. Click the job that you want to stop. To stop a job, the status of the job must be running. In the job details page, click Stop.


2 Answers

Cloud Dataflow currently does not provide a mechanism to restart a Dataflow job that has been stopped or cancelled.

However, for this Pub/Sub -> BigQuery flow, one way to approach this would be to use the Google-provided Pub/Sub to BigQuery template; these templates provide code-free solutions for common data movement patterns using Cloud Dataflow.

You can execute a streaming Dataflow job using this template, via the REST API, using a unique job name to ensure that there is only one instance of this Dataflow job running at any point in time. If the job were cancelled, you could (re)start this streaming Dataflow job by running the same command again.

like image 161
Andrew Mo Avatar answered Oct 27 '22 21:10

Andrew Mo


You can restart the job immediately by cloning. You should see an option clone at the top.

like image 45
Mayank Maheshwari Avatar answered Oct 27 '22 19:10

Mayank Maheshwari