Google Cloud Dataflow: can not kill a Job

Question

I have a Google Cloud Dataflow job that seems stuck since over 2 days now (normally takes about 7-10 mins).

In the web UI "cancel job" did nothing, by now the button has even disappeared.

I did kill the VMs used by the job, but they respawn.

gcloud dataflow jobs drain 2018-08-09_18_44_22-352631881911297226 and/or gcloud dataflow jobs cancel 2018-08-09_18_44_22-352631881911297226 do not work, there only is the error

Could not cancel workflow; user does not have sufficient permissions on project: svb-bi-bq, or the job doesnot exist in the project

However, the user has a role of Dataflow Administrator, so the access rights should be ok.

What else can I do to kill this job?

Answer 1

Stopping/deleting the dataflow VMs is strongly advised against. Check this page in docs and this answer to an older question for more details. What you should do at this point is either try and contact support (you can do so via your console) or create an issue in the public tracker to ask for your job to get force-cancelled.