How do you manually release a mutex for an argo workflow?

Question

I have an argo workflow with a mutex e.g.

kind: Workflow
metadata:
  generateName: synchronization-wf-level-
spec:
  entrypoint: whalesay
  synchronization:
    mutex:
      name:  test
  templates:
    - name: whalesay
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["hello world"]

However I deleted a workflow while it was trying to run. Now argo has gone in a deadlock where no jobs can be created using same mutex.

Where does argo store mutex information? and how can I manually remove the mutex information to get out of dead lock?

Answer 1

Workflow deletion should release the acquired lock. All locks are stored in Controller memory. One workaround is restarting controller will clear the all lock and repopulate it. Please create issue in https://github.com/argoproj/argo with sample workflow and controller logs