Safely shut down snakemake programmatically

Question

I am looking for a way to shutdown/exit/halt a running snakemake workflow programmatically - essentially with a python function that is called in the workflow but may run into an unrecoverable error requiring the workflow to stop for human intervention.

What I am actually trying to do: I start (guppy basecaller) jobs on GPU nodes, and have to specify in the command which cuda core to use. The function checks if lock files exist to specify which cores are in use and which are available. The files are created and removed as part of the shell command of the basecaller. Using a resource the number of parallel gpu jobs is limited to the available number of cores. This works, but I want to be able to catch unexpected problems if e.g. a gpu_lock file got removed or not cleaned.

The function is called in the workflow to specify a parameter, e.g. as the dummy below:

def get_fromel(wildcards):
  if some_number < 0.05:
    sys.exit("yieeeks")
  else:
    return "hiyaaa"

rule foo:
  input: bar.txt
  output: baz.txt
  params: 
     fromel = get_fromel
  shell:
     "fizz -f {params.fromel} {input} > {output}

Do I simply call sys.exit("my message")? I am worried that it will not clean up incomplete files etc

Answer 1

Instead of calling sys.exit(), why not raise an Exception in your python function and let Snakemake stop itself? Like this, other running jobs will finish and Snakemake will clean up everything properly.

def get_fromel(wildcards):
  if some_number < 0.05:
    raise Exception("Something went wrong")
  else:
    return "hiyaaa"

rule foo:
  input: "bar.txt"
  output: "baz.txt"
  params: 
     fromel = get_fromel
  shell:
     "fizz -f {params.fromel} {input} > {output}"

In the example case you provided, if some_number is < 0.05, Snakemake will not even start the run. But if the check is dynamic and depends on files in your workflow, then this should work.