Snakemake memory limiting

Question

In Snakemake, I have 5 rules. For each I set the memory limit by resources mem_mb option. It looks like this:

rule assembly:
     input:
         file1 = os.path.join(MAIN_DIR, "1.txt"), \
         file2 = os.path.join(MAIN_DIR, "2.txt"), \
         file3 = os.path.join(MAIN_DIR, "3.txt")
     output:
         foldr = dir, \
         file4 = os.path.join(dir, "A.png"), \
         file5 = os.path.join(dir, "A.tsv")
     resources:
         mem_mb=100000
     shell:
         " pythonscript.py -i {input.file1} -v {input.file2} -q {input.file3} --cores 5 -o {output.foldr}  "

I want to limit the memory usage of the whole Snakefile by doing something like:

snakamake --snakefile mysnakefile_snakefile --resources mem_mb=100000

So not all jobs would use 100GB each ( if I have 5 rules, meaning as 500GB memory allocation), but all of their executions will be maximum 100GB ( 5 jobs, total of 100 GB allocation?)

Answer 1

The command line argument sets the total limit. The Snakemake scheduler will ensure that for the set of running jobs, the sum of the mem_mb resources will not exceed the total limit.

I think this is exactly what you want, isn't it? You just need to set the per-job expected memory in the rule itself. Note that Snakemake does not measure this for you. You have to define that value yourself in the rule. E.g., if you expect your job to use 100MB memory, put mem_mb=100 into that rule.