awk command fails in snakemake --use-singularity

Question

I am trying to combine Snakemake with Singularity, and I noticed that a simple awk command no longer works when using singularity. The $1 in the last line gets replaced by bash instead of being used as the first field by awk.

Here is a minimal working example (Snakefile):

singularity: "docker://debian:stretch"
rule all:
    input: "test.txt"
rule test:
    output: 
        "test.txt"
    shell:
        "cat /etc/passwd | awk -F':' '{{print $1}}' > {output}"

When I run snakemake without singularity, the output test.txt looks as expected (containing only user names). When I run snakemake --use-singularity, the file contains whole lines, e.g. root:x:0:0:root:/root:/bin/bash.

This is the log from Snakemake:

$ snakemake --use-singularity --printshellcmd                                                                                                               
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       all
        1       test
        2

rule test:
    output: test.txt
    jobid: 1

cat /etc/passwd | awk -F':' '{print $1}' > test.txt
Activating singularity image /scratch/test/.snakemake/singularity/fa9c8c7220ff16e314142a5d78ad6cff.simg
Finished job 1.
1 of 2 steps (50%) done

localrule all:
    input: test.txt
    jobid: 0

Finished job 0.
2 of 2 steps (100%) done

Answer 1

I had a similar issue and after lots of trial and error finally solved it. Currently (November 2018, for Snakemake 5.3), this is somewhat undocumented, so I thought it is good to put it here for future reference to help others...

All examples above incorrectly use the double quotation mark with bash -c, which is NOT how Snakemake constructs it. Instead, Snakemake calls Singularity with bash -c ' modified_command ', so single quotes. First, this changes how special characters are treated within the command. Second, as of now, Snakemake replaces all single quotation marks within the actual command with the escaped version \'. However, this applies ONLY when used with Singularity.

Therefore, if your command contains single quotes, things break either when submitting with --use-singularity or when running in normal mode. The only working solution that I am aware of that works in both cases is the following:

shell: """awk "{{OFS="\\t"}};{{print \$2}}" {input}"""

Thus, the following rules apply:

1. Do not use single quotes within the command, as they are replaced otherwise, which causes errors.
2. Escape certain characters, such as \t by \\t, $ by \$, and { by {{.
3. Use triple quotation marks to surround the command line call.

I hope this helps, I will update this post once there are implementation updates.