How to correctly escape special characters in python subprocess? [duplicate]

Question

Im trying to run this bash command using python subprocess

find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'

output:-

helld.xl.py
parse_maillog.py
replace_pattern.py
split_text_match.py
ssh_bad_login.py

Here is what i have done in python2.7 way, but it gives the output where awk command filter is not working

>>> p1=subprocess.Popen(["find","/Users/johndoe/sandbox","-iname","*.py"],stdout=subprocess.PIPE)

>>> p2=subprocess.Popen(['awk','-F"/"','" {print $NF} "'],stdin=p1.stdout,stdout=subprocess.PIPE)

>>>p2.communicate()
('/Users/johndoe/sandbox/argparse.py\n/Users/johndoe/sandbox/custom_logic_substitute.py\n/Users/johndoe/sandbox/finditer_html_parse.py\n/Users/johndoe/sandbox/finditer_simple.py\n/Users/johndoe/sandbox/group_regex.py\n/Users/johndoe/sandbox/helo.py\n/Users/johndoe/sandbox/newdir/helld.xl.py\n/Users/johndoe/sandbox/parse_maillog.py\n/Users/johndoe/sandbox/replace_pattern.py\n/Users/johndoe/sandbox/split_text_match.py\n/Users/johndoe/sandbox/ssh_bad_login.py\n', None)

I could also get output by using p1 alone here like below,but i cant get the awk working here

list1=[]
result=p1.communicate()[0].split("\n")
for item in res:
    a=item.rstrip('/').split('/')
    list1.append(a[-1])
print list1

Answer 1

You are incorrectly passing in shell quoting (and extra shell quoting which isn't even required by the shell!) when you're not invoking a shell. Don't do that.

p2=subprocess.Popen(['awk', '-F/', '{print $NF}'], stdin=...

When you have shell=True you need extra quotes around some arguments to protect them from the shell, but there is no shell here, so putting them in is incorrect, and will cause parse errors by Awk.

However, you should almost never need to call Awk from Python, especially for trivial tasks which Python can easily do natively:

list1 = [line.split('/')[-1]
            for line in subprocess.check_output(
                ["find", "/Users/johndoe/sandbox",
                 "-iname", "*.py"]).splitlines()]

In this particular case, note also that GNU find already has a facility to produce this result directly:

list1 = subprocess.check_output(
    ["find", "/Users/johndoe/sandbox",
     "-iname", "*.py", "-printf", "%f\\n"]).splitlines()