How can I run this shell script inside python?

Question

I want to run a bash script from a python program. The script has a command like this:

find . -type d -exec bash -c 'cd "$0" && gunzip -c *.gz | cut -f 3 >> ../mydoc.txt' {} \;

Normally I would run a subprocess call like:

subprocess.call('ls | wc -l', shell=True)

But that's not possible here because of the quoting signs. Any suggestions?

Thanks!

Answer 1

While the question is answered already, I'll still jump in because I assume that you want to execute that bash script because you do not have the functionally equivalent Python code (which is lees than 40 lines basically, see below). Why do this instead the bash script?

• Your script now is able to run on any OS that has a Python interpreter
• The functionality is a lot easier to read and understand
• If you need anything special, it is always easier to adapt your own code
• More Pythonic :-)

Please bear in mind that is (as your bash script) without any kind of error checking and the output file is a global variable, but that can be changed easily.

import gzip
import os

# create out output file
outfile = open('/tmp/output.txt', mode='w', encoding='utf-8')

def process_line(line):
    """
    get the third column (delimiter is tab char) and write to output file
    """
    columns = line.split('\t')
    if len(columns) > 3:
        outfile.write(columns[3] + '\n')

def process_zipfile(filename):
    """
    read zip file content (we assume text) and split into lines for processing
    """
    print('Reading {0} ...'.format(filename))
    with gzip.open(filename, mode='rb') as f:
        lines = f.read().decode('utf-8').split('\n')
        for line in lines:
            process_line(line.strip())


def process_directory(dirtuple):
    """
    loop thru the list of files in that directory and process any .gz file
    """
    print('Processing {0} ...'.format(dirtuple[0]))
    for filename in dirtuple[2]:
        if filename.endswith('.gz'):
            process_zipfile(os.path.join(dirtuple[0], filename))

# walk the directory tree from current directory downward
for dirtuple in os.walk('.'):
    process_directory(dirtuple)

outfile.close()