Bash style process substitution with Python's Popen

Question

In Bash you can easily redirect the output of a process to a temporary file descriptor and it is all automagically handled by bash like this:

$ mydaemon --config-file <(echo "autostart: True \n daemonize: True")

or like this:

$ wc -l <(ls)
15 /dev/fd/63

see how it is not stdin redirection:

$ vim <(echo "Hello World") 
vim opens a text file containing "Hello world"
$ echo  "Hello World" | vim
Vim: Warning: Input is not from a terminal

You can see in the second example how bash automatically creates a file descriptor and allows you to pass the output of a program to another program.

Now onto my question: How can I do the same thing with Python, using Popen in the subprocess module?

I have been using a normal file of kmers and just reading it in, but my program now generates a specific list of kmers at runtime based on user parameters. I'd like to avoid writing to a temporary file manually because dealing with file permissions could cause problems for my primitive users.

Here is my code to run my program and capture the stdout with an actual file "kmer_file"

input_file = Popen(["pram_axdnull", str(kmer), input_file, kmer_file], stdout=PIPE)

I created a function called generate_kmers which returns a string that can be written out to a file easily (includes newlines) or to a StringIO. I also have a python script that is standalone to do the same thing

So now I want to pass it in as my 3rd parameter:

This doesn't work:

kmer_file = stringIO(generate_kmers(3))
input_file = Popen(["pram_axdnull", str(kmer), input_file, kmer_file], stdout=PIPE)

Nor does this:

kmer_file = Popen(["generate_kmers", str(kmer)], stdout=PIPE)
input_file = Popen(["pram_axdnull", str(kmer), input_file, kmer_file.stdout], stdout=PIPE)

So I am out of ideas.

Does anyone know of a good way to resolve this? I was thinking using the shell=True option and using the actual bashism of <() but I haven't figured that out.

Thank you!

Answer 1

If pram_axdnull understands "-" convention to mean: "read from stdin" then you could:

p = Popen(["pram_axdnull", str(kmer), input_filename, "-"],
          stdin=PIPE, stdout=PIPE)
output = p.communicate(generate_kmers(3))[0]

If the input is generated by external process:

kmer_proc = Popen(["generate_kmers", str(kmer)], stdout=PIPE)
p = Popen(["pram_axdnull", str(kmer), input_filename, "-"],
          stdin=kmer_proc.stdout, stdout=PIPE)
kmer_proc.stdout.close()
output = p.communicate()[0]

If pram_axdnull doesn't understand "-" convention:

import os
import tempfile
from subprocess import check_output

with tempfile.NamedTemporaryFile() as file:
    file.write(generate_kmers(3))
    file.delete = False

try:
    p = Popen(["pram_axdnull", str(kmer), input_filename, file.name],
              stdout=PIPE)
    output = p.communicate()[0]
    # or
    # output = check_output(["pram_axdnull", str(kmer), input_filename, 
                             file.name])
finally:
    os.remove(file.name)

To generate temporary file using external process:

from subprocess import check_call

with tempfile.NamedTemporaryFile() as file:
    check_call(["generate_kmers", str(kmer)], stdout=file)
    file.delete = False

To avoid waiting for all kmers to be generated i.e., to write/read kmers simultaneously, you could use os.mkfifo() on Unix (suggested by @cdarke):

import os
import shutil
import tempfile
from contextlib import contextmanager
from subprocess import Popen, PIPE

@contextmanager
def named_pipe():
    dirname = tempfile.mkdtemp()
    try:
        path = os.path.join(dirname, 'named_pipe')
        os.mkfifo(path)
        yield path
    finally:
        shutil.rmtree(dirname)

with named_pipe() as path:
    p = Popen(["pram_axdnull", str(kmer), input_filename, path],
              stdout=PIPE) # read from path
    with open(path, 'wb') as wpipe:
        kmer_proc = Popen(["generate_kmers", str(kmer)],
                          stdout=wpipe) # write to path
    output = p.communicate()[0]
    kmer_proc.wait()