In Bash you can easily redirect the output of a process to a temporary file descriptor and it is all automagically handled by bash like this:
$ mydaemon --config-file <(echo "autostart: True \n daemonize: True")
or like this:
$ wc -l <(ls)
15 /dev/fd/63
see how it is not stdin redirection:
$ vim <(echo "Hello World")
vim opens a text file containing "Hello world"
$ echo "Hello World" | vim
Vim: Warning: Input is not from a terminal
You can see in the second example how bash automatically creates a file descriptor and allows you to pass the output of a program to another program.
Now onto my question: How can I do the same thing with Python, using Popen in the subprocess module?
I have been using a normal file of kmers and just reading it in, but my program now generates a specific list of kmers at runtime based on user parameters. I'd like to avoid writing to a temporary file manually because dealing with file permissions could cause problems for my primitive users.
Here is my code to run my program and capture the stdout with an actual file "kmer_file"
input_file = Popen(["pram_axdnull", str(kmer), input_file, kmer_file], stdout=PIPE)
I created a function called generate_kmers which returns a string that can be written out to a file easily (includes newlines) or to a StringIO. I also have a python script that is standalone to do the same thing
So now I want to pass it in as my 3rd parameter:
This doesn't work:
kmer_file = stringIO(generate_kmers(3))
input_file = Popen(["pram_axdnull", str(kmer), input_file, kmer_file], stdout=PIPE)
Nor does this:
kmer_file = Popen(["generate_kmers", str(kmer)], stdout=PIPE)
input_file = Popen(["pram_axdnull", str(kmer), input_file, kmer_file.stdout], stdout=PIPE)
So I am out of ideas.
Does anyone know of a good way to resolve this? I was thinking using the shell=True option and using the actual bashism of <() but I haven't figured that out.
Thank you!
If pram_axdnull
understands "-"
convention to mean: "read from stdin" then you could:
p = Popen(["pram_axdnull", str(kmer), input_filename, "-"],
stdin=PIPE, stdout=PIPE)
output = p.communicate(generate_kmers(3))[0]
If the input is generated by external process:
kmer_proc = Popen(["generate_kmers", str(kmer)], stdout=PIPE)
p = Popen(["pram_axdnull", str(kmer), input_filename, "-"],
stdin=kmer_proc.stdout, stdout=PIPE)
kmer_proc.stdout.close()
output = p.communicate()[0]
If pram_axdnull
doesn't understand "-"
convention:
import os
import tempfile
from subprocess import check_output
with tempfile.NamedTemporaryFile() as file:
file.write(generate_kmers(3))
file.delete = False
try:
p = Popen(["pram_axdnull", str(kmer), input_filename, file.name],
stdout=PIPE)
output = p.communicate()[0]
# or
# output = check_output(["pram_axdnull", str(kmer), input_filename,
file.name])
finally:
os.remove(file.name)
To generate temporary file using external process:
from subprocess import check_call
with tempfile.NamedTemporaryFile() as file:
check_call(["generate_kmers", str(kmer)], stdout=file)
file.delete = False
To avoid waiting for all kmers to be generated i.e., to write/read kmers simultaneously, you could use os.mkfifo()
on Unix (suggested by @cdarke):
import os
import shutil
import tempfile
from contextlib import contextmanager
from subprocess import Popen, PIPE
@contextmanager
def named_pipe():
dirname = tempfile.mkdtemp()
try:
path = os.path.join(dirname, 'named_pipe')
os.mkfifo(path)
yield path
finally:
shutil.rmtree(dirname)
with named_pipe() as path:
p = Popen(["pram_axdnull", str(kmer), input_filename, path],
stdout=PIPE) # read from path
with open(path, 'wb') as wpipe:
kmer_proc = Popen(["generate_kmers", str(kmer)],
stdout=wpipe) # write to path
output = p.communicate()[0]
kmer_proc.wait()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With