Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Multiple inputs and outputs in python subprocess communicate

Tags:

I need to do something like this post, but I need to create a subprocess that can be given input and give output many times. The accepted answer of that post has good code...

from subprocess import Popen, PIPE, STDOUT  p = Popen(['grep', 'f'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)     grep_stdout = p.communicate(input=b'one\ntwo\nthree\nfour\nfive\nsix\n')[0] print(grep_stdout.decode())  # four # five 

...that I would like to continue like this:

grep_stdout2 = p.communicate(input=b'spam\neggs\nfrench fries\nbacon\nspam\nspam\n')[0] print(grep_stdout2.decode())  # french fries 

But alas, I get the following error:

Traceback (most recent call last):   File "<stdin>", line 1, in <module>   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 928, in communicate     raise ValueError("Cannot send input after starting communication") ValueError: Cannot send input after starting communication 

The proc.stdin.write() method not enable you to collect output, if I understand correctly. What is the simplest way to keep the lines open for ongoing input/output?

Edit: ====================

It looks like pexpect is a useful library for what I am trying to do, but I am having trouble getting it to work. Here is a more complete explanation of my actual task. I am using hfst to get grammar analyses of individual (Russian) words. The following demonstrates its behavior in a bash shell:

$ hfst-lookup analyser-gt-desc.hfstol > слово слово   слово+N+Neu+Inan+Sg+Acc 0.000000 слово   слово+N+Neu+Inan+Sg+Nom 0.000000  > сработай сработай    сработать+V+Perf+IV+Imp+Sg2 0.000000 сработай    сработать+V+Perf+TV+Imp+Sg2 0.000000  >  

I want my script to be able to get the analyses of one form at a time. I tried code like this, but it is not working.

import pexpect  analyzer = pexpect.spawnu('hfst-lookup analyser-gt-desc.hfstol') for newWord in ['слово','сработай'] :     print('Trying', newWord, '...')     analyzer.expect('> ')     analyzer.sendline( newWord )     print(analyzer.before)  # trying слово ... #  # trying сработай ... # слово # слово слово+N+Neu+Inan+Sg+Acc 0.000000 # слово слово+N+Neu+Inan+Sg+Nom 0.000000 #  #  

I obviously have misunderstood what pexpect.before does. How can I get the output for each word, one at a time?

like image 279
reynoldsnlp Avatar asked Feb 19 '15 20:02

reynoldsnlp


People also ask

What is communicate in subprocess?

communicate() writes input (there is no input in this case so it just closes subprocess' stdin to indicate to the subprocess that there is no more input), reads all output, and waits for the subprocess to exit.

Are Subprocesses parallel in Python?

All sub processes are run in parallel. (To avoid this one has to wait explicitly for their completion.) They even can write into the log file at the same time, thus garbling the output. To avoid this you should let each process write into a different logfile and collect all outputs when all processes are finished.

What is difference between subprocess Popen and call?

Popen is more general than subprocess. call . Popen doesn't block, allowing you to interact with the process while it's running, or continue with other things in your Python program. The call to Popen returns a Popen object.

What is subprocess popen ()?

The subprocess module defines one class, Popen and a few wrapper functions that use that class. The constructor for Popen takes arguments to set up the new process so the parent can communicate with it via pipes. It provides all of the functionality of the other modules and functions it replaces, and more.


2 Answers

Popen.communicate() is a helper method that does a one-time write of data to stdin and creates threads to pull data from stdout and stderr. It closes stdin when its done writing data and reads stdout and stderr until those pipes close. You can't do a second communicate because the child has already exited by the time it returns.

An interactive session with a child process is quite a bit more complicated.

One problem is whether the child process even recognizes that it should be interactive. In the C libraries that most command line programs use for interaction, programs run from terminals (e.g., a linux console or "pty" pseudo-terminal) are interactive and flush their output frequently, but those run from other programs via PIPES are non-interactive and flush their output infrequently.

Another is how you should read and process stdout and stderr without deadlocking. For instance, if you block reading stdout, but stderr fills its pipe, the child will halt and you are stuck. You can use threads to pull both into internal buffers.

Yet another is how you deal with a child that exits unexpectedly.

For "unixy" systems like linux and OSX, the pexpect module is written to handle the complexities of an interactive child process. For Windows, there is no good tool that I know of to do it.

like image 90
tdelaney Avatar answered Sep 17 '22 20:09

tdelaney


This answer should be attributed to @J.F.Sebastian. Thanks for the comments!

The following code got my expected behavior:

import pexpect  analyzer = pexpect.spawn('hfst-lookup analyser-gt-desc.hfstol', encoding='utf-8') analyzer.expect('> ')  for word in ['слово', 'сработай']:     print('Trying', word, '...')     analyzer.sendline(word)     analyzer.expect('> ')     print(analyzer.before) 
like image 33
reynoldsnlp Avatar answered Sep 17 '22 20:09

reynoldsnlp