Python subprocess running in background before returning output

Question

I have some Python code that I want to debug with perf. For that purpose I want to use subprocess. The following command returns instruction-related information of a process until the command is exited via Ctrl^C.

perf stat -p <my_pid>

Now, I want to run this inside a Python code in background, until some point where I want to be able to terminate its operation and print the commands output. To show what I mean:

x = subprocess.call(["perf","stat","-p",str(GetMyProcessID())])

.. CODE TO DEBUG ..

print x   # I want to terminate subprocess here and output 'x'

Now, I want to determine what to do at the line of 'print x' to terminate the process and check the output. Any idea/help is appreciated.

Cheers and thanks in advance,

Answer 1

Use subprocess.Popen to run perf. Then, use pipe.communicate() to send input and get the process's output.

After you've done, call pipe.terminate() to terminate the process.

For example:

pipe = subprocess.Popen(["perf","stat","-p",str(GetMyProcessID())], stdout=PIPE)

pipe.terminate()
stdout, stderr = pipe.communicate()
print stdout

Answer 2

First: I advise against calling perf from within your python process (as you see in the complexity of the task below), but instead use is from the command line:

sudo perf stat -- python test.py

If you really want to call perf from within python then you'll face some tricky problems:

1. to terminate perf and make it output the gathered performance stats you need to send it the SIGINT signal (try it out with sudo perf stat -p mypid: ctrl-\ will print nothing whereas ctrl-c will)
2. you need to capture stderr as perf sends its output to stderr (at least in my version)
3. you need to use fork() with one process sending SIGINT and the other reading it's output while the process dies. Without forks it won't work because after you SIGINTed the perf process you cannot read from stdin any more as the process is already gone, and when you read from stdin first you won't get any output until perf is correctly terminated.

That means you'd end up with this python program:

import subprocess
import os
import signal
import time

perf = subprocess.Popen(['perf', 'stat',  '-p', str(os.getpid())], stderr=subprocess.PIPE)

# <-- your code goes here

if os.fork() == 0:
    # child
    time.sleep(1)  # wait until parent runs `stderr.read()`
    perf.send_signal(signal.SIGINT)
    exit(0)

# parent
print("got perf stats>>{}<<".format(perf.stderr.read().decode("utf-8")))

The time.sleep(1) bit is ugly, what it does it that it will but I guess it will do the trick for 99% of the cases. It has almost no influence on the perf data, the only influence it has is on the "total runtime" (*xx seconds time elapsed)