Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python piping output between two subprocesses

I'm working on some code that will DD a block device over SSH, and I'm wanting to do this with subprocess so that I can monitor the status of DD during the transfer (killing the dd process with SIGUSR1 to get its current state, and reading that using selects).

The command that I'm trying to implement would be something like this:

dd if=/dev/sda | ssh [email protected] 'dd of=/dev/sda'

The current method I tried was:

dd_process = subprocess.Popen(['dd','if=/dev/sda'],0,None,None,subprocess.PIPE, subprocess.PIPE)  
ssh_process = subprocess.Popen(['ssh','[email protected]','dd of=/dev/sda'],0,None,dd_process.stdout)

However when I run this, the SSH process becomes defunct after 10-40 seconds.
Am I being completely obtuse here, or is there no way to pipe between subprocesses like this?

Edit: Turns out my real code didn't have the hostname in it. This is the correct way to do things.

like image 428
Philip J Avatar asked Jan 31 '11 01:01

Philip J


People also ask

How do you pass a pipe in subprocess Python?

To use a pipe with the subprocess module, you have to pass shell=True . In your particular case, however, the simple solution is to call subprocess. check_output(('ps', '-A')) and then str. find on the output.

How do I capture the output of a subprocess run?

To capture the output of the subprocess. run method, use an additional argument named “capture_output=True”. You can individually access stdout and stderr values by using “output. stdout” and “output.

How do you use the pipe command in Python?

pipe() method in Python is used to create a pipe. A pipe is a method to pass information from one process to another process. It offers only one-way communication and the passed information is held by the system until it is read by the receiving process.

What is Popen and pipe in Python?

Popen() takes two named arguments, one is stdin and the second is stdout. Both of these arguments are optional. These arguments are used to set the PIPE, which the child process uses as its stdin and stdout. The subprocess. PIPE is passed as a constant so that either of the subprocess.


2 Answers

from subprocess import Popen, PIPE
dd_process = Popen(['dd', 'if=/dev/sda'], stdout=PIPE)
ssh_process = Popen(['ssh', '[email protected]', 'dd','of=/dev/sda'],stdin=dd_process.stdout, stdout=PIPE)
dd_process.stdout.close() # enable write error in dd if ssh dies
out, err = ssh_process.communicate()

This is way to PIPE the first process output to the second. (notice stdin in the ssh_process)

like image 126
Senthil Kumaran Avatar answered Sep 30 '22 10:09

Senthil Kumaran


The sh Python library makes it easy to call OS commands and pipe them.

From the documentation:

for line in tr(tail("-f", "test.log", _piped=True), "[:upper:]", "[:lower:]", _iter=True):
    print(line)
like image 42
Apalala Avatar answered Sep 30 '22 08:09

Apalala