Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python's subprocessing with pipes and large files

I'm trying to use python + ffmpeg + oggenc to convert any audiofile to ogg. The program works, almost. But for big files (i think > ~6mb) the ffmpeg process starts to sleep at pipe_wait. I don't know which pipe it waits for.

If I kill the ffmpeg process, the oggenc process continues and I get a resulting ogg-file with about ~2:40 of all the sound.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from subprocess import Popen, PIPE
from sys import argv

ffmpeg = Popen([
    "ffmpeg",
    "-i", argv[1],
    "-vcodec", "null",
    "-acodec", "pcm_s16le",
    "-ac", "2",
    "-ab", "44100",
    "-f", "wav",
    "-"
],stdout = PIPE,stderr = PIPE)

oggenc = Popen([
    "oggenc",
    "-", "--raw",
    "-q", "4",
    "-o", argv[2]
],stdin = ffmpeg.stdout,stderr = PIPE)

oggenc.communicate()
ffmpeg.communicate()

EDIT:

Thought I might add that this works perfectly:

#!/bin/bash

ffmpeg -i "$1" -vcodec null -acodec pcm_s16le -ac 2 -ab 44100 -f wav - | oggenc - --raw -q 4 -o "$2"
like image 362
Linus Unnebäck Avatar asked Mar 01 '10 20:03

Linus Unnebäck


People also ask

How do you use a pipeline in subprocess?

To use a pipe with the subprocess module, you have to pass shell=True . In your particular case, however, the simple solution is to call subprocess. check_output(('ps', '-A')) and then str. find on the output.

What is the difference between subprocess run and subprocess Popen?

The main difference is that subprocess. run() executes a command and waits for it to finish, while with subprocess. Popen you can continue doing your stuff while the process finishes and then just repeatedly call Popen. communicate() yourself to pass and receive data to your process.

What is pipe in Python subprocess?

These arguments are used to set the PIPE, which the child process uses as its stdin and stdout. The subprocess. PIPE is passed as a constant so that either of the subprocess. Popen() or subprocess. PIPE the user specifies that they want the resultant.

How can we avoid shell true in subprocess?

From the docs: args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).


1 Answers

What exactly do you do with the stderr channels of the two pipes?

Encoders/decoders typically produce lots of stderr output, as status updates; this output is piped to your process, and buffers will become full. Perhaps you should add some dummy ffmpeg.stderr.read() call before the (useless, I think) .communicate calls, or even better, drop the stderr=PIPE arguments completely.

UPDATE

For the >/dev/null equivalent, do the following:

nulfp = open(os.devnull, "w")
…
… = subprocess.Popen(…, stderr=nulfp.fileno())

Obviously, you can re-use the same nulfp for all stderrs you want to ignore.

like image 59
tzot Avatar answered Sep 30 '22 13:09

tzot