OSError: [Errno 12] Cannot allocate memory from python subprocess.call

Question

I've read several similar posts on this issue but none seem to help me directly. If this actually is a duplicate post, please direct me to the thread containing the solution!

I'm saving a bunch of images and then calling ffmpeg on them with subprocess.call. I do this some number of times for collections of different images. This is basically what I'm doing:

from subprocess import call
for video in videos:
  call(['ffmpeg', ..., '-i', video, video+'.mp4')])

In isolation this works fine. However, when I also have some other processing done before these calls (not within the loop, literally just holding values in memory before the loop starts), it crashes with a memory error after having made several of the videos (actually while making the last one). According to this comment, subprocess.call forks/clones the current process, which seems to mean that it requests a memory allocation equal to how much I currently have in memory, which seems to be way overkill for what I want to do in calling ffmpeg.

How can I call ffmpeg from within python without asking to allocate unnecessary amounts of memory?

Answer 1

While it is true that subprocess.call does fork the process, and this child process does have its own memory space that is initially identical to the parent process (your python program), modern operating systems will use copy-on-write memory. The memory overhead of the forked process is initially relatively small, only requiring a few KB of memory in the kernel for process accounting. It's not until that child process starts making changes to its memory that extra memory is required.

After forking, the child process spawned by subprocess.call will call one of the exec system calls, which loads ffmpeg into memory and starts executing it.

Furthermore, fork is usually the only mechanism to create a new process on a POSIX system (see here), so I don't think there's an alternative to the fork-then-exec sequence that subprocess.call implements.

You may try running your program through strace or Valgrind to see exactly what system call is not getting the memory it requests. This may help you determine how to lower your memory requirements.

Answer 2

I had the same issue today and I just worked around it by using os:

import os
for video in videos:
    job = ' ffmpeg ' + ... + ' -i ' + video + '.mp4'
    os.system( job )