PipedInputStream / PipedOutputStream, ImageIO and ffmpeg

Question

I have the following code in Scala:

      val pos = new PipedOutputStream()
      val pis = new PipedInputStream(pos)

      Future {
        LOG.trace("Start rendering")
        generateFrames(videoRenderParams.length) {
          img ⇒ ImageIO.write(img, "PNG", pos)
        }
        pos.flush()
        IOUtils.closeQuietly(pos)
        LOG.trace("Finished rendering")
      } onComplete {
        case Success(_) ⇒ 
          LOG.trace("Complete successfully")
        case Failure(err) ⇒
          LOG.error("Can't render stuff", err)
          IOUtils.closeQuietly(pis)
          IOUtils.closeQuietly(pos)
      }

      val prc = (ffmpegCli #< pis).!(logger)

the Future simply writes the generated images one by one to the OutputStream. Now the ffmpeg process reads the input images from stdin and converts them to MP4 file.

That works pretty well, but for some reason sometimes I'm getting the following stacktraces:

I/O error Pipe closed for process: <input stream>
java.io.IOException: Pipe closed
    at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:260)
    at java.io.PipedInputStream.receive(PipedInputStream.java:226)
    at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
    at scala.sys.process.BasicIO$.loop$1(BasicIO.scala:236)
    at scala.sys.process.BasicIO$.transferFullyImpl(BasicIO.scala:242)
    at scala.sys.process.BasicIO$.transferFully(BasicIO.scala:223)
    at scala.sys.process.ProcessImpl$PipeThread.runloop(ProcessImpl.scala:159)
    at scala.sys.process.ProcessImpl$PipeSource.run(ProcessImpl.scala:179)

At the same time I'm getting the following error from another stream:

javax.imageio.IIOException: I/O error writing PNG file!
    at com.sun.imageio.plugins.png.PNGImageWriter.write(PNGImageWriter.java:1168)
    at javax.imageio.ImageWriter.write(ImageWriter.java:615)
    at javax.imageio.ImageIO.doWrite(ImageIO.java:1612)
    at javax.imageio.ImageIO.write(ImageIO.java:1578)
    at 

So it seems that the streams were broken somewhere in between, so ffmpeg can not read the data, and ImageIO can not write the data.

What is even more interesting - the problem is reproducible only on certain Linux server (Amazon). It works flawlessly on other Linux boxes. So I wonder if somebody could point me out to the possible causes of this error.

What I've tried so far:

• use Oracle JDK 8 and OpenJDK
• use different versions of FFMPEG

Nothing worked by the moment.

Answer 1

The problem was kind of predictable and weird at the same time. So there were ten concurrent ffmpeg processes scheduled to handle the input, and the input was a set of hundreds of FullHD pictures. Obviously that takes lot of computation capacity, hence the kernel randomly shut down ffmpeg processes, causing Java wrapper to report broken input and output pipe at the same time.

Thus /var/log/messages contained many of logs like below:

Out of memory: Kill process 25778 (java) score 159 or sacrifice child
Killed process 25931 (ffmpeg) total-vm:2337040kB, anon-rss:966340kB, file-rss:104kB

Reducing the number of concurrent ffmpeg processes solved the issue.