Capturing from the screen and saving to disk multithreaded

Question

The follwing problem should watch the screen, record an event (a measuring textbox turns green) and record all the events leading up to it, producing a "film" of the events leading up to it. Unfortunately the whole screen needs to be recorded. I have so far done the part where the recognition takes part. However i barely get two frames per second. I would like to have around 25 to 30 fps.

My Idea was to do the writing and reading in two seperate threads. Because the writing event is rare and can run in the background, the recording event can take up more time and run faster. Unfortunately the whole thing seems to be too slow. I would like to be able to write on disk the screen the 10 to 20 seconds before the event occurred.

Edit: If possible i would like to stay as platform-independent as possible.

Edit 2: there seems to be a platform independent jar file for Xuggler. Unfortunately i don't really get how i will be able to use it for my purpose: recording the 20 seconds leading up to the point where isarecord is triggered.

Here is what i have done so far:

package fragrecord;

import java.awt.AWTException;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

import javax.imageio.ImageIO;

public class Main {
    public static void main(String[] args) {
        //The numbers are just silly tune parameters. Refer to the API.
        //The important thing is, we are passing a bounded queue.
        ExecutorService consumer = new ThreadPoolExecutor(1,4,30,TimeUnit.SECONDS,new LinkedBlockingQueue<Runnable>(100));
        System.out.println("starting");
        //No need to bound the queue for this executor.
        //Use utility method instead of the complicated Constructor.
        ExecutorService producer = Executors.newSingleThreadExecutor();

        Runnable produce = new Produce(consumer);
        producer.submit(produce);  
        try {
            producer.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        consumer.shutdown();
        try {
            consumer.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }
}

class Produce implements Runnable {
    private final ExecutorService consumer;

    public Produce(ExecutorService consumer) {
        this.consumer = consumer;
    }
    boolean isarecord(BufferedImage image){
        int x=10, y = 10;
        Color c = new Color(image.getRGB(x,y));
        int red = c.getRed();
        int green = c.getGreen();
        int blue = c.getBlue();
        // Determine whether to start recording
        return false;

    }


    @Override
    public void run() {

        Robot robot = null;
        try {
            robot = new Robot();
        } catch (AWTException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        //
        // Capture screen from the top left to bottom right
        //
        int i = 0;
        while(true) {

            i++;
        BufferedImage bufferedImage = robot.createScreenCapture(
                new Rectangle(new Dimension(1024, 798)));

        Runnable consume = new Consume(bufferedImage,i);
        consumer.submit(consume);
        }

    }
}

class Consume implements Runnable {
    private final BufferedImage bufferedImage;
    private final Integer picnr;
    public Consume(BufferedImage bufferedImage, Integer picnr){
        this.bufferedImage = bufferedImage;
        this.picnr = picnr;
    }

    @Override
    public void run() {
        File imageFile = new File("screenshot"+picnr+".png");
        try {
            System.out.println("screenshot"+picnr+".png");
            ImageIO.write(bufferedImage, "png", imageFile);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

Answer 1

I have tried to edit your code a little, Instead of creating png files I have tried creating bmp files which removes overhead time of data compression, but at the cost of disk space.

Result : I do not know how to count fps but the solution is faster that yours. :-)

Answer 2

You should measure how much time robot.createScreenCapture() takes. Chances are that it needs more than 40 ms which means there is no way to achieve what you want with pure Java. In my experience, the call can be very slow.

I did manage to get that time down considerably with a trick but it works only on Unix: Start a VNC server (= desktop in RAM). I patched the source code for TightVNC to use NIO to write the images to disk using a memory mapped file. That gave me about 10-20 fps.

Here is the code to write an image using NIO:

private File pixelFile = new File("tmp", "pixels.nio").getAbsoluteFile();
private IntBuffer intBuffer;
private FileChannel rwChannel;

private MappedByteBuffer byteBuffer;
private int[] pixels;

private void createMemoryMappedFile() {
    File dir = pixelFile.getParentFile();
    if(!dir.exists()) {
        dir.mkdirs();
    }

    try {
        rwChannel = new RandomAccessFile(pixelFile, "rw").getChannel();

        int width = ...;
        int height = ...;
        pixels = new int[width*height];

        byteBuffer = rwChannel.map(MapMode.READ_WRITE, 0, width * height * 4);
        intBuffer = byteBuffer.asIntBuffer();
    } catch(Exception e) {
        throw new RuntimeException("Error creating NIO file " + pixelFile, e);
    }
}

public void saveImage() {

     buffer.position(0);
     buffer.put(image.getRaster().getPixels(0,0,width,height,pixels));

     flushPixels();
}

private void flushPixels() {
    byteBuffer.force();
}