How to take screenshot (high fps) in Linux (programming)

Question

First of all I want to say that I've been reading a lot about this and I've learnt many ways to do it, but I haven't been able to do it in linux.

My project is an ambilight with arduino, so I need to take a screenshot of the desktop and analyze its colour.

At the beginning I used Processing 2.0 with the class 'Robot' from 'java.awt'. Initially I could take 5 frames per second and then I got 13fps. This works but I want more perfomance, so I start reading.

In Windows or Mac you have libraries that let you access directly to the 'frameBuffer', so you can take screenshot really 'easy' and really fast.

In Ubuntu I have tried python with Gtk, PIL, Qt... and the fastest way is GTK but I can only have about 15fps too.

My problem is: I want to do it cross platform but I prefer that my program work in Linux at the beginning and then in Windows (I don't like it too much :P).

So, the first question: is python able to offer that perfomance? Because I think that C++ can be a better option.

And the second question: what do I need to do it? I've read about Xlib (X11) but I can't find documentation that let me take a screenshot. Also I know, for example, FFmpeg which is a powerful tool but I don't know how to implement it.

I hope that you could help me (and excuse me if I've made any mistakes).

Answer 1

Making this work cross platform is likely to be quite a bit of work. If your final target is windows, then why not use the amblone project, which seems to do exactly what you want?

http://amblone.com/guide

At any rate, here is a solution with ffmpeg & graphicsmagick that is pretty fast (on my i7 8GB laptop). ffmpeg captures exactly one screen, reduces it to the smallest square size that it can, pipes the output to graphicsmagick convert, where it is resized to 1x1 pixel and then reports the image rgb values.

#!/bin/bash

mkfifo /tmp/screencap.fifo

while true
    do
        # this version will send the info to a fifo
        # ffmpeg -y -loglevel error -f x11grab -s 1920x1080 -i :0.0 -s 32x32 \
        # -vframes 1 -f image2 -threads 2 - |  gm convert - -resize 1x1 \
        # txt:- > /tmp/screencap.fifo

        # this version will write out the info to the command line
        # and will show you what is going on.
        ffmpeg -y -loglevel error -f x11grab -s 1920x1080 -i :0.0 -s 32x32 \
         -vframes 1 -f image2 -threads 2 - |  gm convert - -resize 1x1 txt:-
    done
exit

This will give you something like the following:

0,0: ( 62, 63, 63) #3E3F3F
0,0: (204,205,203) #CCCDCB
0,0: ( 77, 78, 76) #4D4E4C

The 0,0 is the location of the pixel being read. The numbers in parenthesis are the respective R,G,B values, and the numbers at the end are your typical html-esque hex values. In the case above there is only 1 pixel, but you could (if you wanted to have the cardinal directions as generalized RGB values) simply change the -resize 1x1 part above to -resize 3x3 and you'll get something like:

0,0: ( 62, 63, 65) #3E3F41
1,0: ( 90, 90, 91) #5A5A5B
2,0: (104,105,106) #68696A
0,1: ( 52, 51, 52) #343334
1,1: ( 60, 60, 59) #3C3C3B
2,1: ( 64, 64, 64) #404040
0,2: ( 49, 49, 50) #313132
1,2: ( 60, 60, 60) #3C3C3C
2,2: ( 65, 65, 65) #414141

I'll leave it to you to pass that information to your arduino.

ffmpeg is great, but you'll have to remember to switch out the screen-capture bit (here in my example -f x11grab) with whatever your windows system uses. Here is a SO link that goes into a bit more detail.

If you really insist on making something cross-platform, then I would recommend diving into openCV with python bindings and using the framebuffer device as a video input, scaling the result down to 1x1 pixel and using the resulting color average to drive your pwm through some type of UDP broadcast.