Continuously capture Android screen for processing in an app

Question

I need a live stream or some method to see what is currently being displayed on screen. I do not want to save a screenshot to a file, or record a video etc. I do not want to stream the screen to another device (I'm not trying to screencast/mirracast), I need the screen contents locally on a Android application.

Why would I need that?

I want to write an Android application to manage a lightpack by adapting the lights to what is being displayed on the android device.

• Requiring root is fine (I expected root to be required).
• Needing additional lower level binaries is also acceptable.
• The quality of the image does not have to be exact as on screen, a much lower resolution will still be acceptable

If you say it can't be done, you're wrong. If you download the Prismatik app for Android you'll see exactly what I want, it does screen grabbing and manages my lightpack without any performance issues while grabbing the screen. I just want to create my own application that does similar, except I will make it an open source GitHub project...

edit: I think using /dev/graphics/fb0 could be part of an answer. I don't know how the performance of using this will be though.

Answer 1

I have found an answer to my own question.

Android (linux too) has a framebuffer file (/dev/graphics/fb0 on Android) which you can read that contains the current framebuffer of the entire screen contents. In most cases it will actually contain 2 frames for double buffering...This file will constantly change as the contents on the screen changes (for what I want to use it for, it is fine, I don’t need EXACT to the < millisecond contents, I only need to roughtly know what colors are displayed where on screen)

This file contains the graphics in a RAW format. On my Galaxy S3 it contained 4bytes per pixel. If you need to know how many bytes you should read to get your current screen (only 1 frame), you need to do the math. My screen is 720x1280 pixels.

Therefore I need to read:

720 x 1280 pixels = 921,600 pixels

921,600 pixels * 4 bytes per pixel = 3,686,400 bytes

From this you can extract each color channel, and do the calculations you need. The method to read the data is really up to you. You can write a native bit of code to read the buffer and do manipulations. Or you can read the buffer from Android code with an Input Stream.

Depending on the ROM loaded on your device you might need root to access this file (most cases you will need root)