Reading pixels from OpenGL/GLUT Application

Question

I have an application that uses OpenGL and GLUT to show some stuff on the screen. I'd like to access the colors displayed "pixel by pixel" but I'm having trouble comprehending the methods provided.

It seems the way to access such data is with the function: readPixels :: Position -> Size -> PixelData a -> IO () which is decidedly non-Haskelly as it uses the C pattern of taking a destination pointer and writing to that pointer.

The third argument is the important part, and it is constructed like PixelData PixelFormat DataType (Ptr a). PixelFormat and DataType are both enums, the former taking values such as RGBAInteger, RGBA, CMYK, etc and the latter taking values such as UnsignedByte, Short, Float, etc. The choices are confusing to me, but that's not the real issue. Where I am really struggling with the use of the pointer. I have no idea what kind of Storable data to malloc to create the pointer, or how many bytes to allocate if I choose to use mallocBytes.

I have a function I'm messing around with that currently looks something like this:

pixels :: IO ()
pixels = do
    pointer <- mallocBytes 50000
    let pdata = PixelData RGBA Int pointer
    let pos = Position 0 0
    let siz = Size 10 10
    readPixels pos siz pdata
    print pdata
    mypixels <- peek pointer
    -- TODO, what kind of data is mypixels?
    free pointer 

It runs fine I just have no idea how to use the data I am getting from the Ptr. Maybe I'm not understanding the documentation fully, but how can I determine the type of data at the pointer and how can I use it in my program? Note that my choice of the arguments RGBA and Int was arbitrary, they just sounded harmless enough. What I really want is some list or multi-dimensional list of RGBA pixel values in some format (Color 4 or something of that nature). Any help would be greatly appreciated, I seem to be in over my head.

Answer 1

You're using low-level bindings, don't be surprised they're low-level. readPixels directly corresponds to glReadPixels (they use MathML, so use a browser that supports it to view it).

What you're getting back is an array of 32-bit integers, each one containing RGBA colour value. You only need width * height * 4 bytes for that, you're overallocating a lot.

Reading from it is made pretty easy by FFI machinery — it provides peekArray which can be used to extract the pixel data into a Haskell list. You need to be working with a pointer of correct type, too. mallocBytes returns pointer of special type that corresponds to void*. I'm using ForeignPtr so that runtime takes care of finalising it (with mallocBytes you'd need to use castPtr and also bracket to account for exceptions).

import Data.Word
import Foreign
import Graphics.Rendering.OpenGL

readPixelArray :: Int -> Int -> Int -> Int -> IO [Word32]
readPixelArray x y w h = do
    let arraySize = w * h
    array <- mallocForeignPtrArray arraySize :: IO (ForeignPtr Word32)
    withForeignPtr array $ \ptr -> do
        -- ptr :: Ptr Word32
        -- fromIntegral is needed because Position and Size store GLints not Ints
        let position = Position (fromIntegral x) (fromIntegral y)
        let size = Size (fromIntegral w) (fromIntegral h)
        readPixels position size $ PixelData RGBA UnsignedInt ptr
        peekArray arraySize ptr

Now you've got a one-dimensional array of colour values. We can split it up to a type like this one:

import Data.Bits

data Pixel = Pixel {
    red :: Word8, green :: Word8, blue :: Word8, alpha :: Word8
} deriving (Eq, Show)

readPixels' :: Int -> Int -> Int -> Int -> IO [Pixel]
readPixels' x y w h = do
    rawPixels <- readPixelArray x y w h
    return $ map pixelFromWord32 rawPixels

-- pixelFromWord32 0xAABBCCDD = Pixel 0xAA 0xBB 0xCC 0xDD
pixelFromWord32 :: Word32 -> Pixel
pixelFromWord32 colour = Pixel (extract 3) (extract 2) (extract 1) (extract 0)
    where extract idx = fromIntegral $ (colour `shiftR` (idx * 8)) .&. 0xFF

Answer 2

Here's a version that reads the image into a JuicyPixels image which can be easily saved to a PNG file etc:

import Codec.Picture (Image, PixelRGB8(..), withImage)
import Control.Lens.Operators
import qualified Foreign as F
import qualified Graphics.Rendering.OpenGL as GL

takeScreenshot :: IO (Image PixelRGB8)
takeScreenshot =
    do
        (pos, size@(GL.Size wGl hGl)) <- GL.get GL.viewport
        let width = fromIntegral wGl
        let height = fromIntegral hGl
        let pixelSize = 3
        -- glY converts top-origin coordinates to OpenGL's bottom-origin system
        let glY y = height - 1 - y
        let pixelOffset x y = ((glY y) * width + x) * pixelSize
        F.allocaBytes (pixelSize * width * height) $
            \ptr ->
            do
                GL.readPixels pos size (GL.PixelData GL.RGB GL.UnsignedByte ptr)
                let readPixel x y =
                        F.plusPtr ptr (pixelOffset x y)
                        & F.peekArray pixelSize
                        <&> (\[r, g, b] -> PixelRGB8 r g b)
                withImage width height readPixel