Android Fourier Transform Realtime - Renderscript

Question

I am trying to apply a 2D Fourier Transform on incoming preview camera frames. So here is my renderScript code that executes on each onSurfaceTextureUpdated:

#pragma version(1)
#pragma rs java_package_name(foo.camerarealtimefilters)

rs_allocation inPixels;
int height;
int width;

void root(const uchar4 *in, uchar4 *out, uint32_t x, uint32_t y) {

    float3 fourierPixel;
    for(int k=0; k<=width; k++){
        for(int l=0; l<=height; l++){
            float3 pixel = convert_float4(rsGetElementAt_uchar4(inPixels, k, l)).rgb;
            float greyOrigPixel = (pixel.r + pixel.g + pixel.b)/3;
            float angle = 2 * M_PI * ( ((x * k) / width) + ((y * l) / height) );
            fourierPixel.rgb = greyOrigPixel*cos(angle);
        };
    };

    out->xyz = convert_uchar3(fourierPixel);
}

The inPixels is set by this method,

public void setInAllocation(Bitmap bmp) {
    inAllocation = Allocation.createFromBitmap(rs, bmp);
    fourierScript.set_inPixels(inAllocation);
};

Now the maths behind my code? Basically apply Euler's formula, ignore the phase term as I can't do much with imaginary numbers, and draw the magnitude only, that is the real (cosine) part. I of course grayscale the image as you can see.

Here are my resources:

1) http://homepages.inf.ed.ac.uk/rbf/HIPR2/fourier.htm "...In image processing, often only the magnitude of the Fourier Transform is displayed, as it contains most of the information of the geometric structure of the spatial domain image.."

2) http://www.nayuki.io/page/how-to-implement-the-discrete-fourier-transform Where I got the Euler formula, and how I applied it.

My problem, is that when I start my app, it gives me the original image, whatever the camera sees, and nothing more. It also freezes after 2 to 3 seconds.

What is wrong with my code? Is it too much to handle? Is what I am asking possible (I am running this on a Samsung Galaxy S4 Mini)? I just want to apply realtime simple DFT on a camera frame.

Answer 1

It's tough to say why your image would not be showing updates without seeing the Java code. However, here are a few things you might try to help.

• If you can handle lower precision, use float instead of double as this will improve performance

• If you can handle lower precision, use #pragma rs_fp_relaxed which will help performance

• You can re-structure your RS to have a setup function which should be called before it is run for the first time. Use this to setup the width/height and pre-calculate the fixed parts of the FFT equation

It will look something like this:

rs_allocation angles;
uint32_t      width;
uint32_t      height;
uint32_t      total;

void setupPreCalc(uint32_t w, uint32_t h) {
    uint32_t x;
    uint32_t y;
    float curAngle;

    width = w;
    height = h;
    total = w * h;
    for (x = 0; x < width; x++) {
        for (y = 0; y < height; y++) {
            curAngle = 2 * M_PI * (y * width + x);
            rsSetElementAt_float(angles, curAngle, x, y);
        }
    }
}

• Re-structure your kernel to get the output Allocation element and the x and y coordinates being operated on:

void __attribute__((kernel))doFft(uchar4 out, uint32_t x, uint32_t y)

• Before each frame, set the input allocation similar to what you have done then re-structure your loop to use the pre-calculated parts of the angle.

• Previously, the kernel was looping over all coordinates in the input, calculating a greyscale pixel value, running it through something similar to the equation you found, then setting it to be a new pixel value and when done saving that value from the final iteration of the loop to be the output value. This isn't exactly what you want. RS is already giving you a specific location in the output Allocation, so you need to do the summation of all input points in relation to that specific output point.

Using the pre-calc Allocation and the new form of the kernel, it could look like this:

void __attribute__((kernel)) doFft(uchar4 out, uint32_t x, uint32_t y) {
    //  Loop over all input allocation points
    uint32_t inX;
    uint32_t inY;
    float    curAngle;
    float4   curPixel;
    float4   curSum = 0.0;

    for (inX = 0; inX < width; inX++) {
        for (inY = 0; inY < height; inY++) {
            curPixel = convert_float4(rsGetElementAt_uchar4(inPixels, x, y));
            curPixel.rgb = (curPixel.r + curPixel.g + curPixel.b) / 3;

            curAngle = rsGetElementAt_float(angles, inX, inY);
            curAngle = curAngle * ((x + (y * width)) / total);

            curSum += curPixel * cos(curAngle);
        }
    }

    out = convert_uchar4(curSum);
}