Applying shader filter to a BitmapData object ignores passed rectangle - how to properly apply shader filter?

Question

I have the following code, made for testing only, as this was a kind of a bug I wanted to nail:

        _shader = new Shader(new TheShader() as ByteArray);
        _shader.data.width.value = [64.0];
        _shader.data.height.value = [64.0];
        _shaderFilter = new ShaderFilter(_shader);
        _sequence = new Vector.<BitmapData>();
        var smallBD:BitmapData;
        var i:int;
        _delta = new Point();
        var megabase:BitmapData = new TheBitmap().bitmapData;
        var _rect:Rectangle = new Rectangle(0, 0, 64, 64);
        for (i = 0; i < 64; i++) {
            smallBD = new BitmapData(64, 64, true, 0x00808080);
            //_rect.x = i;
            _rect.y = i;
            smallBD.applyFilter(megabase, _rect, _delta, _shaderFilter);
            _sequence.push(smallBD);
        }

Then I cycle through _sequence in order to see if changing rectangle actually does something. It does nothing, if _shaderFilter is actually a shader filter. Testing with any of the built-in Flash filter works as intended, but with ShaderFilter it sometimes works as if the rectangle supplied is plainly sourceBitmapData.rect, whatever is the source bitmap, and sometimes it behaves like there's no data passed, with boundary being located at weird position - with a bitmap of size 512x384, the edge of the region that's passed to the shader is apparently located at (256,192) or the center of the bitmap. So far I was only able to implement a workaround, that is, first copyPixels() the required region, then applyFilter() in place. Can someone prove that it's a bug and not me doing something wrong?

PS: I am using FlashDevelop with project target being Flash Player 10.3, and I am unaware if FP11 fixes this.

Answer 1

Well, sadly I can't tell you how to fix this, but I can confirm that it isn't your fault!

The problem seems to be that Flash ignores sourceRect entirely when using custom shaders. At first I thought it might be passing the values to an undocumented parameter in the shader, but then I noticed that every pixel of the output bitmap gets changed, even when the sourceRect is smaller or the destPoint is non-zero. Also there doesn't appear to be an inCoord() function to match outCoord(), so it looks like this isn't a use the developers expected!

I can offer one suggestion; instead of copying the ROI to a new BitmapData object, add a float2 offset parameter to your shader, and shift all pixel lookups by this value. It will save some processing.

Here's the reduced test case I used to confirm the behaviour:

ShaderTest.as:

package {
    import flash.display.Sprite;
    import flash.display.Bitmap;
    import flash.display.BitmapData;
    import flash.display.Shader;
    import flash.geom.Point;
    import flash.geom.Rectangle;
    import flash.filters.ShaderFilter;

    final public class ShaderTest extends Sprite {
        [Embed(source="test.pbj",mimeType="application/octet-stream")]
        private static const sCopy : Class;

        final private function R( x, y, w, h ) : Rectangle {
            return new Rectangle( x, y, w, h );
        }
        final public function ShaderTest( ) {
            super( );
            var
            s  : Shader = new Shader( new sCopy( ) ),
            f  : ShaderFilter = new ShaderFilter( s ),
            d1 : BitmapData = new BitmapData( 256, 256, false, 0 ),
            d2 : BitmapData = new BitmapData( 128, 128, false ),
            b1 : Bitmap = new Bitmap( d1 ),
            b2 : Bitmap = new Bitmap( d2 ),
            w  : Rectangle = R( 16, 16, 64, 64 );
            b2.x = 274;
            addChild( b1 );
            addChild( b2 );

            for( var i : int = 0; i < 8; ++ i ) {
                for( var j : int = 0; j < 8; ++ j ) {
                    d1.fillRect( R( i * 32 + 1, j * 32 + 1, 30, 30 ), (((i + j) & 1) * 0x00FF00) | (i << 21) | (j << 5) );
                }
            }
            d2.applyFilter( d1, w, new Point( 10, 10 ), f );
            d1.fillRect( R( w.x, w.y, 1, w.height ), 0xFF0000 );
            d1.fillRect( R( w.x, w.y, w.width, 1 ), 0xFF0000 );
            d1.fillRect( R( w.x, w.y + w.height - 1, w.width, 1 ), 0xFF0000 );
            d1.fillRect( R( w.x + w.width - 1, w.y, 1, w.height ), 0xFF0000 );
        }
    }
}

test.pbk:

<languageVersion:1.0;>

kernel bugtest <namespace:"Me";vendor:"Me";version:1;>{
    input image4 src;
    output pixel4 dst;
    void evaluatePixel(){
        dst = sampleNearest(src,outCoord());
    }
}

Output:

(the small square on the right copies from the large square using a shader. The red box shows the sourceRect. The destPoint is (10,10). Despite both of these settings, it actually renders the entire bitmap)