How to create a mask for a transparent overlay?

Question

I have the following scenario: a Bitmap that is used as background and another Bitmap that is used as overlay which can be either 50% transparent or opaque (changeable at run time) and a third Bitmap that contains a mask for this second Bitmap. I've tried different Xfermodes configurations and drawing orders but wasn't able to find the right one.

I'm using the mask as a bitmap because I need to be able to save it between two runs of the program or between configuration changes. It is created as the user draws on the screen, effectively cleaning the fog of war.

Code snippets from by best attempt. Only thing that didn't work as I wished it did was the transparency of my mask.

@Override
protected void onDraw(Canvas canvas) {      
    canvas.drawBitmap(mFogOfWar, mTransformationMatrix, mPaintFog);
    canvas.drawBitmap(mMaskBitmap, mTransformationMatrix, mPaintMask);
    canvas.drawBitmap(mImage, mTransformationMatrix, mPaintImage);
}

Paint objects:

mPaintImage.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.DST_OVER));
mPaintFog.setAlpha(127);
mPaintMask.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.CLEAR));

This is what I get with the current configuration to be more clear:

I'm not sure if I am going to be able to do this setting the alpha on the Paint object; if not, I don't mind another suggestion or solution for the alpha issue, preferably one where the recreation of the bitmap being used as a fog of war is not necessary.

EDIT:

I was able to get the results I want by doing the following:

@Override
protected void onDraw(Canvas canvas) {
    canvas.drawBitmap(mImage, mTransformationMatrix, mPaintImage);
    if (mMaskBitmap != null) {
        canvas.drawBitmap(mFogOfWar, mTransformationMatrix, mPaintFog);
        canvas.drawBitmap(mMaskBitmap, mTransformationMatrix, mPaintMask);
        canvas.drawBitmap(mMaskBitmap, mTransformationMatrix, mPaintImage);
        canvas.drawBitmap(mImage, mTransformationMatrix, mPaintImageSecondPass);
    }

Paint Objects:

mPaintImage = new Paint(); // No Xfermode here anymore
mPaintFog.setAlpha(127);
mPaintMask.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.CLEAR));
mPaintImageSecondPass.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.LIGHTEN));

But drawing the bitmaps five times seems like a waste. As this runs in an OpenGL texture due to Android hardware aceleration (I rescale the bitmaps to the highest resolution accepted by the device's GPU) and I take a lot of care in my invalidates() it runs surprisingly smooth on both my Nexus S and my A500, but I am not sure about other devices (project is going to be 4.0+ anyways).

But I am convinced there must be a better way to do it. I'd like a way that avoided that much overdrawn or that at least could explain to me correctly what those Xfermodes mean and that I am not overdrawing stuff.

Answer 1

I've tried a completely different approach after having some kind of an epiphany - and realized that the solution for this problem was a much simpler approach - as it usually is. And as I need only two bitmaps, I need much fewer memory to work with it.

For drawing:

canvas.drawBitmap(mImage, mTransformationMatrix, mPaintImageRegular);
if (mFogOfWarState != FOG_OF_WAR_HIDDEN) {
    canvas.drawBitmap(mFogOfWar, mTransformationMatrix, mPaintFog);
}

The "secret" was that instead of drawing on the mask bitmap, I'm erasing the fog of war using another paint:

mFogOfWarCanvas.drawPath(mPath, mEraserPaint);

The only Paint that has an Xfermode is the used one for erasing:

mEraserPaint.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.DST_IN));

And for the loading and saving of my mask, I do the following:

private void createFogAndMask(File dataDir) {
    BitmapDrawable tile = (BitmapDrawable) getResources().getDrawable(R.drawable.fog_of_war); 
    tile.setTileModeXY(TileMode.REPEAT, TileMode.REPEAT);
    mFogOfWar = Bitmap.createBitmap(mImageBounds.width(), mImageBounds.height(), Config.ARGB_8888);
    mFogOfWarCanvas = new Canvas(mFogOfWar);
    tile.setBounds(mImageBounds);
    tile.draw(mFogOfWarCanvas);   
    tile = null;

    // Try to load an existing mask
    File existingMask = new File(dataDir, getMaskFileName());
    if (existingMask.exists()) {
        Bitmap existingMaskBitmap = BitmapFactory.decodeFile(existingMask.getAbsolutePath());
        mFogOfWarCanvas.drawBitmap(existingMaskBitmap, new Matrix(), mPaintImageRegular);
        mFogOfWarCanvas.drawPaint(mMaskEraserPaint);
        existingMaskBitmap.recycle();
        System.gc();
    }
}

public void saveMask(File folder) throws IOException {
    if (!mReady || mImagePath == null) return;
    mImage.recycle();
    System.gc();
    if (!folder.exists()) {
        folder.mkdirs();
    }
    File savedFile = new File(folder, getMaskFileName());

    // Change all transparent pixels to black and all non-transparent pixels to transparent
    final int length = mImageBounds.width() * mImageBounds.height();        
    final int[] pixels =  new int[length];
    mFogOfWar.getPixels(pixels, 0, mImageBounds.width(), 0, 0, mImageBounds.width(), mImageBounds.height());
    for (int i = 0; i < length; i++) {
        if (pixels[i] == Color.TRANSPARENT) {
            pixels[i] = Color.BLACK;
        } else {
            pixels[i] = Color.TRANSPARENT;              
        }
    }
    mFogOfWar.setPixels(pixels, 0, mImageBounds.width(), 0, 0, mImageBounds.width(), mImageBounds.height());

    FileOutputStream output = new FileOutputStream(savedFile);
    mFogOfWar.compress(CompressFormat.PNG, 80, output);
    output.flush();
    output.close();     
}