Libgdx masking with image

Question

I have a framebuffer, where some shapes are drawn using ShapeRenderer. And now I want to mask this framebuffer with mask from image. Before that I got it working with simple circle mask drawn by ShapeRenderer. But I need to use more complex mask so I have to use an image. Mask is a png with black mask and transparent background. Here's my code:

   @Override
   public void draw(Batch batch, float parentAlpha) {

      //disable RGB color, only enable ALPHA to the frame buffer
      Gdx.gl.glColorMask(false, false, false, true);

      //change the blending function for our alpha map
      batch.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_ALPHA);

      //draw alpha mask sprite(s)
      batch.draw(maskTexture, MASK_OFFSET_X + getX(), MASK_OFFSET_Y + getY());   

      //flush the batch to the GPU
      batch.flush();

      Gdx.gl.glColorMask(true, true, true, true);
      batch.setBlendFunction(GL20.GL_DST_ALPHA, GL20.GL_ONE_MINUS_DST_ALPHA);

      //The scissor test is optional, but it depends 
      Gdx.gl.glEnable(GL20.GL_SCISSOR_TEST);
      Gdx.gl.glScissor(MASK_OFFSET_X + (int) getX(), MASK_OFFSET_Y + (int) getY(), maskTexture.getWidth(), maskTexture.getHeight());

      //draw framebuffer to be masked
      batch.draw(frm, getX(), getY(), frmSizeX, frmSizeY);

      //remember to flush before changing GL states again
      batch.flush();

      //disable scissor before continuing
      Gdx.gl.glDisable(GL20.GL_SCISSOR_TEST);

      //set default blend function
      batch.setBlendFunction(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
   }

My image is masked indeed, but there's a black background from mask image (it should be transparent). It's looking like this now:

And it should look for example like this (except this example is without mask so ofc paint shoudn't go outside head):

Also take a note that paint is half-transparent. (I don't know if it will change some code).

Ofc I'm using RGBA8888 format, here's initialization code:

frmBuff = new FrameBuffer(Format.RGBA8888, frmSizeX, frmSizeY, false);
frm = new TextureRegion(frmBuff.getColorBufferTexture());
frmCam = new OrthographicCamera(frmSizeX, frmSizeY);
frmCam.translate(frmSizeX / 2, frmSizeY / 2);

maskTexture = game.manager.get("my_mask.png", Texture.class);
maskTexture.setFilter(TextureFilter.Linear, TextureFilter.Linear);

I was messing around setBlendFunction and achieved very different results, but none of them was actually right.

How can I fix this?

Btw my code is based on this example: https://gist.github.com/mattdesl/6076846

I've also already read this: https://github.com/mattdesl/lwjgl-basics/wiki/LibGDX-Masking

Answer 1

I know this is an old post. But all my research on this issue only lead me here. I read the post on https://gist.github.com/mattdesl/6076846 and realized that the issue is that you are only taking into consideration the alpha of the mask, but really you want the alpha of the product of the mask and the foreground (the purple paint).

//disable RGB color, only enable ALPHA to the frame buffer
Gdx.gl.glColorMask(false, false, false, true);

//change the blending function for our alpha map
batch.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_ALPHA);

//draw alpha mask sprite(s)
batch.draw(maskTexture, MASK_OFFSET_X + getX(), MASK_OFFSET_Y + getY());

//change the function for our source
batch.setBlendFunction(GL20.GL_ONE_MINUS_SRC_ALPHA, GL20.GL_SRC_ALPHA);

//draw the source alpha
river_sprite.draw(batch);

//flush the batch to the GPU
batch.flush();