javaCV Android, Strange colors on overlay while streaming to rtmp server

Question

I wanna stream to facebook live from android. I was able to adapt an existing example for streaming to FB.

That first step more or less works (audio is still a problem, but not in the scope her). I can stream to FB.

I now wanna overlay the stream with a transparent png image.

I am creating a FFmpegFrameFilter on start up by:

try{
    filter = new FFmpegFrameFilter("movie="+path+"/image.png [logo];[in][logo]overlay=0:0:format=rgb [out]",imageWidth, imageHeight);
    filter.start();
}catch (FrameFilter.Exception e){
    Log.e(CLASS_LABEL,"Error while starting filter: "+e.getMessage());
    e.printStackTrace();
}

and on each frame i do the following:

filter.push(yuvImage);

Frame frame;
while ((frame = filter.pull()) != null) {
    recorder.record(frame,avutil.AV_PIX_FMT_NV21);
}

The problem is that i have no idea which pixel format i should use. My overlay image has rgb colors (https://postimg.org/image/f1ri3vj43/)

With the above pixel format i get something like this: https://postimg.org/image/45ha64q9z/

I am pretty frustrated since i already tried many pixel formats. All with a different output, sometimes the logo appears multiple times.

Is there a way to find out which one i should choose from the avutil.java possibilities?

EDIT: You can find the whole code on https://github.com/y4nnick/android_streaming/blob/master/app/src/main/java/com/example/yannick/olay1/RecordActivity.java

EDIT: I allready tried the following formats:

// AV_PIX_FMT_ARGB --> 4 at once, all black/white
// AV_PIX_FMT_0RGB --> 4 at once, all black/white
// AV_PIX_FMT_BGR8 --> 1 a bit to big, strange colors
// AV_PIX_FMT_BGR4_BYTE --> 1 a bit to big, stranger blue tint
// AV_PIX_FMT_YUVA422P_LIBAV --> error: Cannot initialize the conversion context.
// AV_PIX_FMT_FLAG_ALPHA --> error: Cannot initialize the conversion context.
// AV_PIX_FMT_FLAG_PLANAR --> error: Cannot initialize the conversion context.
// AV_PIX_FMT_RGB4 --> error: Cannot initialize the conversion context.
// AV_PIX_FMT_RGB32_1 --> 4 at a time, all black/white
// AV_PIX_FMT_0BGR --> 4 at a time, all black/white
// AV_PIX_FMT_YVYU422 --> 2 side by side, gruen, purple tint
// AV_PIX_FMT_YUVJ422P --> Fatal signal 11 (SIGSEGV) at 0x61f7xf000 (code=1), thread 18401 (e.yannick.olay1)
// AV_PIX_FMT_BAYER_BGGR8 --> 1 a bit to big, black/white
// AV_PIX_FMT_BAYER_GBRG8 --> 1 a bit to big, black/white
// AV_PIX_FMT_FLAG_RGB --> 2 a bit to big, black/white
// AV_PIX_FMT_RGB555LE --> 2 a bit to big, strange colors
// AV_PIX_FMT_RGB555BE --> 2 a bit to big, strange colors
// AV_PIX_FMT_RGB555 --> 2 a bit to big, strange colors
// AV_PIX_FMT_RGB4_BYTE --> 1 a bit to big, orange tint
// AV_PIX_FMT_RGBA64 --> 8 side by side, black/white
// AV_PIX_FMT_RGB24 --> 3 side by side, red tint

Answer 1

I figured it out:

1. Find out which formats your android camera supports by running

   mCamera = Camera.open(); 
   Camera.Parameters params = mCamera.getParameters(); 
   for(int i: params.getSupportedPreviewFormats()) {
      Log.e(TAG, "preview format supported are = "+i);
   }
   

   You will get a list of Integers. In my case 17 and 842094169. I will go with 842094169 (YV12). So set your camera to that by calling

   mCamera.getParameters().setPreviewFormat(ImageFormat.YV12);
   

2. Goto https://developer.android.com/reference/android/graphics/ImageFormat.html and check how your supported formats are defined exactly. 842094169 is YV12 is a 4:2:0 YCrCb planar format comprised of a WxH Y plane followed by (W/2) x (H/2) Cr and Cb planes.

3. Goto https://ffmpeg.org/ffmpeg-filters.html#overlay-1 and check for your filter which you like to use, mine is the overlay filter. This filter provides a parameter called 'format', set it to the closest (YV12 is not in the list) you get from your camera. In my case 'yuv420'. The filter now looks like this:

   filter = new FFmpegFrameFilter("movie="+path+"/image.png [logo];[in][logo]overlay=0:0:format=yuv420 [out]",imageWidth, imageHeight);
   

4. Set the correct pixel format while recording the frame:

   Frame frame;
   while ((frame = filter.pull()) != null) {
       Log.i(LOG_TAG,"In record-pull");
       recorder.record(frame,avutil.AV_PIX_FMT_YUV420P);
   }
   

In fact you must set all your pixelformats to the same. It is a little bit tricky to search throw all the sites to find a format which all your 'components' support.

A suggested alternativ would be to convert the YUV image from the camera to a RGB image. See Converting YUV->RGB(Image processing)->YUV during onPreviewFrame in android?

I did not tried that because i taught this would to be to slow and tried a simpler solution first.