How to Integrate Xamarin.android with OpenCV

Question

im trying to connect Xamarin.android with OpenCV in visual Studio the documentation is really poor can anyone provide me with some steps how to do it

Answer 1

There is more than one way to use OpenCV on Xamarin.Android:

1 - Using a Binding of OpenCV4Android: OpenCV4Android is a wrapper of OpenCV (C++) for Android (Java) using JNI. With the binding, we have an interface between Java and C# (More information in https://learn.microsoft.com/en-us/xamarin/android/platform/binding-java-library/ ).

It was implemented in https://github.com/jeremy-ellis-tech/Xamarin.Android.OpenCV, where it was used OpenCV 3.1.0. You can follow the installation instructions and "Reducing the .dll size" to generate a dll and reference it, or you can insert in your Visual Studio Solution the Visual Studio Project of folder "/src/OpenCV.Binding" and add a Reference from your project to this new project.

If you want to use a newer version, you can download a newer version of OpenCV4Android (file with name opencv-version-android-sdk.zip in OpenCV SourceForge, Ex: OpenCV4Android 4.1.0 on link https://sourceforge.net/projects/opencvlibrary/files/4.1.0/) and replace the content of folder "/src/OpenCV.Binding/Jars" in the previous project with the content of folders "/sdk/native/libs" and "/sdk/native/3rdparty/libs" of an extracted OpenCV4Android.

There's also a NuGet of a fork of this project: https://www.nuget.org/packages/Xamarin.OpenCV.Droid that can simplify the installation and use, but I didn't use this so I can't say if it works.

As this method is a Binding of OpenCV4Android and not pure OpenCV, you'll use the documentation of OpenCV4Android (https://opencv.org/android/). It's also worth saying that this way, we have three layers of programming languages (C# - Java - C++), so we have performance losses with method calling (JNI is a burden). Therefore, it's suggested that you use as few calling as you can.

2 - Using a Wrapper of OpenCV C++: with this way, we'll use C++ Shared Libraries (.so) and call its methods from C# (https://learn.microsoft.com/en-us/xamarin/android/platform/native-libraries). To make this, we would need to write PInvoke of OpenCV methods, which are many, which means a lot of time. So we'll use what someone already made.

We have OpenCvSharp, which is a wrapper of OpenCV to .NET, and apparently works well. Problem: it's not compatible with ARM, so it won't run on smartphones. However, a good soul adapted it to run on ARM devices: https://github.com/Kawaian/OpenCvSharp.

How to use it fast: you insert the project of folder "/src/OpenCvSharp" in your solution and reference it. You copy the content of "/src/OpenCvSharp.Android/Native" to the folder "lib" or "libs" of your project. Then you configure the ".so" files to "Always copy" to OutPut Directory and configure their Build Action to "AndroidNativeLibrary"(if your project is an app) or "Embedded Native Library" (if your project is an Android Library).

Another way is to install the NuGet (https://www.nuget.org/packages/Kawaian.OpenCVSharp/), which makes a little easier, but it'll also be required to copy the ".so" files to "lib" or "libs" and configure them.

This wrapper use OpenCV 3.2.0. I'm studying a way to update the OpenCV version on this project, but for now, it works.

The great advantage of this way is performance (~30% of improving in my application when comparing the two implementations). But a single disadvantage is the absence of already made Android.Bitmap - OpenCV.Mat conversion methods. I implemented them by adapting the conversion methods of OpenCV4Android:

// Method adapted from the original OpenCV convert at https://github.com/opencv/opencv/blob/b39cd06249213220e802bb64260727711d9fc98c/modules/java/generator/src/cpp/utils.cpp
///<summary>
///This function converts an image in the OpenCV Mat representation to the Android Bitmap.
///The input Mat object has to be of the types 'CV_8UC1' (gray-scale), 'CV_8UC3' (RGB) or 'CV_8UC4' (RGBA).
///The output Bitmap object has to be of the same size as the input Mat and of the types 'ARGB_8888' or 'RGB_565'.
///This function throws an exception if the conversion fails.
///</summary>
///<param name="srcImage">srcImage is a valid input Mat object of types 'CV_8UC1', 'CV_8UC3' or 'CV_8UC4'</param>
///<param name="dstImage">dstImage is a valid Bitmap object of the same size as the Mat and of type 'ARGB_8888' or 'RGB_565'</param>
///<param name="needPremultiplyAlpha">premultiplyAlpha is a flag, that determines, whether the Mat needs to be converted to alpha premultiplied format (like Android keeps 'ARGB_8888' bitmaps); the flag is ignored for 'RGB_565' bitmaps.</param>
public static void MatToBitmap(Mat srcImage, Bitmap dstImage, bool needPremultiplyAlpha = false)
{
    var bitmapInfo = dstImage.GetBitmapInfo();
    var bitmapPixels = dstImage.LockPixels();
 
    if (bitmapInfo.Format != Format.Rgba8888 && bitmapInfo.Format != Format.Rgb565)
        throw new Exception("Invalid Bitmap Format: It is of format " + bitmapInfo.Format.ToString() + " and is expected Rgba8888 or Rgb565");
    if (srcImage.Dims() != 2)
        throw new Exception("The source image has " + srcImage.Dims() + " dimensions, while it is expected 2");
    if (srcImage.Cols != bitmapInfo.Width || srcImage.Rows != bitmapInfo.Height)
        throw new Exception("The source image and the output Bitmap don't have the same amount of rows and columns");
    if (srcImage.Type() != MatType.CV_8UC1 && srcImage.Type() != MatType.CV_8UC3 && srcImage.Type() != MatType.CV_8UC4)
        throw new Exception("The source image has the type " + srcImage.Type().ToString() + ", while it is expected CV_8UC1, CV_8UC3 or CV_8UC4");
    if (bitmapPixels == null)
        throw new Exception("Can't lock the output bitmap");
 
    if (bitmapInfo.Format == Format.Rgba8888)
    {
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC4, bitmapPixels);
        if (srcImage.Type() == MatType.CV_8UC1)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC1 -> RGBA_8888");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.GRAY2RGBA);
        }
        else if (srcImage.Type() == MatType.CV_8UC3)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC3 -> RGBA_8888");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGB2RGBA);
        }
        else if (srcImage.Type() == MatType.CV_8UC4)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC4 -> RGBA_8888");
            if (needPremultiplyAlpha)
                Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGBA2mRGBA);
            else
                srcImage.CopyTo(tmp);
        }
    }
    else
    {
        // info.format == ANDROID_BITMAP_FORMAT_RGB_565
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC2, bitmapPixels);
        if (srcImage.Type() == MatType.CV_8UC1)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC1 -> RGB_565");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.GRAY2BGR565);
        }
        else if (srcImage.Type() == MatType.CV_8UC3)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC3 -> RGB_565");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGB2BGR565);
        }
        else if (srcImage.Type() == MatType.CV_8UC4)
        {
            Android.Util.Log.Info("MatToBitmap", "CV_8UC4 -> RGB_565");
            Cv2.CvtColor(srcImage, tmp, ColorConversionCodes.RGBA2BGR565);
        }
    }
    dstImage.UnlockPixels();
    return;
}

// Method adapted from the original OpenCV convert at https://github.com/opencv/opencv/blob/b39cd06249213220e802bb64260727711d9fc98c/modules/java/generator/src/cpp/utils.cpp
///<summary>
///This function converts an Android Bitmap image to the OpenCV Mat.
///'ARGB_8888' and 'RGB_565' input Bitmap formats are supported.
///The output Mat is always created of the same size as the input Bitmap and of the 'CV_8UC4' type,it keeps the image in RGBA format.
///This function throws an exception if the conversion fails.
///</summary>
///<param name="srcImage">srcImage is a valid input Bitmap object of the type 'ARGB_8888' or 'RGB_565'</param>
///<param name="dstImage">dstImage is a valid output Mat object, it will be reallocated if needed, so it may be empty.</param>
///<param name="needUnPremultiplyAlpha">unPremultiplyAlpha is a flag, that determines, whether the bitmap needs to be converted from alpha premultiplied format (like Android keeps 'ARGB_8888' ones) to regular one; this flag is ignored for 'RGB_565' bitmaps.</param>
public static void BitmapToMat(Bitmap srcImage, Mat dstImage, bool needUnPremultiplyAlpha = false)
{
    var bitmapInfo = srcImage.GetBitmapInfo();
    var bitmapPixels = srcImage.LockPixels();
 
    if (bitmapInfo.Format != Format.Rgba8888 && bitmapInfo.Format != Format.Rgb565)
        throw new Exception("Invalid Bitmap Format: It is of format " + bitmapInfo.Format.ToString() + " and is expected Rgba8888 or Rgb565");
    if (bitmapPixels == null)
        throw new Exception("Can't lock the source bitmap");
 
    dstImage.Create((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC4);
 
    if (bitmapInfo.Format == Format.Rgba8888)
    {
        Android.Util.Log.Info("nBitmapToMat", "RGBA_8888 -> CV_8UC4");
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC4, bitmapPixels);
        if (needUnPremultiplyAlpha)
            Cv2.CvtColor(tmp, dstImage, ColorConversionCodes.mRGBA2RGBA);
        else
            tmp.CopyTo(dstImage);
    }
    else
    {
        // info.format == ANDROID_BITMAP_FORMAT_RGB_565
        Android.Util.Log.Info("nBitmapToMat", "RGB_565 -> CV_8UC4");
        Mat tmp = new Mat((int)bitmapInfo.Height, (int)bitmapInfo.Width, MatType.CV_8UC2, bitmapPixels);
        Cv2.CvtColor(tmp, dstImage, ColorConversionCodes.BGR5652RGBA);
    }
 
    srcImage.UnlockPixels();
    return;
}