OpenCV + Android + Unity

Question

I am looking for a way to use OpenCV in a Unity project and my target platform is an Android device.

I know that some assets exists on Unity asset store but I DO NOT want to use them as I find them way too expensive.

I manage to use use opencv in Unity as a C++ native pluggins by precompiling OpenCV in dlls using this tutorial, but dll means Windows Desktop so it doesn't help me much to build my project for Android.

I also found opencv jar archive, I know they can be easily imported into Unity, but I don't know how to do the next step: that is how to access OpenCV stuff from Unity C# scripts.

So, if anyone knows how to configure even a dummy hello world project using OpenCV in Unity editor for build to Android, or even has hints, I would take any infos about that.

Thanks in advance.

PS: I know this question is some sort of vague, and trust me it is not a LMGFY question as on google there is a lot of question like this and no real answer, so please don't rush -1 vote.

UPDATE

Using this tutorial, I managed to build opencv for Android using Android studio, but still I don't know how to use OpenCV from C# scripting. For example, how to create a cv::Mat?

So what I managed to do:

• Build OpenCV and run some native code using Android studio (so from Java).
• Build the Unity native example (one single C function) and call it from a C#script.

But I still can't figure out how to build some C++ code with OpenCV dependencies and call this code from a C# script.

Answer 1

So I finally managed to get it work !!!!!!!!!! =)

I am posting the way it worked for me here. So what I managed to do is build an .so C++ library with link to OpenCV with Visual Studio. Import this library in Unity and build a very simple Android application calling function defined in the .so library. Run the app on an Android phone.

Configuration used:

• Unity 2017.2.0f2
• Visual Studio 2017
• OpenCV 3.3.1 for Android. opencv-3.3.1-android-sdk.zip downloadable from OpenCV website.
• Android smartphone : tested on Lenovo Phab 2 Pro (Google Tango Phone) and HTC 10

Note that the steps I will describe worked for me but it might be different for you if you have a different CPU on your Android device (you'll have to build for ARM64 instead of ARM for instance, but the truth is that these steps are just an example).

I will assume that you already have Android SDK, NDK and Unity installed on your computer, so and that you are already able to build android app with Unity.

STEP 1: create a C++ Android Library with visual Studio 2017.

• File > New > Project

From the Dropdown menu on the left go to Templates > Visual C++ > Cross Platform > Android, and select "Dynamic Shared Library (Android)". (You might have to install the VS tools the be able to build for Android with VS2017). I'll keep the default "SharedObject1" name for this example.

In "Solution platform" (next to Debug/Release dropdown) select "ARM". I suggest you build in release but here we'll stick in debug as the workflow is exactly the same.

STEP 2: Link this C++ Android Project with OpenCV for Android.

• Extract opencv-3.3.1-android-sdk.zip wherever you want (choose and remember the place nevertheless because you'll need to keep the files in this place). You should have a folder with 3 subfolders called "apk", "samples" and "sdk".
• In Visual Studio, go to Project > SharedObject1 Properties. In configuration choose "All configurations" (so it applies in both Debug and Release), and platform choose "ARM". Then:
  • Under C/C++, add the FULL path to OpenCV's includes to "Additional Include Directories". This path is: C:\Path-to-OpenCV-android-sdk\sdk\native\jni\include.
  • Under Linker > General, add the FULL path to OpenCV's libraries to "Additional Library Directories". This path is: C:\Path-to-OpenCV-android-sdk\sdk\native\libs\armeabi-v7a.
  • Under Linker > Input, add the FULL path to OpenCV's libraries file to "Additional Dependencies". This path is: C:\Path-to-OpenCV-android-sdk\sdk\native\libs\armeabi-v7a\libopencv_java3.so.

Note that instead of the full path, you can use environment variable if you know how to set them. I won't explain that here.

STEP 3: Time to write some C++/OpenCV code for our library and build it

ShareObject1.h

extern "C"
{
    namespace SharedObject1
    {
        float Foopluginmethod();
    }
}

ShareObject1.cpp

#include "SharedObject1.h"
#include <opencv2\core.hpp> // use OpenCV in this C++ Android Library

extern "C" 
{
    float SharedObject1::Foopluginmethod()
    {
        cv::Mat img(10,10,CV_8UC1); // use some OpenCV objects
        return img.rows * 1.0f;     // should return 10.0f
    }
}

Then build the library: Build > Build Solution. If you have errors here like "Files not found blablabla", check that you put the full paths in STEP 2 or check your environment variables. If you have other errors, I don't know, ask in comments.

This should have generated a libSharedObject1.so file under Path-to-your-VS-Project\SharedObject1\ARM\Debug (or Release).

STEP 4: Let's go to Unity

• Create a new Unity Project and name it like you want.
• Create a new Scene and save it.
• File > Build Settings. Under Platform select Android and click "Switch Plateform".
• Click on Player Settings. Under "Other Settings", change the Package Name to what you feel like at the moment (Unity doesn't like the default value). Select "ARMv7" for Device Filter.
• Add your scene to the build.

In your scene, select the Main Camera and add a new C# Script named "CallNativeCode" to it: "Add Component" > type "CallNativeCode" > New Script > Create And Add. In the inspector, in Clear Flags choose "Solid color" and put a dark color (this is just for the quick demo).

CallNativeCode.cs

using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;

public class CallNativeCode : MonoBehaviour
{    
    [DllImport("SharedObject1")]
    private static extern float Foopluginmethod();

    void OnGUI ()
    {
        // This Line should display "Foopluginmethod: 10"
        GUI.Label(new Rect(15, 125, 450, 100), "Foopluginmethod: " + Foopluginmethod());
    }
}

Under the Assets folder, create a subfolder called "Plugins" (the spelling is important), and an other sub folder under Plugins called "Android". In this folder, copy the files libSharedObject1.so and libopencv_java3.so (the paths to these two files is in STEP 2 and 3). So you should have something like that :

Select libSharedObject1.so in Unity Editor and check that in the inspector the Selected plaforms for plugin has only Android checked, and that the CPU is ARMv7. Do the same for libopencv_java3.so.

Now you can Build and Run your app on a phone, and enjoy ! ;-)

So this is just a dummy application, but it displays the right phrase, it works !!! =) That means that we managed to make our Android Unity app call OpenCV C++ code. Regarding more complex OpenCV C++ code, well, this is not the topic here, it's time to let your creativity flow.

Answer 2

This is just to add on to Pierre's answer, I wanted to comment but I don't have enough reputation. I followed his answer using the latest version of OpenCV(4.0.1) and there were a few extra things I needed to do.

First, replace the import with #include <opencv2/opencv.hpp>. The other import statement points to old code.

Then go to the project's properties and select C/C++->Language. Set the language standard to C++ 11 and set Enable Run-Time Type Information to yes.

Then go to Code Generation and set Enable C++ Exceptions to yes.

After that, you should be all set.