How to use OpenCV 4 in native C++ of Flutter (in 2021) (support Flutter 2.0)? [closed]

Question

I need to write some C++ code which uses OpenCV, and the Flutter code will call those C++ code.

There are tutorials about writing C++ with Flutter, but I cannot find any up-to-date and easy-to-deploy solution about working with OpenCV. How to do that?

Answer 1

Here is my solution.

Features

1. Works for both Android and iOS.
2. Use static linking instead of dynamic linking. (Thus code size is much smaller.)
3. Up-to-date at 2022.04.03 working with OpenCV 4.5.5 and tested on both Flutter 1.x and 2.10.4. (Since those APIs change rapidly and many articles are a little bit old.)

Getting Started

NOTE: If you already have an app, you can skip this section :) This section assumes that you have no code at all.

The sample code can be downloaded from here.

step 0: Ensure you have Flutter environment, and have followed the official "writing C++ with Flutter" tutorial.

NOTE: It is a must to follow the step of "On iOS, you need to tell Xcode to statically link the file: ...". Otherwise, at our last step iOS will complain the symbol cannot be found.

NOTE: Also, you need to change the "strip style" as mentioned here for iOS. Otherwise, you will see problems in iOS release build while in debug build everything works.

step 1: Write whatever code you like using OpenCV. For instance, I change ios/Classes/native_add.cpp to the following silly code, which is almost identical as in the official tutorial:

#include <stdint.h>
#include <opencv2/core.hpp>

extern "C" __attribute__((visibility("default"))) __attribute__((used))
int32_t native_add(int32_t x, int32_t y) {
    cv::Mat m = cv::Mat::zeros(x, y, CV_8UC3);
    return m.rows + m.cols;
}

Android

Step 0: Download the Android OpenCV sdk from the official website. Say I put it in /Users/tom/Others/OpenCVRelease/OpenCV-android-sdk in my desktop.

Step 1.1: Change the android/CMakeLists.txt to the content of this gist. NOTE: First change the OPENCV_BASE_DIR to your folder.

Of course, the lib/native_with_opencv.dart should change the .so file name to "libnative_with_opencv.so".

Remark: If you need more OpenCV features (such as imread), have a try using this gist.

Step 1.2: Change the android/build.gradle as following:

android {
    ...
    defaultConfig {
        ...
        // [[[CHANGE 1: Make minSdkVersion bigger]]]
        // see https://github.com/opencv/opencv/issues/14419
        minSdkVersion 21

        // [[[CHANGE 2: Add these flags and filters]]]
        externalNativeBuild {
            cmake {
                cppFlags "-frtti -fexceptions -std=c++11"
                abiFilters 'armeabi-v7a', 'arm64-v8a'
            }
        }
        ...
    }
    ...
}

Of course, the minSdkVersion in your actual project (native_with_opencv/example/android/app/build.gradle) should also change to 21.

Done! Compile and enjoy it (and go to the next section for iOS)! If you see 1 + 2 == 3, then everything is fine.

Bonus: If you build in release mode and look at the apk size, you will see our .so file is less than 1MB. Thus static linking and file size reduction does work :)

iOS

Step 0: In ios/native_with_opencv.podspec, add:

  s.static_framework = true
  s.dependency 'OpenCV', '~> 4.5'

Step 1: Run pod install under native_with_opencv/example/ios.

Step 2: Compile and enjoy!

Remark 0: It is a must to follow the step of "On iOS, you need to tell Xcode to statically link the file: ..." in the tut. Otherwise, at our last step iOS will complain the symbol cannot be found.

Remark 0b: May need to check (verify) the following settings in XCode (which seems to be automatically included when dragging some files into XCode but not sure). Otherwise, your final IPA file (can be generated by this) will contain your .cpp source file besides compiled code, thus the source code is leaked.

The setting: Go to "Build Phase" of the "Runner" Target. (1) Look at "Copy Bundle Resources", and verify that your .cpp file or folders are not there. (2) Look at "Compile Sources", and verify your .cpp files are there. (You may need to first add your file in "Compile Sources" before removing it in "Copy Bundle Resources".)

Remark 1: If you are using other .hpp headers and see strange errors, such as OpenCV should be built with C++, or include of a non-modular header inside framework module, then may try this:

Create xxx.modulemap file containing the following framework module the_name_of_your_module {}. Then change your podspec to use this modulemap s.module_map = 'xxx.modulemap'. Then run pod install again to refresh. Then compile and run and it should be OK.

My guess about this problem is that, Cocoapod generates an "umbrella header" (say vision_utils-umbrella.h), and your header is automatically included there. Thus, when compiling that header, things get broken. Thus, my method above tries to remove this umbrella header.

Remark 2: When you add or remove some c++ files, you may have to run pod install again (just like in step 1). Otherwise, you may see errors like "cannot find the symbol" (because Xcode does not look at your newly added C++ file).

Remark 3: If you see problems like Failed to lookup symbol (dlsym(RTLD_DEFAULT, your_function): symbol not found), have a look at this link. I seem to solve it by applying what is suggested in that comment (but not sure since also have tried other ways). If still not solved please reply to me.

---

(Optional) Explanations of how do the Android configuration work: (1) Originally, I just link the core, but there are hundreds of linking errors. Then I search and fix for each group of them. For instance, error: undefined reference to 'carotene_o4t::...' means I need to link with libtegra_hal, thus I add several lines. (2) Strangely, the tbb should be put after core, otherwise, it still does not link. (3) The abiFilters is needed, since tegra_hal does not support x86 (thus no .a file exists). (4) minSdkVersion needs to be raised up, otherwise fegetenv will not be found.