How can Wikitude be used in a project which also uses OpenCV?

Question

We can't get Wikitude working in a Xamarin crossplatform project (Android and iOS), which also uses OpenCV.

The Setup

Xamarin crossplatform project using MvvmCross with the following setup:

• Core-project, PCL providing view models, business logic and platform-agnostic logic in general.
• OpenCV.Droid, bindings project providing OpenCV to the Android part.
  • Android Library (AAR), created from the official OpenCV SDK for Android.
• OpenCV.iOS, bindings project providing OpenCV to the iOS part using P/Invoke.
  • Own static library project.
  • Includes the static library from the official OpenCV SDK for iOS, which is just an Apple Framework (combining the static library and headers).
  • Exposes OpenCV functionality via own API to minimize the amount of marshaling code.
• UI.Droid, Android platform project providing views and platform specific logic.
• UI.iOS, iOS platform project providing views and platform specific logic.

Our ARchitectWorld uses Object Tracking and could successfully be tested in the Wikitude examples contained in the Xamarin Component for both Android and iOS.

The Problem

When starting the ARchitectWorld in our iOS project, the world is loaded and we receive the onLoaded-callback when creating the TargetCollectionResource, as well as the onTargetsLoaded-callback when creating the ObjectTracker. So far, the behavior is the same in Wikitude's sample project, as well as our project.

The problem is, that the same object using the same WTO-file is not being recognized in our project, while it's working very well in the sample project.

The Investigation

When that problem occurred, we streamlined the logic in the ViewController which handles the ARchitectWorld, to rule out problems in our code there.

We then investigated our project by adding the ARchitectWorld to different commits in our repo's history, which we deemed critical changes and found commits, were object tracking was working.

To find the critical change, we used git-bisect and identified the commit which introduced the problem. With that commit the static library of our OpenCV bindings project was updated, because it was missing the device-architectures - it only had the simulator archs, basically x86_64.

After that, we assumed, Wikitude might use OpenCV internally and we could get symbol collisions, even though the linker didn't complain about duplicate symbols. Though, Wikitude does not use OpenCV but a search for the combined use of Wikitude and OpenCV brought up some results without solutions:

• Crash when work with OpenCV
• Wikitude iOS JavaScript Crash on detection

So then we found out, Wikitude and OpenCV both include partly the same third party code.

The questions

In the linked posts, the Wikitude support answered quite openly, suggesting an internal discussion and coming back to the OPs. Though no further answers are in the thread, suggesting either there is none or the information has been exchanged through another channel.

In other posts, we saw the suggestion of using OpenCV with Wikitude, by preparing visual data and feed it to an input plugin. This suggests, that OpenCV and Wikitude can be used together.

• Can, and if, how can this be achieved in a Xamarin project like the one I described?
• Does it require a certain version of OpenCV? (We are currently using v3.3.1.)

Maybe the community also achieved this and I'm very looking forward to answers about that. We also contacted Wikitude support and will feedback information here, if we receive any, that is useful to answer the question.

Answer 1

We finally have a working application using Wikitude and OpenCV. Though we were not able to clearly identify the root cause. The steps that led to a working solution are as follows:

1. Build OpenCV from the source with only the modules we need using the --without parameter for the build script build_framework.py to reduce the amount of possible conflicts. This was wrong anyways, as the binaries get stripped during native linking and no errors appeared.
   • Doing this with the downloadable source of v3.3.1, the version of OpenCV we have been using, we ran into a symbol problem during native linking, when compiling the Xamarin project.
   • This led to an issue in OpenCV, that has just recently been fixed (2018-01-10): https://github.com/opencv/opencv/issues/10565
     (We were not building with --dyanmic, still faced the error during native linking when compiling the Xamarin project.)
2. Building OpenCV from current master on GitHub (commit 17233c687e2d51010dab2587f2532a460ccdb4b6, something beyond v3.4.0) turns out to work and not break Wikitude: https://github.com/opencv/opencv
3. Using the downloadable OpenCV Framework for iOS (v3.4.0) led to the same issue during native linking.

TL;DR

Building OpenCV from its source on GitHub, at least commit 17233c687e2d51010dab2587f2532a460ccdb4b6 turned out to not break Wikitude.

We will have a look at the downloadable iOS framework of the next release of OpenCV, to see, whether that works and return to using that, instead of building OpenCV ourselves.