Using OpenCV in Hololens

Question

We have developed in the past years our own tracking technology for custom markers. That tracker uses OpenCV for many image processing functions. It has been designed to work on PC systems (Windows and Linux) using USB and GigE cameras but now we would like to modify it to work on Hololens using Hololens cameras. Ideally make use of Hololens depth sensors but let's stick to color images for this post.

1. Hololens has a CPU, a GPU plus a Microsoft made HPU for image processing/holographic computation I believe. Thereby, is there any build of OpenCV that can fully exploit that kind of hardware? I doubt there is and I did made some searches but here is my question anyway!

2. Is there any OpenCL (CL, not CV) implementation that could run on the Hololens GPU/HPU? Because OpenCV could use that implementation.

3. I know that Hololens uses the new UWP (Universal Windows Platform) paradigm or an app that runs anywhere on the Microsoft ecosystem. Is there any CPU only build of OpenCV that could be used in this regard?

Thank you very much!

Answer 1

There is a NuGet package that you can get from Visual Studios called OpenCV-HoloLens by Sylvain Prevost. It is a UWP Dll that you can incorporate into your HoloLens C++ project.

Basically, all you have to do is create a new project -> Windows Universal -> DLL and then import the NuGet package once this is done you can build the project and go into your project folders and get your DLL files from there.

I assume the previous answer just assumed you were using Unity since that is what the majority of developers are using, so I will also include an answer for Unity users as well.

For people that use Unity, this Dll can also be imported into Unity. However, you wont be able to run it in the editor as Unity does not support UWP plugins but, you can still build it to the HoloLens. Once you build the project all you need to do is create a new folder in Unity called Plugins and put all of your DLL files from your project into this folder.

In order to communicate from the DLL to your Unity scripts you create a C++ file in the DLL project(before you compile) and from there you can make extern methods that you are able to use after you compile the plugins and are able to use them in Unity.