Access HoloLens depth image in Unity?

Question

Does anyone know how to access the raw depth image from the Hololens depth camera in a Unity app? I know that I have to enable "research mode" on the Hololens. I have seen the example of "SensorStreamViewer" at https://github.com/Microsoft/HoloLensForCV/tree/master/Samples/SensorStreamViewer, but this is a DirectX app. I would like to use Unity because it seems easier than DirectX for prototyping and development (especially for a new developer like me!).

Incidentally, I plan to stream the depth images to an external desktop PC for object recognition, and receive the results back on the Hololens so that I can render holograms aligned with the real object.

Answer 1

Dont know If I understood it right. I would recommend to check the SpatialMappingExample and SpatialProcessing. Its about visualizing and accessing spatial mapping data. Processing the raw spatial mapping data, finding and creating planes based on spatial mapping meshes... But right now they deleted all the guides because of the new MRTK V2 -.- so I cant give you reference

Even the references on the microsoft documentation (on the bottom) are referencing to a not exisiting page

Answer 2

After talking to numerous people, including Microsoft tech support, I have found out that you can only access the Hololens depth camera image if you enable research mode and then write C++ code and use DirectX. It is not possible to access the depth camera using C# and Unity.

If someone else wants to pursue this, it may be possible to write C++ code that accesses the depth camera, and put that into a plugin dll that can be used from Unity. As an example, the following project shows how to access the webcam stream from within Unity (but not the depth camera): https://github.com/VulcanTechnologies/HoloLensCameraStream.