Unity3D on iOS, inspecting the device camera image in Obj-C

Question

I have a Unity/iOS app that captures the user's photo and displays it in the 3D environment. Now I'd like to leverage CIFaceFeature to find eye positions, which requires accessing the native (Objective-C) layer. My flow looks like:

Unity -> WebCamTexture (encode and send image to native -- this is SLOW)

Obj-C -> CIFaceFeature (find eye coords)

Unity -> Display eye positions

I've got a working prototype, but it's slow because I'm capturing the image in Unity (WebCamTexture) and then sending it to Obj-C to do the FaceFeature detection. It seems like there should be a way to simply ask my Obj-C class to "inspect the active camera". This would have to be much, much faster than encoding and passing an image.

So my question, in a nutshell:

• Can I query in Obj-C 'is there a camera currently capturing?'
• If so, how do I 'snapshot' the image from that currently running session?

Thanks!

Answer 1

You can access the Camera's preview capture stream by changing CameraCapture.mm in unity.

I suggest that you have a look at some existing plugin called Camera Capture for an example of how additional camera I/O functionality can be added to the capture session / "capture pipeline".

To set you off in the right direction. have a look at the function initCapture in CameraCapture.mm :

- (bool)initCapture:(AVCaptureDevice*)device width:(int)w height:(int)h fps:(float)fps 

Here you will be able to add to the capture session.

And then you should have a look at the code sample provided by Apple on Facial Recognition :

https://developer.apple.com/library/ios/samplecode/SquareCam/Introduction/Intro.html

Cheers

Answer 2

Unity 3D allows execution of native code. In the scripting reference, look for native plugins. In this way you can display a native iOS view (with the camera view, possibly hidden depending on your requirements) and run Objective C code. Then return the results of eye detection to Unity if you need it in a 3D view.