How do you integrate a Leap Motion Controller into a Mac application?

Question

I'm trying to start my own Leap Motion project on the Mac, but I'm having some problems.

I'd like to use Objective-C for this project, and I have read that there is a Leap Motion library for this language. However, I'm not sure how to integrate Leap Motion controls using this library into a Mac application.

Something similar was asked here, only they were asking using the Python Leap Motion library.

How can you add Leap Motion controls to an Objective-C Mac application?

Answer 1

I did this recently, so I can provide the steps that I used to add Leap Motion controls to a Mac application. In fact, the source code to my Leap-enabled Molecules application is available on GitHub if you want an example to work from. All you need to build it is the Leap SDK.

The Leap Motion Objective-C headers are basically wrappers around their underlying C++ API, but you don't need to care about that because you can access this via only Objective-C.

To add the library to your project, first install the Leap SDK somewhere in your system. From there, add references to the Leap.h, LeapMath.h, LeapObjectiveC.h, and LeapObjectiveC.mm files in your project. Add the libLeap.dylib library to your linked libraries.

To avoid compiler and linker errors during the beta period (which may have since been resolved), I needed to go to my build settings and change the C++ Standard Library to libstdc++.

You need to make sure the Leap library gets bundled with your application, so make sure it is copied over into the bundled frameworks during a build phase. I also needed to use the following Run Script build phase to make sure its internal path was set right (again, not sure if this is needed now):

echo TARGET_BUILD_DIR=${TARGET_BUILD_DIR}
echo TARGET_NAME=${TARGET_NAME}
cd "${TARGET_BUILD_DIR}/${TARGET_NAME}.app/Contents/MacOS"
ls -la
# then remap the loader path
install_name_tool -change @loader_path/libLeap.dylib @executable_path/../Resources/libLeap.dylib "${TARGET_NAME}"

One last caution when setting this up is that if you are sandboxing your Mac application, you need to enable the outgoing network connections entitlement, or the application will not be able to connect to the Leap Motion server application running on your Mac.

Once all the setup is completed, you can start getting input from the Leap Motion Controller. A central LeapController object is what will provide delegate callbacks for Leap Motion events. You set one up using code like the following:

controller = [[LeapController alloc] init];
[controller addListener:self];

Your delegate needs to satisfy the LeapListener protocol, which has a bunch of optional callback methods:

// Controller object has initialized
- (void)onInit:(NSNotification *)notification
// Controller has connected
- (void)onConnect:(NSNotification *)notification;
// Controller has disconnected
- (void)onDisconnect:(NSNotification *)notification;
// Exiting your LeapController object
- (void)onExit:(NSNotification *)notification;
// New frame data has arrived from the controller
- (void)onFrame:(NSNotification *)notification;

The connection and disconnection callbacks are obvious, although one thing you'll notice is that the disconnection callback is never triggered while debugging your application in Xcode. You need to run your application outside of a debugger to get that to fire when you disconnect the Leap Motion Controller.

You'll spend most of your time in the -onFrame: callback, because that's where you get your positioning updates. You get your LeapController back as the notification object from that, and you can extract the current frame data using the following:

LeapController *aController = (LeapController *)[notification object];
LeapFrame *frame = [aController frame:0];

The LeapFrame has within it all the scene data observed by the Leap, including hand and finger positions as well as overall scaling of those. The -hands method on LeapFrame gives you an array of all the detected LeapHand objects. In turn, you get each LeapFinger for a LeapHand from its -fingers method. You can also extract the palmPosition, palmNormal, and overall direction of the hand.

Directions are provided as LeapVectors, which are wrapper objects around 3-D vectors. You can extract X, Y, and Z components from them, or perform vector manipulations or comparisons between other LeapVectors.

In Molecules, I adjust the scale and orientation of my molecular structures by reading the movement in-and-out or left-to-right relative to the screen. I do this by comparing the position of an open hand from one frame to the next. I store the previous frame and then do a comparison between now and the last time we saw this hand using

LeapVector *handTranslation = [firstHand translation:previousLeapFrame];

You can also compare scale, rotation, etc. between frames for hands or for all objects in the frame as one group. From the frame-to-frame translation I obtained in the above code, I can extract the individual X, Y, and Z components to rotate my model in response to X and Y translations and scale based on the Z.

I use gross positioning for my application, but you can obviously go a lot finer by tracking individual fingertips. After you've got your application up and running with the device, I recommend reading through the Leap SDK header files for the Objective-C side to see what other capabilities are exposed to you. Also, experiment with different modes of interaction, because you might be surprised at what does and does not work well in 3-D space.