Can ARKit detect specific surfaces as planes?

Question

Using iOS 11 and iOS 12 and ARKit, we are currently able to detect planes on horizontal surfaces, and we may also visualize that plane on the surface.

I am wondering if we can declare, through some sort of image file, specific surfaces in which we want to detect planes? (possibly ignoring all other planes that ARKit detects from other surfaces)

If that is not possible, could we then perhaps capture the plane detected (via an image), to which we could then process through a CoreML model which identifies that specific surface?

Answer 1

ARKit has no support for such thing at the moment. You can indeed capture the plane detected as an image and if you're able to match this through core ML in real time, I'm sure lot of people would be interested!

You should:

• get the 3D position of the corners of the plane
• find their 2D position in the frame, using sceneView.projectPoint
• extract the frame from the currentFrame.capturedImage
• do an affine transform on the image to be left with the your plane, reprojected to a rectangle
• do some ML / image processing to detect a match

Keep in mind that the ARKit rectangle detection is often not well aligned, and can have only part of the full plane.

Finally, unfortunately, the feature points that ARKit exposes are not useful since they dont contain any characteristics used for matching feature points across frames, and Apple has not say what algorithm they use to compute their feature points.