Storing ARKit point cloud data and retrieving for display

Question

I'm hoping to store point cloud data recorded using ARKit and Unity such that it can be retrieved and displayed as it was originally recorded. Let's say that I am simply displaying point cloud data exactly how the UnityPointCloudExample.cs script does, which is included in the Unity plugin. However, when a point is detected and displayed, I also store some relevant information about that point. Then I close the application completely. When I open the application again, I want to be able to reload the data in the same positions relative to the camera as it was when originally detected, is this possible using ARKit and the Unity plugin as-is?

I know that this would require storing some information about the camera's position relative to a point's position, and then when restarting the application some kind of translation would need to be done between the camera's new position on restart and its position from the previous session in which the points were recorded, and then using this information to place points in the correct position. Looking through the ARKit documentation I am not exactly sure how I would achieve this using the native interface, and I am even less certain how I would achieve it using the Unity plugin. If someone could at least direct me towards elements of the unity plugin or the native ARKit interface that would most easily facilitate the implementation of the above functionality I would greatly appreciate it.

Or, if this would be beyond the scope of ARKit/Unity plugin in its current state, explaining how and why this is the case would be equally helpful. Thanks!

Answer 1

ARKit sets the origin to 0,0,0 when AR tracking first begins. There's no way to reload the AR Scene properly on subsequent runs using coordinates from a previous run without defining a relationship between the points from the previous run and the points from the new run.

In order to relate previous ARKit runs with a new run, we can use landmarks, either manually placed, or detected using some kind of object recognition. Lets assume we are manually placing landmarks for simplicity.

Here's the pipeline that will allow us to save and restore an ARKit scene between subsequent runs.

1. Initial scene set-up procedure.

   • Begin our ARKit app for the first run, to place objects or play a game.
   • Allow ARKit to initialize.
   • Select two reference points along a flat horizontal plane in our environment. For example, if indoors, we could select two corners of the room. These points will be what we use to reload our ARKit scene.

2. Place objects in ARSpace as desired. When done, save the position of our AR objects and our two reference points to a file.

3. On reload, place the same two reference points in the position we previously saved. With these two points defined, it is now possible to reload the assets in their previous locations by obtaining their locations relative to the old points, and then placing them relative to the newly defined points.

To reduce the required user interaction, we could extend this with image tracking / detection. When a landmark image or object is detected, we automatically set its location in ARSpace as one of the two points. When both landmarks have been detected, we can 'automatically' reload the scene as described in step 3. This would eliminate the initial point placement error.