Why does my function hang when using multiprocessing in Python with Open3D?

Question

I'm trying to use the multiprocessing module in Python to run a function that processes point clouds using the Open3D library. However, when I attempt to run this function in a separate process, it hangs and does not complete.

Here is a simplified version of my code:

import multiprocessing
import numpy as np
import open3d as o3d

def foo():
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(np.array([[0, 0, 0], [1, 1, 1], [2, 2, 2]]))    
    pcd.estimate_normals(search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=0.1, max_nn=30))
    print("normal", np.asarray(pcd.normals))

if __name__ == '__main__':
    print("RUN foo")
    foo()

    print("RUN foo with multiprocessing")
    p = multiprocessing.Process(target=foo)
    p.start()
    p.join()

When I run the code, the function foo() executes correctly when called directly, but when I attempt to run it with multiprocessing, it hangs after starting the process.

**Versions**:
- numpy==1.24.2
- open3d==0.18.0
- python==3.10.15
- ubuntu==20.04

I couldn't find any solutions, so I came here.

Answer 1

One of the reason code hangs is due to multiprocess using fork and Open3D uses OpenMP library which has its own internal threads. See related Open3D issue and Python docs here:

Note that safely forking a multithreaded process is problematic.

Try calling either

multiprocessing.set_start_method("spawn") or multiprocessing.set_start_method("forkserver")

before creating multiprocessing.Process object.

P.S. - The forkserver is not available on Windows but neither is fork. The code hang problem should not happen in Windows.

Also, similar problem is documented well on joblib documentation site.