What are appropriate starting points for converting video on canvas to a 3d mesh

Question

Is there currently a library in JavaScript for converting a video stream in Canvas into a 3D mesh?

An example situation is to point a camera at a cup and to capture the raw imagery, comparing previous data with new data to generate an edge outline of the objects within view, ideally converting it into the simplest set of polygons and displaying the calculated polygon/model in a second canvas element.

I've seen some excellent work by mrdoob on 3d javascript going from object files to rendering in canvas, but I'm looking to capture raw data and try to generate a mesh. The intention is to send the mesh points between 2 users over sockets for any object currently within the camera view that gets captured.

Any help appreciated.

Answer 1

It's quite a complex problem. Most algorithms require 2 images (stereo vision systems) in order to recognize and process 2d information into 3d objects. Doing this with a webcam and supported by Javascript is quite a challenge :)

Basically, I think that you'll have to combine some computer vision techniques (edge detection, filtering, 3d to 2d mapping, ...).

Here is some information that could be useful

A computer vision library for Javascript - https://github.com/alex-m/seevee.js/wiki (ports some of the well known opencv functions) - can be useful to get webcam image, 3d processing, etc

lightweight 3D library - https://github.com/mrdoob/three.js/

See this approach http://www.eng.cam.ac.uk/news/stories/2009/3D_models/

Answer 2

It's not a library, but here is a demonstration of precisely the technique you're looking for.

Webcam Mesh