Rendering a 3D mesh into an image using python

Question

I'm a working a neural network that does face reconstruction from a single image using tensorflow.

I'm trying to figure out how to render the output of the network (which is a 3D mesh) into an image in python.

Most of the libraries I found do 3D rendering in real time, I'm only looking to render and single frame.

I also need something that is fast, because this is potentially going to be part of the pre-processing for the network in real-time (apply the network to live video.

The mesh is quite simple, it's a simple face with about 30k vertices and 50k triangles.

Answer 1

I've just had a similar problem and solved it with the pyrender and its off-screen renderer.

Take a look at the minimum working example here PyrenderOSMesaSample.ipynb. It is pretty straightforward to build your own mesh and obtain rendered RGB of a scene.

UPD: Here is my MWE

import os
# switch to "osmesa" or "egl" before loading pyrender
os.environ["PYOPENGL_PLATFORM"] = "osmesa"

import numpy as np
import pyrender
import trimesh
import matplotlib.pyplot as plt

# generate mesh
sphere = trimesh.creation.icosphere(subdivisions=4, radius=0.8)
sphere.vertices+=1e-2*np.random.randn(*sphere.vertices.shape)
mesh = pyrender.Mesh.from_trimesh(sphere, smooth=False)

# compose scene
scene = pyrender.Scene(ambient_light=[.1, .1, .3], bg_color=[0, 0, 0])
camera = pyrender.PerspectiveCamera( yfov=np.pi / 3.0)
light = pyrender.DirectionalLight(color=[1,1,1], intensity=2e3)

scene.add(mesh, pose=  np.eye(4))
scene.add(light, pose=  np.eye(4))

c = 2**-0.5
scene.add(camera, pose=[[ 1,  0,  0,  0],
                        [ 0,  c, -c, -2],
                        [ 0,  c,  c,  2],
                        [ 0,  0,  0,  1]])

# render scene
r = pyrender.OffscreenRenderer(512, 512)
color, _ = r.render(scene)

plt.figure(figsize=(8,8)), plt.imshow(color);