Displaying true-colour 2D RGB textures in a 3D plot?

Question

I'm trying to make a 3D plot that consists of a series of 2D planes through an RGB stack, like this:

I know that it's possible to do this using mpl_toolkits.mplot3d by passing the x, y, z coordinates and the RGB(A) colours of each pixel to plot_surface:

import numpy as np
from matplotlib import pyplot as pp
from mpl_toolkits.mplot3d.axes3d import Axes3D

def plot_stack_slices(rgbstack, scale=(1., 1., 1.), z_interval=10.):

    fig, ax = pp.subplots(1,1,subplot_kw={'projection':'3d'})
    ax.invert_zaxis()
    ax.hold(True)

    sx, sy, sz = scale
    nz, ny, nx, nc = rgbstack.shape

    stack_xyz = np.mgrid[:nx*sx:nx*1j, :ny*sy:ny*1j, :nz*sz:nz*1j]

    slices = rgbstack[::-z_interval]
    slice_xyz = np.rollaxis(stack_xyz, 3, 0)[::-z_interval]

    surflist = []

    for (img,xyz) in zip(slices, slice_xyz):
        x, y, z = xyz
        s = ax.plot_surface(x, y, z, facecolors=img**0.75, 
            rstride=50, cstride=50)
        surflist.append(s)

    return fig, ax, surflist

Unfortunately this becomes extremely slow if I set rstride=1, cstride=1 in order to display the textures at full resolution.

I'm also aware that Mayavi can easily handle displaying multiple 2D textures at full resolution:

from mayavi import mlab

def plot_stack_slices2(stack, scale=(1., 1., 20.), z_interval=10.):

    mfig = mlab.figure(bgcolor=(1,)*3)

    sx, sy, sz = scale
    nz, ny, nx = stack.shape

    slices = stack[::-z_interval]
    slice_z = np.linspace(0,nz*sz,nz)[::z_interval]

    surflist = []

    for (img,z) in zip(slices, slice_z):
        im = mlab.imshow(img.T, colormap='gray', figure=mfig)
        im.actor.scale = [sx,sy,sz]
        im.actor.position = [0, 0, z]
        surflist.append(z)


    return fig, surflist

However, the problem now is that there does not seem to be any way of displaying true-colour RGB textures using Mayavi - according to the docs I can only specify either a single (R, G, B) tuple, or a pre-defined colourmap.

Does anyone know of a better way to display true-colour 2D RGB textures in a 3D plot?

Given enough time I could probably figure out how do do this in Vtk or even pure OpenGL if necessary, but I'm really hoping that there are existing libraries that will do the job.

Answer 1

Big thanks to aestrivex for providing working solutions using Mayavi/VTK - it's useful info that I may need for doing more complicated things in the future.

In the end I actually chose to go with cgohlke's suggestion of using visvis, which turned out to be a lot simpler to implement:

import visvis as vv
vv.use('wx')

import numpy as np
from matplotlib.image import imread
from matplotlib.cbook import get_sample_data

imgdata = imread(get_sample_data('lena.png'))

nr, nc = imgdata.shape[:2]
x,y = np.mgrid[:nr, :nc]
z = np.ones((nr, nc))

for ii in xrange(5):
    vv.functions.surf(x, y, z*ii*100, imgdata, aa=3)

Answer 2

I don't know about other libraries -- volshow looks neat but I havent tested it -- but you can do this in vtk.

I have been working on doing this generally in mayavi (see How to directly set RGB/RGBA colors in mayavi) but for certain image sources mayavi structures the vtk pipeline in a way that was not designed to deal with this at all. My efforts to convert a 2D vtk.ImageData to true color starting with mlab.imshow were met with resistance at every step, but I managed it.

First, here is how I have managed to do it in mayavi using mlab. This is far too hacky and "magic"-reliant even for my standards:

from mayavi import mlab
import numpy as np
from tvtk.api import tvtk

k=mlab.imshow(np.random.random((10,10)),colormap='bone')

colors=tvtk.UnsignedCharArray()
colors.from_array(np.random.randint(256,size=(100,3)))
k.mlab_source.dataset.point_data.scalars=colors
k.actor.input.point_data.scalars=colors
#the latter set of scalars is what is actually used in the VTK pipeline in this 
#case, but if they don't play nice with the mayavi source then tvtk will
#complain because we are circumventing the structure it expects
k.actor.input.scalar_type='unsigned_char'
k.actor.input.number_of_scalar_components=3
k.image_map_to_color.lookup_table=None

k.actor.input.modified()
mlab.draw()
#this draw fails.  As it fails, there is an interaction here, somewhere deep in 
#tvtk, causing the ImageData to partially reset.. I have not been able to track 
#it down yet.  ignore the error output

k.actor.input.scalar_type='unsigned_char'
k.actor.input.number_of_scalar_components=3
#now after we reset these back to what they should be, it works

mlab.draw()
mlab.show()

But in pure tvtk it's not nearly so bad:

import numpy as np
from tvtk.api import tvtk

colors=np.random.randint(256,size=(100,3))

an_image=tvtk.ImageData()
an_image.number_of_scalar_components=3
an_image.scalar_type='unsigned_char'
an_image.point_data.scalars=tvtk.UnsignedCharArray()
an_image.point_data.scalars.from_array(colors)
an_image.dimensions=np.array((10,10,1))

an_actor=tvtk.ImageActor()
an_actor.input=an_image
an_actor.interpolate=False

ren=tvtk.Renderer()
renWin=tvtk.RenderWindow()
renWin.add_renderer(ren)
ren.add_actor2d(an_actor)
iren=tvtk.RenderWindowInteractor()
iren.render_window=renWin
iren.interactor_style=tvtk.InteractorStyleTrackballCamera()
renWin.render()
iren.start()

Of course, doing it in vtk is more work. You might even be able to wrap this nicely so that it's pretty reasonable.

I want to fix mayavi to handle this properly, but as you can see from my snippet it is not straightforward and could take a while.