Numpy (OpenCV) image array to OpenGL texture (pi3d)

Question

I'm using pi3d to display an ImageSprite on the screen the texture of which comes form an image I'm loading.

displayTexture = pi3d.Texture("display/display.jpg", blend=True, mipmap=True) 
displaySlide = pi3d.ImageSprite(texture=displayTexture, shader=shader, w=800, h=600)

This texture image is actually something I'm creating in-program. It's an openCV2 image and therefore just a numpy array. At the moment I'm saving it just to load it again as a texture, but is there a way to just constantly update the texture of the sprite with the changing numpy array values?

I looked into the openCV OpenGL support but from what I could see it only supports Windows at this stage and is therefore not suitable for this use.

Edit: Should have mentioned I'm happy for a lower level solution too. I'm currently trying to use .toString() on the image array and use the resulting byte list with glTexImage2D to produce a texture but no dice so far.

Answer 1

Yes you can pass a PIL.Image to pi3d.Texture and it will create a new Texture using that. There is a bit of work involved there so it will impact on frame rate if it's a big Texture. Also you need to update the pointer in the Buffer that holds the Texture array so the new Texture gets used.

There is a method to load a numpy array to a PIL.Image (Image.fromarray()) so this would be an easy route. However it's a bit convoluted as pi3d already converts the PIL.Image into a numpy array see https://github.com/tipam/pi3d/blob/master/pi3d/Texture.py#L163

The following works ok as a short-cut into the workings of pi3d.Texture but it's a bit of a hack calling the 'private' function _load_opengl. I might look at making a more robust method of doing this (i.e. for mapping videos to 3D objects etc)

#!/usr/bin/python
from __future__ import absolute_import, division, print_function, unicode_literals

import demo
import pi3d
import random
import numpy as np
from PIL import Image, ImageDraw
DISPLAY = pi3d.Display.create(x=150, y=150)
shader = pi3d.Shader("uv_flat")
im = Image.open("textures/PATRN.PNG")
#draw = ImageDraw.Draw(im) # there are various PIL libraries you could use
nparr = np.array(im)
tex = pi3d.Texture(im) # can pass PIL.Image rather than path as string
sprite = pi3d.ImageSprite(tex, shader, w=10.0, h=10.0)
mykeys = pi3d.Keyboard()
while DISPLAY.loop_running():
  #draw.line((random.randint(0,im.size[0]),
  #           random.randint(0,im.size[1]),
  #           random.randint(0,im.size[0]),
  #           random.randint(0,im.size[1])), fill=128) # draw random lines
  #nparr = np.array(im)
  nparr += np.random.randint(-2, 2, nparr.shape) # random noise
  tex.image = nparr
  tex._load_opengl()
  sprite.draw()
  if mykeys.read() == 27:
    mykeys.close()
    DISPLAY.destroy()
    break

PS I can't remember what version of pi3d the switch to numpy textures happened but it's quite recent so you probably have to upgrade

EDIT:

The switch from Texture.image being a bytes object to numpy array was v1.14 posted on 18Mar15

To clarify the steps to use a numpy array to initialise and refresh a changing image:

...
im = Image.fromarray(cv2im)       # cv2im is a numpy array
tex = pi3d.Texture(im)            # create Texture from PIL image
sprite = pi3d.ImageSprite(tex, shader, w=10.0, h=10.0)
...
  tex.image = cv2im               # set Texture.image to modified numpy array
  tex._load_opengl()              # re-run OpenGLESv2 routines