Problems reading data from 3 Kinect cameras

Question

I am trying to read data from multiple Kinect sensors (3 at the moment) and having issues when there's more than 2 devices.

I'm using Daniel Shiffman's OpenKinect Processing wrapper, slightly modified so it allows to open multiple Device instances. Everything works fine with 2 devices. The problem is when I use 3. One Kinect is connected straight into one of the available two usb ports, and the other two are plugged into a USB 2.0 Hub (that has it's own power adapter).

The devices all initialize succesfully:

org.openkinect.Device@1deeb40 initialized
org.openkinect.Device@2c35e initialized
org.openkinect.Device@1cffeb4 initialized

The problem is when I try to get the depth map from the 3rd device, I get an array filled with zeroes. I've thought it's the device, but if swap devices, it's always the 3rd (last connected device) that presents this behaviour.

Here's my code so far:

package librarytests;

import org.openkinect.Context;
import org.openkinect.processing.Kinect;

import processing.core.PApplet;
import processing.core.PVector;

public class PointCloudxN extends PApplet {

    // Kinect Library object
    int numKinects;// = 3;
    Kinect[] kinects;
    int[] colours = {color(192,0,0),color(0,192,0),color(0,0,192),color(192,192,0),color(0,192,192),color(192,0,192)};

    // Size of kinect image
    int w = 640;
    int h = 480;

    // We'll use a lookup table so that we don't have to repeat the math over and over
    float[] depthLookUp = new float[2048];

    // Scale up by 200
    float factor = 200;

    public void setup() {
        size(800,600,P3D);
        numKinects = Context.getContext().devices();
        kinects = new Kinect[numKinects];
        for (int i = 0; i < numKinects; i++) {
            kinects[i] = new Kinect(this);
            kinects[i].start(i);
            kinects[i].enableDepth(true);
            kinects[i].processDepthImage(false);
        }
        // Lookup table for all possible depth values (0 - 2047)
        for (int i = 0; i < depthLookUp.length; i++) {
            depthLookUp[i] = rawDepthToMeters(i);
        }
    }

    public void draw() {
        background(0);

        translate(width/2,height/2,-50);
        rotateY(map(mouseX,0,width,-PI,PI));
        rotateX(map(mouseY,0,height,-PI,PI));
        int skip = 4;//res
        //*
        for (int i = 0; i < numKinects; i++) {
            Kinect kinect = kinects[i];
            int[] depth = kinect.getRawDepth(); 
            //if(frameCount % 60 == 0 && i == 2) println(depth);
            if (depth != null) {

                // Translate and rotate

                for(int x=0; x<w; x+=skip) {
                    for(int y=0; y<h; y+=skip) {
                        int offset = x+y*w;

                        // Convert kinect data to world xyz coordinate
                        int rawDepth = depth[offset];
                        PVector v = depthToWorld(x,y,rawDepth);

                        stroke(colours[i]);
                        // Draw a point
                        point(v.x*factor,v.y*factor,factor-v.z*factor);

                    }
                }
            }
        }


        //*/
    }

    public void stop() {
        for (int i = 0; i < numKinects; i++) kinects[i].quit();
        super.stop();
    }

    public static void main(String _args[]) {
        PApplet.main(new String[] { librarytests.PointCloudxN.class.getName() });
    }

    // These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
    float rawDepthToMeters(int depthValue) {
        if (depthValue < 2047)  {
            return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
        }
        return 0.0f;
    }

    PVector depthToWorld(int x, int y, int depthValue) {

        final double fx_d = 1.0 / 5.9421434211923247e+02;
        final double fy_d = 1.0 / 5.9104053696870778e+02;
        final double cx_d = 3.3930780975300314e+02;
        final double cy_d = 2.4273913761751615e+02;

        PVector result = new PVector();
        double depth =  depthLookUp[depthValue];//rawDepthToMeters(depthValue);
        result.x = (float)((x - cx_d) * depth * fx_d);
        result.y = (float)((y - cy_d) * depth * fy_d);
        result.z = (float)(depth);
        return result;
    }








}

The only major change I've done to Daniel's Kinect class was adding an extra start() method:

public void start(int id) {

        context = Context.getContext();
        if(context.devices() < 1)
        {
            System.out.println("No Kinect devices found.");
        }
        device = context.getDevice(id);
        //device.acceleration(this);

        device.acceleration(new Acceleration()
        {
            void Acceleration(){
                System.out.println("new Acceleration implementation");
            }
            public void direction(float x, float y, float z)
            {
                System.out.printf("Acceleration: %f %f %f\n", x ,y ,z);
            }
        });

        kimg = new RGBImage(p5parent);
        dimg = new DepthImage(p5parent);
        running = true;

        super.start();
    }

I've also tried with MaxMSP/Jitter and the jit.freenect external and I get the same behaviour: I can get 2 depth maps, but the 3rd is not updating.

So it seems to be an issue related to the driver, not the wrapper, since the same behaviour is present using 2 different wrappers to libfreenect (Java/Processing and Max), but am clueless why this happens to be honest.

Has anyone had a similar issue (getting depth feeds from 3 devices) using the OpenKinect/libfreenect Driver ? Any ideas on how I can get past this issue ?

Answer 1

The Kinect is extremely demanding on USB - generally, you can only get one Kinect per USB host controller on your motherboard (most PCs and laptops have two). The only solution I've seen is to buy a PCI-E USB controller and plug the third one into it.

Also, you might be lucky if you reduce the bandwidth requirements by disabling the RGB stream on all the Kinects (I'm blithely assuming you aren't using it since it wasn't mentioned)