Detecting balls on a pool table

Question

I'm currently working on a project where I need to be able to very reliable get the positions of the balls on a pool table.

I'm using a Kinect v2 above the table as the source.

Initial image looks like this (after converting it to 8-bit from 16-bit by throwing away pixels which is not around table level):

Then a I subtract a reference image with the empty table from the current image.

After thresholding and equalization it looks like this: image

It's fairly easy to detect the individual balls on a single image, the problem is that I have to do it constantly with 30fps.

Difficulties:

• Low resolution image (512*424), a ball is around 4-5 pixel in diameter
• Kinect depth image has a lot of noise from this distance (2 meters)
• Balls look different on the depth image, for example the black ball is kind of inverted compared to the others
• If they touch each other then they can become one blob on the image, if I try to separate them with depth thresholding (only using the top of the balls) then some of the balls can disappear from the image
• It's really important that anything other than balls should not be detected e.g.: cue, hands etc...

My process which kind of works but not reliable enough:

• 16bit to 8bit by thresholding
• Subtracting sample image with empty table
• Cropping
• Thresholding
• Equalizing
• Eroding
• Dilating
• Binary threshold
• Contour finder
• Some further algorithms on the output coordinates

The problem is that a pool cue or hand can be detected as a ball and also if two ball touches then it can cause issues. Also tried with hough circles but with even less success. (Works nicely if the Kinect is closer but then it cant cover the whole table)

Any clues would be much appreciated.

Answer 1

Expanding comments above:

I recommend improving the IRL setup as much as possible. Most of the time it's easier to ensure a reliable setup than to try to "fix" that user computer vision before even getting to detecting/tracking anything.

My suggestions are:

1. Move the camera closer to the table. (the image you posted can be 117% bigger and still cover the pockets)
2. Align the camera to be perfectly perpendicular to the table (and ensure the sensor stand is sturdy and well fixed): it will be easier to process a perfect top down view than a slightly tilted view (which is what the depth gradient shows). (sure the data can be rotated, but why waste CPU cycles when you can simply keep the sensor straight)

With a more reliable setup you should be able to threshold based on depth. You can possible threshold to the centre of balls since the information bellow is occluded anyway. The balls do not deform, so it the radius decreases fast the ball probably went in a pocket.

One you have a clear threshold image, you can findContours() and minEnclosingCircle(). Additionally you should contrain the result based on min and max radius values to avoid other objects that may be in the view (hands, pool cues, etc.). Also have a look at moments() and be sure to read Adrian's excellent Ball Tracking with OpenCV article

It's using Python, but you should be able to find OpenCV equivalent call for the language you use.

In terms tracking If you use OpenCV 2.4 you should look into OpenCV 2.4's tracking algorithms (such as Lucas-Kanade). If you already use OpenCV 3.0, it has it's own list of contributed tracking algorithms (such as TLD).

I recommend starting with Moments first: use the simplest and least computationally expensive setup initially and see how robuts the results are before going into the more complex algorithms (which will take to understand and get the parameters right to get expected results out of)