Detecting Chess moves from successive image differences using OpenCV tools

Question

Hey, I am coding up a simple chess playing robot's vision system, I am trying to improve on some previous research to allow camera and a standard chess set be used and both be allowed to move during the game. So far I can locate the board in an image acquired via web-cam, and I want to detect moves by taking difference of successive images to determine what has changed then use previous information about the board occupancy to detect moves.

My problem is that I can't seem to reliably detect changes at the moment, my current pipeline goes like this: Subtract two images -> Histogram equalize the difference image -> erode and dilate diff image to remove minor changes -> make a binary copy and do distance transform -> Get the largest blob(corresponding to the highest value after DT and flood fill that blob) -> repeat again until DT returns a value small enough to ignore change.

I am coding all this in OpenCV and C++. but my flood fill seem to always either not fill the blobs, hence most cases I just get one change detected. I have tried also using cv::inpaint but that didn't help either. So my question is; am I just using the wrong approach or somehow turing can make the change detection more reliable. In case of the former, could people suggest alternative routes, preferable codable in C++/Python and/or OpenCV in a reasonable time?

thanks

Answer 1

The problem of getting a fix on the board and detecting movement of pieces can be solved independently, assuming one does not move the board while also moving pieces around..

Some thoughts on how I would approach it:

Detecting the orientation of the board

You have to be able to handle the board being rotated in place, as well as moved around as long as some angle is maintained that lets you see the pieces. It would help if there were something on the board that you could easily identify (e.g. a marker on each corner) so that if you lose orientation (e.g. someone moves the board away from the camera completely) you could easily find it again.

In order to keep track of the board you need to model the position of the camera relative to the board in 3D space. This is the same problem as determining the location of the camera being moved around a fixed board. A problem of Egomotion. Once you solve that you can move on to the next stage, which is detecting movement and tracking objects.

Detecting movement of pieces

This probably the simpler part of the problem. There are lots of algorithms out there for object detection in video. I would only add that you can use "key" frames. What I mean by that is to identify those frames in which you see only the board before and after a single move. e.g. you don't see the hand moving it obscuring the pieces, etc. Once you have the before/after frame you can figure out what moved and where it is positioned relative to the board.

You can probably get away with not being able to recognize the shape of each piece if you assume continuity (i.e. that you've tracked all movements since the initial arrangement of the board, which is well known).