What's the best marker in a video for recognition?

Question

I'm currently building an automatic synchronisation processing between a video track (coming from a camera) and an audio tracks (coming from a micro). To do that, I plan to build a small web application who will display a clapper with my smartphone (Ex: Iphone) and emit a specific bip when I will be filmed. After that, I will execute a small OpenCV application (JavaCV in reality) which will detect the image where my smartphone has displayed the clapper. It will remain for me only find the specific bip to synchronise the two tracks.

You can see here(http://vimeo.com/47002882) and here(http://vimeo.com/46213073) videos that we have already built and synchronise manually. Based on this context (distance, lighting, etc of the previous videos) What is it for you, the best thing (shape, color or other) that can be detected almost all the time ? And what kind of transformation, I should use ?

You must imagine that I will display the clapper from my smartphone with a camera who will be placed at five-seven meters from me.

I built a first basic clapper (http://jsbin.com/zuqumiso/45/) and used a video treatment with OpenCV like that :

Frame -> a RGB to GRAY conversion -> GaussianBlur -> AdaptativeThreshold -> Hought Transformation (standard) to detect the lines.

But it's efficient only when my smartphone is at thirty centimeters from the camera...

Answer 1

Given the very small size of your phone screen in the video, I do not think that you will be able to automatically detect any standard marker (e.g. QR codes, circles or chessboard) displayed on the phone.

If I were you, I would try with a much simpler approach. I would start by displaying a fullscreen blinking black and white pattern on the phone. If the phone is kept reasonnably steady, this should enable detecting the blinking area as the area where intensity variance is highest on a few successive frames. Then, for audio/video synchronisation, I would simultaneously emit the bip and turn the fullscreen pattern into a specific color (e.g. bright red or bright green). Again, this transition from black/white to fully saturated color should be detected quite easily, by detecting the color saturation discontinuity on the blinking area identified previously.

This might not be what you were after when asking this question in the first place, but given your tight constraints on phone/camrea distance, I don't think there is any other way.

Answer 2

Most image algorithms work by looking at transitions (aka; edges). As your camera is a RGB camera, the ideal detectable edge would be an edge that is visible for all colors.

Th easiest option would be to use a simple QR code. It's edges are easy to detect and you can even destinguish between multiple codes.

Shape and color tend to be quite a bit more fickle then QR codes.