Segmentation - Separating Touching Objects

Question

I have built a system to segment a binary image containing handwritten symbols and classify them (specifically for music). I'm aware there are commercial applications which do this but this is me trying to do it ground-up as a project.

For the purposes of simplicity, let's assume that I will have two elements in my whole image:

AND

I've built something which segments an image in to regions and classifies them. This works fine most of the time.

However, sometimes the elements touch, at which point my classifier breaks down. For example:

OR

What's the best way to separate the two? I've done quite a bit of research but I think my lack of domain knowledge may be letting me down here!

Things I've found:

• Template matching doesn't work well as, the symbols are handwritten
• Thinning/eroding doesn't really work either, especially when it's two sharps (above right) overlapping as they degrade too much.
• Watershed filling doesn't really work on two complex shapes

Things which might work and I'd appreciate a "go for it" or "avoid" vote before I go down the rabbit hole.

• Slide a window L->R of varying sizes and try and classify it. Pick the window and position which has the highest positive classification confidence.
• Take projections (horizontal and vertical) and "cut" the image at the minimum value (which would be the thinnest place on the respective axis

Answer 1

This seems to me like a very hard problem, and I do not have a good general solution. Especially the case of multiple connected # will be difficult to solve.

In your particular case I would try the following, assuming that there are usually not more than two or three symbols clumped together:

• When a blob is too big for a single symbol
  • for each possible symbol
    • take a region in the top left, top right, bottom left, bottom right corner with the correct size for the symbol
    • run your recognition for that region
    • if succesfull, remove the recognized symbol, repeat for the rest

This is not a very sophisticated solution, and how well it works strongly depends on your particular character recognition

---

Another idea:

If most of your shapes tend to have thin vertical segments, you may be able to identify these segments via probabilistic Hough transform, and use the found vertical line segments as starting points for your recognition, whenever a blob contains more than one symbol.

---

Yet another idea to separate shapes:

split the blob at the biggest convexity defect that has a given minimum distance from the border of the blob. Caveat: this works best for convex shapes, and probably not at all for your # signs

---

Alternative 4:

In sheet music, the same sort of symbols tend to appear together, like # followed by a note on the same row, or multiple # at the start of the line in a certain pattern. It might be worthwhile to have a special combined recognizer for such symbols that tend to clump together.

(On that note, how do you currently separate the symbols from the staff lines?)