Recognizing checkboxes in azure form recognizer

Question

The forms I want to use in form recognizer have several checkboxes. Currently, form recognizer's OCR engine doesn't seem to detect checkboxes. Is there a work-around for this, or is there a way to train checkbox recognition?

I've also tried using tesseract on windows and linux, but I couldn't solve it there either.

The checkboxes are your standard open squares. Sometimes they may have an "x" or checkmark on them.

If there is no solution, then I may ask the customer to change from checkboxes to filled in circles (radio-buttons). I haven't tested that yet, to see what OCR would make of them.

Edit: I read the form recognizer documentation and I saw that it explicitly says checkboxes and radio buttons are not supported. I wonder if anyone has a awork-around?

Answer 1

An OCR is by definition made for "character recognition" (see here). A checkbox or a radio button is not a character, so it will not be recognized by an OCR.

You can still give a try to a custom vision detector to find those items, but it will be a bit complicated as this service will only find those items in your document and you will have to combine with an OCR call to get the text, then try to match the zones in the document to know which text is associated with which combo/radio button

Answer 2

Support for checkboxes was added to Form Recognizer in version 2.1 (in public preview as of September 2020). From the announcement:

Checkbox / Selection Mark detection – Form Recognizer supports detection and extraction of selection marks such as check boxes and radio buttons. Selection Marks are extracted in Layout and you can now also label and train in Train Custom Model - Train with Labels to extract key value pairs for selection marks.

There is now a selectionMarks object in the Get Analyze Layout Result API response that lists detected selection marks and their state, either selected or unselected.

Support for labeling checkboxes and selection marks was also added to the sample labeling tool as of version 2.1.