Extracting fields from forms with varying structures

Question

I am trying to extract certain fields from a balance sheet. For example, I would like to be able to tell that the value of 'Inventory' is 1,277,838 for the following balance sheet:

Currently, I am using Tesseract to convert images to text. However, this conversion results in a stream of text, so it is difficult to associate fields with their values (since these values are not always right next to the text of their corresponding fields).

After some searching, I read the Tesseract can use uzn files to read from zones of an image. However, the specific zones of the balance sheet values may shift from form to form, so I am interested in any solutions that can determine that 'Inventory' and 1,277,838 are on the same line. Ideally, I would like a grid structure output of text (so that I can tell the spatially that which chunks of text are in the same rows/columns).

Could anyone help explain how I can achieve this result?

Answer 1

I have been performing a similar task using Tesseract and Python (pytesseract library). I have been able to use Tesseract's .hocr output files (https://en.wikipedia.org/wiki/HOCR) to find the location of my search term (e.g 'Inventory') on the page and then rerun Tesseract on a small section of the page which gives it higher accuracy for that area. Here's the code I use to parse the HOCR output from Tesseract:

def parse_hocr(search_terms=None, hocr_file=None, regex=None):
    """Parse the hocr file and find a reasonable bounding box for each of the strings
    in search_terms.  Return a dictionary with values as the bounding box to be used for 
    extracting the appropriate text.

    inputs:
        search_terms = Tuple, A tuple of search terms to look for in the HOCR file.

    outputs:
        box_dict = Dictionary, A dictionary whose keys are the elements of search_terms and values
        are the bounding boxes where those terms are located in the document.
    """
    # Make sure the search terms provided are a tuple.
    if not isinstance(search_terms,tuple):
        raise ValueError('The search_terms parameter must be a tuple')

    # Make sure we got a HOCR file handle when called.
    if not hocr_file:
        raise ValueError('The parser must be provided with an HOCR file handle.')

    # Open the hocr file, read it into BeautifulSoup and extract all the ocr words.
    hocr = open(hocr_file,'r').read()
    soup = bs.BeautifulSoup(hocr,'html.parser')
    words = soup.find_all('span',class_='ocrx_word')

    result = dict()

    # Loop through all the words and look for our search terms.        
    for word in words:

        w = word.get_text().lower()

        for s in search_terms:

            # If the word is in our search terms, find the bounding box
            if len(w) > 1 and difflib.SequenceMatcher(None, s, w).ratio() > .5:
                bbox = word['title'].split(';')
                bbox = bbox[0].split(' ')
                bbox = tuple([int(x) for x in bbox[1:]])

                # Update the result dictionary or raise an error if the search term is in there twice.
                if s not in result.keys():
                    result.update({s:bbox})

            else:
                pass

    return result 

This allows me to search an HOCR file for the appropriate terms and return the bounding box of that particular word. I can then expand the bounding box slightly to run Tesseract on a very small subset of the page. This allows for MUCH greater accuracy than just OCRing the whole page. Obviously, some of this code is particular to my use, but it should give you a place to start.

This page is very helpful for finding the appropriate arguments to give to Tesseract. I found the Page Segmentation Modes to be VERY important for obtaining accurate results for small sections of an image.

Answer 2

As already mentioned by gaw89, Tesseract can output more information than only the text as a stream. The hocr fileformat gives you also the position (bounding boxes) of each paragraph, line, word:

$ tesseract 4LV05.png out -l eng hocr

Then you can for example find the bounding box of the word "Inventory" by simply

$ grep 'Inventory' out.hocr
 <span class='ocr_line' id='line_1_5' title="bbox 23 183 112 204; baseline 0 -5; x_size 21; x_descenders 5; x_ascenders 4"><span class='ocrx_word' id='word_1_15' title='bbox 23 183 112 204; x_wconf 93'>Inventory</span>

Thus, the bounding box of this word spans vertically from 183 to 204 and for corresponding value of this label, we have now to search for boxes in the same vertical space. This can for example been achieved here by

$ grep 'bbox [0-9]* 18[0-9]' out.hocr
<p class='ocr_par' id='par_1_4' lang='eng' title="bbox 23 183 112 204">
 <span class='ocr_line' id='line_1_5' title="bbox 23 183 112 204; baseline 0 -5; x_size 21; x_descenders 5; x_ascenders 4"><span class='ocrx_word' id='word_1_15' title='bbox 23 183 112 204; x_wconf 93'>Inventory</span>
 <span class='ocr_line' id='line_1_30' title="bbox 1082 183 1178 202; baseline 0 -3; x_size 22; x_descenders 5.5; x_ascenders 5.5"><span class='ocrx_word' id='word_1_82' title='bbox 1082 183 1178 202; x_wconf 93'>1,277,838</span>
 <span class='ocr_line' id='line_1_54' title="bbox 1301 183 1379 202; baseline 0 -3; x_size 22; x_descenders 5.5; x_ascenders 5.5"><span class='ocrx_word' id='word_1_107' title='bbox 1301 183 1379 202; x_wconf 95'>953,675</span>

The second result contains the targeted value. You can compare the vertical coordinates of the bbox to be sure to extract the first column.

The command grep was just enough in this example here, but there are certainly other ways to do something similar. Note also, that the regular expression should maybe replaced with some other calculation depending on how skewed your pages are.

Alternatively, you can try out the open source Tabula, which will try to extract tabular data from pdfs.