Rearranging License Plate characters based on country

Question

I am doing a License/Number plate recognition project and I'm on the stage of completion but there is a small problem, I have successfully recognized the characters, consider the below example:

This is an input image, I got the prediction as 2791 2g rj14

As you can, the ocr did a great job but the arrangement is destroyed (DESTROYING the whole purpose). Sometimes it does outputs in the correct sequence but sometimes it does not, so when it does not output in the correct sequence I'm trying to develop an algorithm which will take the predicted num_plate string as input and rearrange it on the basis of my country (India).

Below are some images which tell us about the format of Indian Number/License Plate.

Also, I have collected all the states but for right now, I just want to do for only the 3 states which are: Delhi (DL), Haryana (HR), UttarPradesh (UP). More info : https://en.wikipedia.org/wiki/List_of_Regional_Transport_Office_districts_in_India

total_states_list = [
    'AN','AP','AR','AS','BR','CG','CH','DD','DL','DN','GA','GJ','HR','HP','JH','JK','KA','KL',
    'LD','MH','ML','MN','MP','MZ','NL','OD','PB','PY','RJ','SK','TN','TR','TS','UK','UP','WB'
]

district_codes = {
    'DL': ['1','2','3','4','5','6','7','8','9','10','11','12','13'],
    'HR': [01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,
            40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,
            71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99
    ]
}

So, I have been trying but cannot come up with an algorithm which rearranges the sequence in the required sequence if it is not. Any help would be really appreciated.

Details about OCR

Using keras-ocr, I'm getting the following output for the input image:

[
  ('hrlz',  array([[ 68.343796,  42.088367],
                   [196.68803 ,  26.907867],
                   [203.00832 ,  80.343094],
                   [ 74.66408 ,  95.5236  ]], dtype=float32)), 
  ('c1044', array([[ 50.215836, 113.09602 ],
                   [217.72466 ,  92.58473 ],
                   [224.3968  , 147.07387 ],
                   [ 56.887985, 167.58516 ]], dtype=float32))
]

source: https://keras-ocr.readthedocs.io/en/latest/examples/using_pretrained_models.html

Inside the keras_ocr.tools.drawAnnotations they are I think getting the predictions boxes. So I located this file and found the implementation of drawAnnotations function and here it is:

def drawAnnotations(image, predictions, ax=None):
  if ax is None:
        _, ax = plt.subplots()
    ax.imshow(drawBoxes(image=image, boxes=predictions, boxes_format='predictions'))
    predictions = sorted(predictions, key=lambda p: p[1][:, 1].min())
    left = []
    right = []
    for word, box in predictions:
        if box[:, 0].min() < image.shape[1] / 2:
            left.append((word, box))
        else:
            right.append((word, box))
    ax.set_yticks([])
    ax.set_xticks([])
    for side, group in zip(['left', 'right'], [left, right]):
        for index, (text, box) in enumerate(group):
            y = 1 - (index / len(group))
            xy = box[0] / np.array([image.shape[1], image.shape[0]])
            xy[1] = 1 - xy[1]
            ax.annotate(s=text,
                        xy=xy,
                        xytext=(-0.05 if side == 'left' else 1.05, y),
                        xycoords='axes fraction',
                        arrowprops={
                            'arrowstyle': '->',
                            'color': 'r'
                        },
                        color='r',
                        fontsize=14,
                        horizontalalignment='right' if side == 'left' else 'left')
    return ax

How should I go about and get the (x,y,w,h) and then somehow sort/print according to y/x of number_plate bbox?

EDIT - 2

I managed to get the bounding box of characters as you can see in the image below:

using the function cv2.polylines(box), where box are the same coordinates where I have pasted the output earlier. Now how can I print them in a sequence like, left to right... using the y/x as suggested by people in the comments.

Answer 1

If you can get the coordinates of each identified text box, then:

• Rotate the coordinates so the boxes are parallel with the X-axis
• Scale the Y-coordinates so they can be rounded to integers, so that boxes that are side-by-side will get the same integer Y-coordinate (like a line number)
• Sort the data by Y, then X coordinate
• Extract the texts in that order

Here is an example of such sequence:

data = [
  ('hrlz', [[ 68.343796,  42.088367],
            [196.68803 ,  26.907867],
            [203.00832 ,  80.343094],
            [ 74.66408 ,  95.5236  ]]), 
  ('c1044',[[ 50.215836, 113.09602 ],
            [217.72466 ,  92.58473 ],
            [224.3968  , 147.07387 ],
            [ 56.887985, 167.58516 ]])
]

# rotate data to align with X-axis
a, b = data[0][1][:2]
dist = ((b[1] - a[1]) ** 2 + (b[0] - a[0]) ** 2) ** 0.5
sin = (b[1] - a[1]) / dist
cos = (b[0] - a[0]) / dist
data = [
    (text, [(x * cos + y * sin, y * cos - x * sin) for x, y in box]) for text, box in data
]

# scale Y coordinate to integers
a, b = data[0][1][1:3]
height = b[1] - a[1]
data = [
    (round(box[0][1] / height), box[0][0], text) 
        for text, box in data
]

# sort by Y, then X
data.sort()

# Get text in the right order
print("".join(text for _, _, text in data))

This assumes that the points of the boxes are given in the following clockwise order:

top-left, top-right, bottom-right, bottom-left