Segmenting License Plate Characters

Question

I am facing a problem in segmenting characters from a license plate image. I have applied following method to extract license plate characters"

1. Adaptive threshold the license plate image.
2. Select contours which having particular aspect ratio.

If there is any shade in the license plate image as in attached file, I am not able to properly segment the characters due to improper binarization. The shade in the image merges adjacent characters in the image.

I have thresholded the images with different window sizes. The results are attached. How can I segment characters from image if there is shade in the image? I am using OpenCV.

I have used following function in OpenCV to threshold my license plate image:

cvAdaptiveThreshold(licensePlateImg, threshImg, 255, CV_ADAPTIVE_THRESH_GAUSSIAN_C, CV_THRESH_BINARY_INV, wind);

I have tried with different window sizes (wind) and different adaptiveMethod (ADAPTIVE_THRESH_MEAN_C and ADAPTIVE_THRESH_GAUSSIAN_C) to get the thresholded images.

Answer 1

Before I start, I know you are seeking an implementation of this algorithm in OpenCV C++, but my algorithm requires the FFT and the numpy / scipy packages are awesome for that. As such, I will give you an implementation of the algorithm in OpenCV using Python instead. The code is actually quite similar to the C++ API that you can easily transcribe that over instead. That way, it minimizes the amount of time it will take for me to learn (or rather relearn...) the API and I would rather give you the algorithm and the steps I did to perform this task to not waste any time at all.

As such, I will give you a general overview of what I would do. I will then show you Python code that uses numpy, scipy and the OpenCV packages. As a bonus for those who use MATLAB, I will show you the MATLAB equivalent, with MATLAB code to boot!

---

What you can do is try to use homomorphic filtering. In basic terms, we can represent an image in terms of a product of illumination and reflectance. Illumination is assumed to be slowly varying and the main contributor of dynamic range. This is essentially low frequency content. Reflectance represents details of objects and assumed to vary rapidly. This is also the primary contributor to local contrast and is essentially high frequency content.

The image can be represented as a product of these two. Homomorphic filtering tries and splits up these components and we filter them individually. We then combine the results together when we are finished. As this is a multiplicative model, it's customary to use a log operation so that we can express the product as a sum of two terms. These two terms are filtered individually, scaled to emphasize or de-emphasize their contributions to the image, summed, then the anti-log is taken.

The shading is due to the illumination, and so what we can do is decrease the contribution that this shading does over the image. We can also boost the reflectance so we can get some better edges as edges are associated with high frequency information.

We usually filter the illumination using a low-pass filter, while the reflectance with a high-pass filter. In this case, I'm going to choose a Gaussian kernel with a sigma of 10 as the low-pass filter. A high-pass filter can be obtained by taking 1 and subtracting with the low-pass filter. I transform the image into the log domain, then filter the image in the frequency domain using the low-pass and high-pass filters. I then scale the low pass and high pass results, add these components back, then take the anti-log. This image is now better suited to be thresholded as the image has low variation.

What I do as additional post-processing is that I threshold the image. The letters are darker than the overall background, so any pixels that are lower than a certain threshold would be classified as text. I chose the threshold to be intensity 65. After this, I also clear off any pixels that are touching the border, then remove any areas of the image that have less than 160 (MATLAB) or 120 (Python) pixels of total area. I also crop out some of the columns of the image as they are not needed for our analysis.

---

Here are a couple of caveats for you:

Caveat #1 - Removing borders

Removing any pixels that touch the border is not built into OpenCV. However, MATLAB has an equivalent called imclearborder. I'll use this in my MATLAB code, but for OpenCV, this was the following algorithm:

• Find all of the contours in the image
• For each contour that is in the image, check to see if any of the contour pixels are within the border of the image
• If any are, mark this contour for removal
• For each contour we want to remove, simply draw this whole contour in black

I created a method called imclearborder(imgBW, radius) in my code, where radius is how many pixels within the border you want to clear stuff up.

Caveat #2 - Removing pixel areas below a certain area

Removing any areas where they are less than a certain amount is also not implemented in OpenCV. In MATLAB, this is conveniently given using bwareaopen. The basic algorithm for this is:

• Find all of the contours in the image
• Analyze how much each contour's area fills up if you were to fill in the interior
• Any areas that are less than a certain amount, clear this contour by filling the interior with black

I created a method called bwareaopen(imgBW) that does this for us.

Caveat #3 - Area parameter for removing pixel areas

For the Python code, I had to play around with this parameter and I settled for 120. 160 was used for MATLAB. For python, 120 got rid of some of the characters, which is not desired. I'm guessing my implementation of bwareaopen in comparison to MATLAB's is different, which is probably why I'm getting different results.

---

Without further ado, here's the code. Take note that I did not use spatial filtering. You could use filter2D in OpenCV and convolve this image with the Gaussian kernel, but I did not do that as Homomorphic Filtering when using low-pass and high-pass filters are traditionally done in the frequency domain. You could explore this using spatial filtering, but you would also have to know the size of your kernels before hand. With frequency domain filtering, you just need to know the standard deviation of the filter, and that's just one parameter in comparison to two.

Also, for the Python code, I downloaded your image on to my computer and ran the script. For MATLAB, you can directly reference the hyperlink to the image when reading it in with the Image Processing toolbox.

---

Python code

import cv2 # For OpenCV modules (For Image I/O and Contour Finding)
import numpy as np # For general purpose array manipulation
import scipy.fftpack # For FFT2 

#### imclearborder definition

def imclearborder(imgBW, radius):

    # Given a black and white image, first find all of its contours
    imgBWcopy = imgBW.copy()
    contours,hierarchy = cv2.findContours(imgBWcopy.copy(), cv2.RETR_LIST, 
        cv2.CHAIN_APPROX_SIMPLE)

    # Get dimensions of image
    imgRows = imgBW.shape[0]
    imgCols = imgBW.shape[1]    

    contourList = [] # ID list of contours that touch the border

    # For each contour...
    for idx in np.arange(len(contours)):
        # Get the i'th contour
        cnt = contours[idx]

        # Look at each point in the contour
        for pt in cnt:
            rowCnt = pt[0][1]
            colCnt = pt[0][0]

            # If this is within the radius of the border
            # this contour goes bye bye!
            check1 = (rowCnt >= 0 and rowCnt < radius) or (rowCnt >= imgRows-1-radius and rowCnt < imgRows)
            check2 = (colCnt >= 0 and colCnt < radius) or (colCnt >= imgCols-1-radius and colCnt < imgCols)

            if check1 or check2:
                contourList.append(idx)
                break

    for idx in contourList:
        cv2.drawContours(imgBWcopy, contours, idx, (0,0,0), -1)

    return imgBWcopy

#### bwareaopen definition
def bwareaopen(imgBW, areaPixels):
    # Given a black and white image, first find all of its contours
    imgBWcopy = imgBW.copy()
    contours,hierarchy = cv2.findContours(imgBWcopy.copy(), cv2.RETR_LIST, 
        cv2.CHAIN_APPROX_SIMPLE)

    # For each contour, determine its total occupying area
    for idx in np.arange(len(contours)):
        area = cv2.contourArea(contours[idx])
        if (area >= 0 and area <= areaPixels):
            cv2.drawContours(imgBWcopy, contours, idx, (0,0,0), -1)

    return imgBWcopy

#### Main program

# Read in image
img = cv2.imread('5DnwY.jpg', 0)

# Number of rows and columns
rows = img.shape[0]
cols = img.shape[1]

# Remove some columns from the beginning and end
img = img[:, 59:cols-20]

# Number of rows and columns
rows = img.shape[0]
cols = img.shape[1]

# Convert image to 0 to 1, then do log(1 + I)
imgLog = np.log1p(np.array(img, dtype="float") / 255)

# Create Gaussian mask of sigma = 10
M = 2*rows + 1
N = 2*cols + 1
sigma = 10
(X,Y) = np.meshgrid(np.linspace(0,N-1,N), np.linspace(0,M-1,M))
centerX = np.ceil(N/2)
centerY = np.ceil(M/2)
gaussianNumerator = (X - centerX)**2 + (Y - centerY)**2

# Low pass and high pass filters
Hlow = np.exp(-gaussianNumerator / (2*sigma*sigma))
Hhigh = 1 - Hlow

# Move origin of filters so that it's at the top left corner to
# match with the input image
HlowShift = scipy.fftpack.ifftshift(Hlow.copy())
HhighShift = scipy.fftpack.ifftshift(Hhigh.copy())

# Filter the image and crop
If = scipy.fftpack.fft2(imgLog.copy(), (M,N))
Ioutlow = scipy.real(scipy.fftpack.ifft2(If.copy() * HlowShift, (M,N)))
Iouthigh = scipy.real(scipy.fftpack.ifft2(If.copy() * HhighShift, (M,N)))

# Set scaling factors and add
gamma1 = 0.3
gamma2 = 1.5
Iout = gamma1*Ioutlow[0:rows,0:cols] + gamma2*Iouthigh[0:rows,0:cols]

# Anti-log then rescale to [0,1]
Ihmf = np.expm1(Iout)
Ihmf = (Ihmf - np.min(Ihmf)) / (np.max(Ihmf) - np.min(Ihmf))
Ihmf2 = np.array(255*Ihmf, dtype="uint8")

# Threshold the image - Anything below intensity 65 gets set to white
Ithresh = Ihmf2 < 65
Ithresh = 255*Ithresh.astype("uint8")

# Clear off the border.  Choose a border radius of 5 pixels
Iclear = imclearborder(Ithresh, 5)

# Eliminate regions that have areas below 120 pixels
Iopen = bwareaopen(Iclear, 120)

# Show all images
cv2.imshow('Original Image', img)
cv2.imshow('Homomorphic Filtered Result', Ihmf2)
cv2.imshow('Thresholded Result', Ithresh)
cv2.imshow('Opened Result', Iopen)
cv2.waitKey(0)
cv2.destroyAllWindows()

---

MATLAB code

clear all;
close all;

% Read in image
I = imread('http://i.stack.imgur.com/5DnwY.jpg');

% Remove some columns from the beginning and end
I = I(:,60:end-20);

% Cast to double and do log.  We add with 1 to avoid log(0) error.
I = im2double(I);
I = log(1 + I);

% Create Gaussian mask in frequency domain
% We must specify our mask to be twice the size of the image to avoid
% aliasing.
M = 2*size(I,1) + 1;
N = 2*size(I,2) + 1;
sigma = 10;
[X, Y] = meshgrid(1:N,1:M);
centerX = ceil(N/2);
centerY = ceil(M/2);
gaussianNumerator = (X - centerX).^2 + (Y - centerY).^2;

% Low pass and high pass filters
Hlow = exp(-gaussianNumerator./(2*sigma.^2));
Hhigh = 1 - Hlow;

% Move origin of filters so that it's at the top left corner to match with
% input image
Hlow = ifftshift(Hlow);
Hhigh = ifftshift(Hhigh);

% Filter the image, and crop
If = fft2(I, M, N);
Ioutlow = real(ifft2(Hlow .* If));
Iouthigh = real(ifft2(Hhigh .* If));

% Set scaling factors then add
gamma1 = 0.3;
gamma2 = 1.5;
Iout = gamma1*Ioutlow(1:size(I,1),1:size(I,2)) + ...
       gamma2*Iouthigh(1:size(I,1),1:size(I,2));

% Anti-log then rescale to [0,1]
Ihmf = exp(Iout) - 1;
Ihmf = (Ihmf - min(Ihmf(:))) / (max(Ihmf(:)) - min(Ihmf(:)));

% Threshold the image - Anything below intensity 65 gets set to white
Ithresh = Ihmf < 65/255;

% Remove border pixels
Iclear = imclearborder(Ithresh, 8);

% Eliminate regions that have areas below 160 pixels
Iopen = bwareaopen(Iclear, 160);

% Show all of the results
figure;
subplot(4,1,1);
imshow(I);
title('Original Image');
subplot(4,1,2);
imshow(Ihmf);
title('Homomorphic Filtered Result');
subplot(4,1,3);
imshow(Ithresh);
title('Thresholded Result');
subplot(4,1,4);
imshow(Iopen);
title('Opened Result');

---

This is the result I get:

Python

Take note that I re-arranged the windows so that they're aligned in a single column.

MATLAB