How to expand 2d array (30x20) to 36x60, which is contained in 620x480 2d array?

Question

The start data: 2d array (620x480) is contained image, where shows human face, and 2d array (30x20) which is contained eye image. Face image includes eye image.

How I can expand eye image to 36x60 to include pixels from face image? Are there ready-made solutions?

Another similar task: the eye image have 37x27 size. How I can expand eye image to target(closest to 36x60) size, e.g. 39x65 i.e maintain the aspect ratio required before resizing and then resize to 36x60.

Code for testing (project is available by reference):

import dlib
import cv2 as cv
from imutils.face_utils import shape_to_np
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor('res/model.dat')


frame = cv.imread('photo.jpg')
gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)

img = frame.copy()
dets = detector(gray, 0)

for i, det in enumerate(dets):
    shape = shape_to_np(predictor(gray, det))
    shape_left_eye = shape[36:42]

    x, y, h, w = cv.boundingRect(shape_left_eye)
    cv.rectangle(img, (x, y), (x + h, y + w), (0, 255, 0), 1)
    cv.imwrite('file.png', frame[y: y+w, x: x+h])

The image 42x13:

Answer 1

For the first part you can use cv2.matchTemplate to find the eye region in the face and then according to the size you want you can enlarge it. You can read more about it here.

FACE IMAGE USED

EYE IMAGE USED

The size of eye I have (12, 32).

face = cv2.imread('face.jpg', 0)
eye = cv2.imread('eye.jpg', 0)
w, h = eye.shape[::-1]
res = cv2.matchTemplate(face,eye,cv2.TM_CCOEFF)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
top_left = max_loc
bottom_right = (top_left[0] + w, top_left[1] + h)
cv2.rectangle(face ,top_left, bottom_right, 255, 2)
cv2.imshow('image', face)
cv2.waitKey(0)
cv2.destroyAllWindows()

The result with this code is:

Now I have the top left and bottom right co-ordinates of the eye that is matched where top_left = (112, 108) and bottom_right = (144, 120). Now to expand them to dimensions of 36x60 I simply subtract the required values from top_left and add the required values in bottom_right.

EDIT 1

The question has been edited which suggests that dlib has been used along with a model trained to perform left eye detection. Using the same code I obtained

After that as proposed above I find top_left = (x,y) and bottom_right = (x+w, y+h).

Now if the eye size is smaller 36x60 then we just have to take the area around it to expand it to 36x60 otherwise we have to expand it as such that the aspect ratio is not disturbed and then resized and it cannot be hard coded. The full code used is:

import dlib
from imutils.face_utils import shape_to_np
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor('res/model.dat')

face = cv2.imread('face.jpg', 0)
img = face.copy()
dets = detector(img, 0)
for i, det in enumerate(dets):
    shape = shape_to_np(predictor(img, det))
    shape_left_eye = shape[36:42]
    x, y, w, h = cv2.boundingRect(shape_left_eye)
    cv2.rectangle(face, (x, y), (x + w, y + h), (255, 255, 255), 1)
top_left = (x, y)
bottom_right = (x + w, y + h)
if w <= 36 and h <= 60:
    x = int((36 - w)/2)
    y = int((60 - h)/2) 
else:
    x1 = w - 36
    y1 = h - 60
    if x1 > y1:
        x = int((w % 3)/2)
        req = (w+x) * 5 / 3
        y = int((req - h)/2)
    else:
        y = int((h % 5)/2)
        req = (y+h) * 3 / 5
        x = int((req - w)/2)
top_left = (top_left[0] - x, top_left[1] - y)
bottom_right = (bottom_right[0] + x, bottom_right[1] + y)        
extracted = face[top_left[1]:bottom_right[1], top_left[0]:bottom_right[0]]
result = cv2.resize(extracted, (36, 60),  interpolation = cv2.INTER_LINEAR)
cv2.imshow('image', face)
cv2.imshow('imag', result)
cv2.waitKey(0)
cv2.destroyAllWindows()

Which gives us a 36x60 region of the eye:

This takes care of the case when size of eye is smaller than 36x60. For the second case when the size of eye is larger than 36x60 region I used face = cv2.resize(face, None, fx=4, fy=4, interpolation = cv2.INTER_CUBIC). The result was:

The size of eye detected is (95, 33) and the extracted region is (97, 159) which is very close to the aspect ration of 3:5 before resizing which also satisfies that second task.