Apply heatmap on video with OpenCV and Python

Question

I want to write a code that will apply heatmap on the video on places where there is a movement. I wrote a code that detects movement, with contours but I do not know how to make heat map.

This is the code that I have

import cv2
import numpy as np

# upload video
cap = cv2.VideoCapture('test_video.mp4')

#reading two frames
ret, frame1 = cap.read()
ret, frame2 = cap.read()

while cap.isOpened():

     # get diference between two frames
     diff = cv2.absdiff(frame1, frame2)

     # convert diference in gray
     gray = cv2.cvtColor(diff, cv2.COLOR_BGR2GRAY)

     # bluring and treshold
     blur = cv2.GaussianBlur(gray, (5,5), 0)
     _, thresh = cv2.threshold(blur, 20, 255, cv2.THRESH_BINARY)

     dilated = cv2.dilate(thresh, None, iterations = 3)

     # define contours
     contours, _ = cv2.findContours(dilated, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
     
     # draw contours
     cv2.drawContours(frame1, contours, -1, (255,0,0), 1)

     # show frames
     cv2.imshow('frame', frame1)
     frame1 = frame2
     ret, frame2 = cap.read()

     if cv2.waitKey(60) == 60:
          break

cv2.destroyAllWindows()
cap.release()

I have seen this link: Build a Motion Heatmap Video Using OpenCV With Python. And I wanted to reproduce the code, but a lot of stuff such as fourcc, image_folder, and images is not defined, so I tried to do it the other way.

Can you help me with this? Basically, I want to apply the heatmap to the video where there is a movement.

Answer 1

Here is an idea. You know how you loop through the frames of a video using opencv, right? Well, for each frame the while loop is at, have the frame that would come after the current frame stored in a variable, and compare the difference between the current frame and the future frame.

With the difference between the two frames, you can detect the contours of the movement. Let's say we draw the contours on the image with green.

Have a blank array defined before the while loop to be the heat map; every iteration of the loop, add a certain amount to every coordinate of the heat map where the coordinate is green on the frame, and remove a certain amount from the heat map where the coordinate is not green on the image.

1. Import the necessary modules:

import cv2
import numpy as np

2. Define a function to process the difference between two frames into something that will allow better detection of contours for our case. The values can be tweaked according to your needs:

def process(img):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray, (5, 5), 25)
    img_canny = cv2.Canny(img_blur, 5, 50)
    kernel = np.ones((3, 3))
    img_dilate = cv2.dilate(img_canny, kernel, iterations=4)
    img_erode = cv2.erode(img_dilate, kernel, iterations=1)
    return img_erode

3. Define a function that will take in a processed image of the difference between two frames, and the image you want the contours to be drawn on. It will return the image with the contours (green) drawn on it:

def get_contours(img, img_original):
    img_contours = img_original.copy()
    contours, hierarchies = cv2.findContours(img, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    cv2.drawContours(img_contours, contours, -1, (0, 255, 0), -1) 
    return img_contours

4. Define your capture device, and read two frames before the while loop; the first one to be the present frame of each iteration of the while loop, and the second to be the future frame of each iteration of the while loop. Also, define the blank image for the heat map, heat_map:

cap = cv2.VideoCapture("Bar fight security cam.mp4")

success, img1 = cap.read()
success, img2 = cap.read()
heat_map = np.zeros(img1.shape[:-1])

5. In the while loop, find the difference between the two frames and get the current frame of each loop with the contours from the difference drawn on it:

while success:
    diff = cv2.absdiff(img1, img2)
    img_contours = get_contours(process(diff), img1)

6. Add a number, say 3, to every coordinate of the heat_map where it is green on the image returned by the get_contours function defined earlier, and subtract 3 from every coordinate of the heat_map where it is not green on the image. To make sure that no colors result in channel values less than 0 and greater than 255, apply the boundaries to the heat map:

    heat_map[np.all(img_contours == [0, 255, 0], 2)] += 3
    heat_map[np.any(img_contours != [0, 255, 0], 2)] -= 3
    heat_map[heat_map < 0] = 0
    heat_map[heat_map > 255] = 255

7. Convert the heat_map array to a grayscale image, and then to a heat ma image:

    img_mapped = cv2.applyColorMap(heat_map.astype('uint8'), cv2.COLORMAP_JET)

8. Finally, show the frames and retrieve the frames for the next iteration:

    cv2.imshow("Original", img1)
    cv2.imshow("Heat Map", img_mapped)
    
    img1 = img2
    success, img2 = cap.read()
    
    if cv2.waitKey(1) == ord('q'):
        break

Altogether:

import cv2
import numpy as np

def process(img):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray, (5, 5), 25)
    img_canny = cv2.Canny(img_blur, 5, 50)
    kernel = np.ones((3, 3))
    img_dilate = cv2.dilate(img_canny, kernel, iterations=4)
    img_erode = cv2.erode(img_dilate, kernel, iterations=1)
    return img_erode

def get_contours(img, img_original):
    img_contours = img_original.copy()
    contours, hierarchies = cv2.findContours(img, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

    cv2.drawContours(img_contours, contours, -1, (0, 255, 0), -1) 
    # If you want to omit smaller contours, loop through the detected contours, and only draw them on the image if they are at least a specific area. Don't forget to remove the line above if you choose the below block of code.
    # for cnt in contours: 
    #     if cv2.contourArea(cnt) > 500:
    #         cv2.drawContours(img_contours, [cnt], -1, (0, 255, 0), -1) 

    return img_contours

cap = cv2.VideoCapture("Bar fight security cam.mp4")

success, img1 = cap.read()
success, img2 = cap.read()
heat_map = np.zeros(img1.shape[:-1])

while success:
    diff = cv2.absdiff(img1, img2)
    img_contours = get_contours(process(diff), img1)

    heat_map[np.all(img_contours == [0, 255, 0], 2)] += 3 # The 3 can be tweaked depending on how fast you want the colors to respond
    heat_map[np.any(img_contours != [0, 255, 0], 2)] -= 3
    heat_map[heat_map < 0] = 0
    heat_map[heat_map > 255] = 255

    img_mapped = cv2.applyColorMap(heat_map.astype('uint8'), cv2.COLORMAP_JET)

#    img1[heat_map > 160] = img_mapped[heat_map > 160] Use this line to draw the heat map on the original video at a specific temperature range. For this it's where ever the temperature is above 160 (min is 0 and max is 255)

    cv2.imshow("Original", img1)
    cv2.imshow("Heat Map", img_mapped)
    
    img1 = img2
    success, img2 = cap.read()
    
    if cv2.waitKey(1) == ord('q'):
        break