Need to make a cartoon comic version of a picture with Python and OpenCV

Question

I'm trying to make a function which will make any image look like a cartooney comic strip. Here is my code so far:

import numpy
import cv2

__author__ = "Michael Beyeler"
__license__ = "GNU GPL 3.0 or later"

class Cartoonizer:

    def __init__(self):
        self.numDownSamples = 1
        self.numBilateralFilters = 7

    def render(self, img_rgb):

        # downsample image using Gaussian pyramid
        img_color = img_rgb
        for _ in range(self.numDownSamples):
            img_color = cv2.pyrDown(img_color)
        # repeatedly apply small bilateral filter instead of applying
        # one large filter
        for _ in range(self.numBilateralFilters):
            img_color = cv2.bilateralFilter(img_color, 9, 9, 7)
        # upsample image to original size
        for _ in range(self.numDownSamples):
            img_color = cv2.pyrUp(img_color)
        # convert to grayscale and apply bilateral blur
        img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2GRAY)
        for _ in range(self.numBilateralFilters):
            img_gray_blur = cv2.bilateralFilter(img_gray, 9, 9, 7)
        # detect and enhance edges
        img_edge = cv2.adaptiveThreshold(img_gray_blur, 255,
                                         cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                         cv2.THRESH_BINARY, 9, 5)
        # convert back to color so that it can be bit-ANDed with color image
        img_edge = cv2.cvtColor(img_edge, cv2.COLOR_GRAY2RGB)
        #Ensure that img_color and img_edge are the same size, otherwise bitwise_and will not work
        height = min(len(img_color), len(img_edge))
        width = min(len(img_color[0]), len(img_edge[0]))
        img_color = img_color[0:height, 0:width]
        img_edge = img_edge[0:height, 0:width]
        return cv2.bitwise_and(img_color, img_edge)

I have taken it from here, license preserved, and slightly modified it: http://www.askaswiss.com/2016/01/how-to-create-cartoon-effect-opencv-python.html

What I have originally is this:

Here is what my script outputs:

And here is what I need:

What I've noticed so far is:

1. My code for blurring the image has got too many colors, I need less smooth transition from light colors to dark ones.
2. The target image has got smooth edges, which are lines, when my code creates a lot of noise ("lonely" black dots) and splits lines. I've tried changing some parameters, added a couple of random filters, but I really have no clue what to do next.

Any help is greatly appreciated.

Answer 1

I don't have Python code, it's written in MATLAB (using DIPimage 3). But I think you might get some ideas from it. Here is what it does:

1- s is a slightly smoothed version of the input image img, and will be used for creating the lines. For the smoothing I use a trivial non-linear diffusion. This preserves (even enhances) edges. It is similar to the bilateral filter.

2- Using s, I first apply the Laplacian operator (this one uses Gaussian derivatives, the parameter 1.5 is the sigma for the Gaussian). This is similar to a difference of Gaussians. Your cv2.adaptiveThreshold call does the equivalent to gaussf(img,2)-img. My Laplacian does something similar to gaussf(img,2)-gaussf(img,1) (a difference of Gaussians). That is, there is somewhat less detail in this output than in the one from cv2.adaptiveThreshold.

3- The Laplacian was applied to a color image, so it yields a color output. I convert this to grey-value by taking the max color element. Then I clip and stretch this, essentially doing the other half of what cv2.adaptiveThreshold does, except the output is not binary, but still grey-value. That is, there are darker and lighter lines. More importantly, the lines don't look pixelated because there is a gradual change from dark to light at the edges of each line. I had to tweak these parameters a bit to get a good result. l is now an image that is 1 where there will be no lines, and lower (darker) where there will be lines.

4- Now I apply a path closing to l. This is a rather specialized morphological operator, you might have to do some effort to find an implementation. It removes dark lines in l that are very short. This basically gets rid of the problem you had with the dots. I'm sure there are other ways to solve the dot problem.

5- To put color in between the lines we want to both smooth and quantize the original image. I overwrite s with a more strongly smoothed version of img, to which I apply color quantization using an algorithm I described in another answer. This quantization leaves only 10 distinct colors. I apply a little bit of smoothing to avoid the too-sharp transition between colors.

6- Finally, the color image s and the lines image l are multiplied together. Where l was 1, nothing changes. Where l had lower values, s will become darker. This effectively draws the lines on the image. It's a nicer effect than the bitwise and operator that you use.

img = readim('https://i.stack.imgur.com/Zq1f4.jpg');
% Simplify using non-linear diffusion
s = colordiffusion(img,2);
% Find lines -- the positive response of the Laplace operator
l = laplace(s,1.5);
l = tensorfun('immax',l);
l = stretch(clip(l,0.4,4),0,100,1,0);
% Remove short lines
l = pathopening(l,8,'closing','constrained');
% Simplify color image using diffusion and k-means clustering
s = colordiffusion(gaussf(img),5);
s = quantize(s,10,'minvariance');
s = gaussf(s);
% Paint lines on simplified image
out = s * l;

% Color diffusion:
function out = colordiffusion(out,iterations)
   sigma = 0.8;
   K = 10;
   for ii = 1:iterations
      grey = colorspace(out,'grey');
      nabla_out = gradientvector(grey,sigma);
      D = exp(-(norm(nabla_out)/K)^2);
      out = out + divergence(D * nabla_out);
   end
end