How can I isolate and recolor specific color range?

Question

Given an image of the region containing the lips and other "noise" (teeth, skin), how can we isolate and recolor only the lips (simulating a "lipstick" effect)? Attached is a photo describing the lips/mouth states.

What we have tried so far is a three-part process:

1. Color matching the lips using a stable point on the lips (provided by internal API).
2. Use this color as the base color for the lips isolation.
3. Recolor the lips (lipstick behavior)

We tried a few algorithms like hue difference, HSV difference, ∆and E after converting them to CIE color space. Unfortunately, nothing has panned out or has produced artifacts due to the skin's relative similarity in color to the lips and the discoloration from shadows cast by the nose and mouth.

What are we missing? Is there a better way to approach it?

We are looking for a solution/direction from a classic Computer Vision color algorithm, not a solution from the Machine Learning/Depp Learning domain. Thanks!

Answer 1

You probably won't like this answer, but your question is ill-posed (there is no measurable solution that is better than others, there are only peoples' opinions.)

In this case, the best answer you can hope for then is usually:

Ask an expert for a large set of examples that would be acceptable in practice.

Your problem can easily be solved by an appropriate artist (who you trust will produce usable results) with access to the right tools (for example photoshop,) but a single artist (or even a group of them) can't possibly scale to millions (or whatever large number you care about) of examples.

To address the short-coming of the artist-based solution, you can use the following strategy:

• Collect a sufficiently large set of before and after images created by artists, who you deem trustworthy.
• Apply your favorite machine learning algorithm to learn a mapping from the before to the after images. There are many possible choices, and it almost really doesn't matter which you choose as long as you know how to use it well.

Note, the above two steps are usually not one-and-done, as most algorithms are. Usually, you will come across pathological or not-well behaved examples to your ML solution above in using the product. The key is to collect these examples, pass them through the artist and retrain or update your ML model. Repeat this enough times and you will produce a state-of-the-art solution to your problem.

Whether you have the funding, time, motivation and resources to accomplish this is another matter.