Give a video rounded transparent edges so that it can be overlayed on another video using FFMPEG

Question

Im trying to overlay a smaller video (200x200)on top of a bigger video (800x800).

I've used the FFMPEG overlay filter to achieve this

    ffmpeg -i big.mp4 -vf "movie=small.mkv[clip2]; [in][clip2] overlay=1:5 [out]"  final.mp4

Challenge is that the smaller video needs its edges to be rounded. I have tried working with alphaextract and alphamerge. The documentation on FFMPEG is sparse and im not sure how to go about it.

Answer 1

Although I think the existing answer is good for a circle, it sounds like you wanted more of a “rounded rectangle” instead. Here is how I have done that:

[1]format=yuva420p,geq=lum='p(X,Y)':a='if(gt(abs(W/2-X),W/2-10)*gt(abs(H/2-Y),H/2-10),if(lte(hypot(10-(W/2-abs(W/2-X)),10-(H/2-abs(H/2-Y))),10),255,0),255)'[rounded];

About format:

The alpha values can’t be changed in a source that doesn’t have an existing alpha channel. So the format=yuva420p filter adds one (that is the a in yuva). From what I have seen, yuva444p is also common (used in other answers), but other formats like rgba ought to work, provided they have an alpha channel. My original source was yuv420p so I made mine yuva420p.

About geq:

It seems you can’t change only the alpha channel (if you try, you’ll get an error “A luminance or RGB expression is mandatory”), so the lum='p(X,Y)' is basically a dummy no-op to allow you to change the alpha without changing anything else.

So that it’s not necessary to check each corner individually, X and Y are converted into the upper left quadrant by the formula:

X' = W/2-abs(W/2-X)
Y' = H/2-abs(H/2-Y)

The first part of the formula checks whether X' and Y' are within the thatched area:

That is, whether:

    X' < 10
and Y' < 10

If they are not, a is set to 255 (ie, show the pixel).

If they fall within this area, the distance is calculated between the point (X,Y) and the centre of the circle at (10,10). If the point (X,Y) falls within the circle, a is set to 255; otherwise, a is set to 0, and that pixel is hidden.

The radius can be changed to give more or less ‘rounding’. A full example command with a radius of 20 is:

ffmpeg -f lavfi -i color=darkblue:size=800x600 -f lavfi -i color=gray:size=600x450 -frames:v 180 -filter_complex "[1]format=yuva420p,geq=lum='p(X,Y)':a='if(gt(abs(W/2-X),W/2-20)*gt(abs(H/2-Y),H/2-20),if(lte(hypot(20-(W/2-abs(W/2-X)),20-(H/2-abs(H/2-Y))),20),255,0),255)'[rounded];[0][rounded]overlay=x=(W-w)/2:y=(H-h)/2" example.mp4

Giving the result:

In my examples I have used 10 or 20 as the radius of the circle, but you can change this to any number to make the corners more or less rounded as you like.

Answer 2

Main movie with circular shaped overlay (2nd movie)

ffmpeg \
-i "trailer_iphone.m4v" \
-i "sintel_trailer-480p.mp4" \
-filter_complex "\
[1]format=yuva444p,geq=lum='p(X,Y)':a='st(1,pow(min(W/2,H/2),2))+st(3,pow(X-(W/2),2)+pow(Y-(H/2),2));if(lte(ld(3),ld(1)),255,0)'[circular shaped video];\
[circular shaped video]scale=w=-1:h=200[circular shaped video small];\
[0][circular shaped video small]overlay" \
-filter_complex_threads 1 \
-map 0:a \
-metadata:s:a:0 title="Sound main movie" \
-disposition:a:0 default \
-map 1:a \
-metadata:s:a:1 title="Sound overlayed movie" \
-disposition:a:1 none \
-c:v libx264 \
-preset ultrafast \
-shortest \
"result.mp4"

Explanation:

Line 5: To create a circular mask we use the "geq" filter. This is used to create circular shaped video with alpha channel from 2nd video input

Line 6: scale height of circular shaped video to 200px, preserve aspect ratio

Line 7: overlay video

Line 8: Gyan came up with this solution for the geq filter here.

You can skip Lines 9 - 14. Then ffmpeg will default to use the Audio track of the main movie.

Line 9: optional: Audio of the 1st input will be 1st Audio track in the output

Line 10: optional: give 1st Audio track in the output a name (works in VLC)

Line 11: optional: make sure that 1st Audio track in the output will be played as the default track

Line 12: optional: Audio of the 2nd input will be 2nd Audio track in the output

Line 13: optional: give 2nd Audio track in the output a name (works in VLC)

Line 14: optional: make sure that 2nd Audio track in the output will not be played unless you choose it with the player (VLC)

Line 17: duration of the output will be as long as the shortest input