FFmpeg: Combine video files with different start times

Question

I have two webm files with audio and video recordings of a video conference session. Both files only contain one side of the conversation. They are not of the same length (someone has joined before the other one), but I have the unix timestamp in milliseconds of the start time of each video file.

On a timeline they look like this:

webm 1:   -----------------------------------------------
webm 2:                 -----------------------------

or like this:

webm 1:   -----------------------------------------------
webm 2:                          -----------------------------

I would like to combine these two video files into one file so that:

1. They appear next to each other (using the hstack option), and
2. That they are mixed with taking the time stamps of the start times into account. The final video should then look like this:

Target result : --------------===========================----

The beginning and the end of the new video would show a black placeholder for the video file that has no data at this time of the mixed stream.

At the moment I use this command:

ffmpeg -i 1463408731413703.webm -i 1463408880317860.webm -filter_complex \
"[0:v][1:v]hstack=inputs=2[v]; \
 [0:a][1:a]amerge[a]" \
-map "[v]" -map "[a]" -ac 2  -c:v libvpx output.webm

This creates a video like this:

Not good result: =====================------------------

i.e. the conversation is out of sync.

How can I combine two video streams with different length and different start times using ffmpeg so that I will end up with "Target result" above?

Thanks a lot!

Answer 1

This command below presents the case for 3 inputs. First input is the longest and remains on the left. The 2nd input starts at t=1 second and lasts for 3 seconds, the 3rd lasts for 2 seconds starting at t=4s. They show on the right.

ffmpeg -i 1.mp4 -i 2.mp4 -i 3.mp4 -filter_complex \
       "[0]pad=2*iw:ih[l];[1]setpts=PTS-STARTPTS+1/TB[1v]; [l][1v]overlay=x=W/2[a]; \
        [2]setpts=PTS-STARTPTS+4/TB[2v]; [a][2v]overlay=x=W/2[v]; \
        [1]adelay=1000|1000[1a]; [2]adelay=4000|4000[2a]; [0][1a][2a]amix=inputs=3[a]" \
       -map "[v]" -map "[a]" out.mp4

The adelay filters are for stereo input streams.