get a picture of the frame of a video at every second [closed]

Question

Youtube has this cool feature where you can 'scrub' through the video and they give you the thumbnail view of the video at each second:

Does anybody know how this is achieved? Does the server have to send over each image individually, or can they be extracted from the video on the clients machine with javascript?

Regarding the implementation of the "extraction", I am a bit lost, but I imagine a canvas element could be used to draw the frames once their extracted.

If the server has to be used: since the images are small, I imagine I could use sprites, but if we're talking about every second of video, then that file could get pretty big. (still small in comparison to the video data I suppose, but not exactly optimal)

Answer 1

tl;dr server generated sprite sheets, downloaded as needed.

We can guess going in that the images are generated server side. YouTube does much more intense processing on every video than pulling a few thumbnails. Also, a quick Google suggests this feature has been around for a few years - probably longer than the required HTML5/JS/browser-horsepower to do this client-side.

I hit up Download Tools > Resources in my browser and checked out a newly-posted video from my feed. Interestingly, there were no previews yet (the video was only up for about 20 min when I checked). This indicates the images are probably generated server-side and just haven't finished processing.

Checking an older video and looking at Resources > Images didn't reveal anything interesting. So I switched to Timelines and hit record, then started mousing over the timeline and watching for network traffic. As I moved the mouse, *.jpg files started loading, and they contained 25 thumbnails from a given section of video:

I also noticed that the initial file M0.jpg is the same size image, but contains about 100 thumbnails from across the entire video, rather than 25 thumbnails from one segment. Example:

Testing again with a new video, it looks like the 100-image M0.jpg gets downloaded first and provides the basic lower-res less-granular thumbnail previews. Then, as you mouseover different sections of video, the higher-res M0.jpg, M1.jpg, etc. get downloaded as needed.

Interestingly, this doesn't change for longer videos, which explains why the thumbnails can sometimes suck. If your connection or YouTube is too slow at getting the higher-res thumbnails, then you get stuck with only 100 low-res thumbnails of a really long video. Not sure how this works on shorter videos. Also, it might be interesting to see what distribution they pull the thumbnails from (is it just linear every 1/100th of the video? or something else).

Last tidbit, is I noticed if you use a url with a timecode in it, you don't get a full 100-image M0.jpg sheet but rather a completely different size M#.jpg containing about 25 low-res thumbnails from the timecode to the end of the video.

I guess they're assuming that when people link to a specific timecode, users aren't likely to jump to an earlier point in the video. Also, this is way less granularity than the ~75 images you would've gotten by sending the normal 100-image M0.jpg. On the other hand, it's also about 30% of the size, so maybe speed was that important.

As for generating the thumbnails, ffmpeg is a good way to go:

To make multiple screenshots and place them into a single image file (creating tiles), you can use FFmpeg's tile video filter, like this:

ffmpeg -ss 00:00:10 -i movie.avi -frames 1 -vf "select=not(mod(n\,1000)),scale=320:240,tile=2x3" out.png

That will seek 10 seconds into the movie, select every 1000th frame, scale it to 320x240 pixels and create 2x3 tiles in the output image out.png, which will look like this: