How to read a video frame by frame?

Question

i would like to read a Mp4 file in java8-64bit frame by frame and write each frame as a jpg to my harddisk. my first attempt was to use JavaFX 2.2 media player to play the file on a View component. i thought maybe there would be an option to register an observer to get an event each time a new frame was loaded and ready to be painted on the component surface but seems there is no such method. it would be enough to grab just those frames/pixels that got painted on the component.

Can this be done by using the media player? the reason why i use the media player is bcs it was the simplest solution i got workin. i tryed vlcj, just 32bit, and gstreamer but without luck :(

what i got so far:

public class VideoGrabber extends extends JFrame { 

// code for scene setup omitted

final MediaView view = createMediaView(...)

// some other stuff happens here

// now start the video
view.getMediaPlayer().seek(Duration.ZERO);
view.getMediaPlayer().play();
view.getMediaPlayer().setOnEndOfMedia(new Runnable()
{ // save image when done
  BufferedImage img = new BufferedImage(getWidth(), getHeight(), BufferedImage.TYPE_INT_BGR 
  view.paint(img.getGraphics());
  ImageIO.write(img, "JPEG", new File("pic-"+System.currentTimeMillis()+".jpg"));
});

// somewhere else to create 
private MediaView createMediaView(String url)
{
  final Media clip = new Media(url);
  final MediaPlayer player = new MediaPlayer(clip);
  final MediaView view = new MediaView(player);
  view.setFitWidth(VID_WIDTH);
  view.setFitHeight(VID_HEIGHT);
  return view;
}

is there somehow a way to do the following:

player.setOnNextFrameReady(final Event evt) { writeImage(evt.getFrame()) };

Thanks!

Answer 1

You can use the snapshot() of MediaView. First connect a mediaPlayer to a MediaView component, then use mediaPlayer.seek() to seek the video position. And then you can use the following code to extract the image frame:

    int width = mediaPlayer.getMedia().getWidth();
    int height = mediaPlayer.getMedia().getHeight();
    WritableImage wim = new WritableImage(width, height);
    MediaView mv = new MediaView();
    mv.setFitWidth(width);
    mv.setFitHeight(height);
    mv.setMediaPlayer(mediaPlayer);
    mv.snapshot(null, wim);
    try {
        ImageIO.write(SwingFXUtils.fromFXImage(wim, null), "png", new File("/test.png"));
    } catch (Exception s) {
        System.out.println(s);
    }

Answer 2

Marvin Framework provides methods to process media files frame by frame. Below it is shown a simple video processing example that highlights the path of a snooker ball.

At the left the video is played frame by frame with an interval of 30ms between frames. At the right the video is played frame by frame, but keeping all positions of the white ball through a simple image processing approach.

The source code can be checked here and the video here.

Below the essential source code to request media file frames using Marvin:

public class MediaFileExample implements Runnable{

    private MarvinVideoInterface    videoAdapter;
    private MarvinImage             videoFrame;

    public MediaFileExample(){
        try{
            // Create the VideoAdapter used to load the video file
            videoAdapter = new MarvinJavaCVAdapter();
            videoAdapter.loadResource("./res/snooker.wmv");

            // Start the thread for requesting the video frames 
            new Thread(this).start();
        }
        catch(MarvinVideoInterfaceException e){e.printStackTrace();}
    }

    @Override
    public void run() {
        try{
            while(true){
                // Request a video frame
                videoFrame = videoAdapter.getFrame();
            }
        }catch(MarvinVideoInterfaceException e){e.printStackTrace();}
    }

    public static void main(String[] args) {
        MediaFileExample m = new MediaFileExample();
    }
}

Answer 3

Frameworks like ffmpeg come with command line tools to split a video into individual frames (i.e. images in a folder).

See this question: ffmpeg split avi into frames with known frame rate

The answers contains links to Java frameworks which wrap ffmpeg.