Render Flutter animation directly to video

Question

Considering that Flutter uses its own graphics engine, is there a way to render Flutter animations directly to video, or create screenshots in a frame by frame fashion?

One use case would be that this allows easier demonstration for the audience.

For example, an author wants to create a Flutter animation tutorial, where they builds a demo app and writes a companion blog post, using the animation GIF/videos rendered directly with Flutter.

Another example would be one developer outside the UI team discovers a complex animation has tiny mistake in it. Without actually learning the animation code, they can render the animation into a video and edit that short clip with annotations, then send it to the UI team for diagnoses.

Answer 1

It's not pretty, but I have managed to get a prototype working. Firstly, all animations need to be powered by a single main animation controller so that we can step through to any part of the animation we want. Secondly, the widget tree that we want to record has to be wrapped inside a RepaintBoundary with a global key. The RepaintBoundary and it's key can produce snapshots of the widget tree as such:

Future<Uint8List> _capturePngToUint8List() async {
    // renderBoxKey is the global key of my RepaintBoundary
    RenderRepaintBoundary boundary = renderBoxKey.currentContext.findRenderObject(); 
    
    // pixelratio allows you to render it at a higher resolution than the actual widget in the application.
    ui.Image image = await boundary.toImage(pixelRatio: 2.0);
    ByteData byteData = await image.toByteData(format: ui.ImageByteFormat.png);
    Uint8List pngBytes = byteData.buffer.asUint8List();

    return pngBytes;
  }

The above method can then be used inside a loop which captures the widget tree into pngBytes, and steps the animationController forward by a deltaT specified by the framerate you want as such:

double t = 0;
int i = 1;

setState(() {
  animationController.value = 0.0;
});

Map<int, Uint8List> frames = {};
double dt = (1 / 60) / animationController.duration.inSeconds.toDouble();

while (t <= 1.0) {
  print("Rendering... ${t * 100}%");
  var bytes = await _capturePngToUint8List();
  frames[i] = bytes;

  t += dt;
  setState(() {
    animationController.value = t;
  });
  i++;
}

Finally, all these png frames can be piped into an ffmpeg subprocess to be written into a video. I haven't managed to get this part working nicely yet (UPDATE: Scroll down to see solution to this), so what I've done instead is write out all the png-frames into actual png files and I then manually run ffmpeg inside the folder where they are written. (Note: I've used flutter desktop to be able to access my installation of ffmpeg, but there is a package on pub.dev to get ffmpeg on mobile too)

List<Future<File>> fileWriterFutures = [];

frames.forEach((key, value) {
  fileWriterFutures.add(_writeFile(bytes: value, location: r"D:\path\to\my\images\folder\" + "frame_$key.png"));
});

await Future.wait(fileWriterFutures);

_runFFmpeg();

Here is my file-writer help-function:

Future<File> _writeFile({@required String location, @required Uint8List bytes}) async {
  File file = File(location);
  return file.writeAsBytes(bytes);
}

And here is my FFmpeg runner function:

void _runFFmpeg() async {
  // ffmpeg -y -r 60 -start_number 1 -i frame_%d.png -c:v libx264 -preset medium -tune animation -pix_fmt yuv420p test.mp4
  var process = await Process.start(
      "ffmpeg",
      [
        "-y", // replace output file if it already exists
        "-r", "60", // framrate
        "-start_number", "1",
        "-i", r"./test/frame_%d.png", // <- Change to location of images
        "-an", // don't expect audio
        "-c:v", "libx264rgb", // H.264 encoding
        "-preset", "medium",
        "-crf",
        "10", // Ranges 0-51 indicates lossless compression to worst compression. Sane options are 0-30
        "-tune", "animation",
        "-preset", "medium",
        "-pix_fmt", "yuv420p",
        r"./test/test.mp4" // <- Change to location of output
      ],
      mode: ProcessStartMode.inheritStdio // This mode causes some issues at times, so just remove it if it doesn't work. I use it mostly to debug the ffmpeg process' output
   );

  print("Done Rendering");
}

Update:

Since this answer was posted I have figured out how to pipe the images directly to ffmpeg without the need of first writing out all the files. The following is my updated render function taken from one of my widgets. Some variables are present within the widget's context, but I hope that their values can be inferred from the context:

void render([double? pixelRatio]) async {
    // If already rendering, return
    if (isRendering) return;

    String outputFileLocation = "final.mp4";

    setState(() {
      isRendering = true;
    });

    timeline.stop();

    await timeline.animateTo(0.0, duration: const Duration(milliseconds: 700), curve: Curves.easeInOutQuad);
    setState(() {
      timeline.value = 0.0;
    });

    await Future.delayed(const Duration(milliseconds: 100));

    try {
      int width = canvasSize.width.toInt();
      int height = canvasSize.height.toInt();
      int frameRate = 60;
      int numberOfFrames = frameRate * (timeline.duration!.inSeconds);

      print("starting ffmpeg..");
      var process = await Process.start(
          "ffmpeg",
          [
            "-y", // replace output file if it already exists
            // "-f", "rawvideo",
            // "-pix_fmt", "rgba",
            "-s", "${width}x$height", // size
            "-r", "$frameRate", // framrate
            "-i", "-",
            "-frames", "$numberOfFrames",
            "-an", // don't expect audio
            "-c:v", "libx264rgb", // H.264 encoding
            "-preset", "medium",
            "-crf",
            "10", // Ranges 0-51 indicates lossless compression to worst compression. Sane options are 0-30
            "-tune", "animation",
            "-preset", "medium",
            "-pix_fmt", "yuv420p",
            "-vf",
            "pad=ceil(iw/2)*2:ceil(ih/2)*2", // ensure width and height is divisible by 2
            outputFileLocation
          ],
          mode: ProcessStartMode.detachedWithStdio,
          runInShell: true);

      print("writing to ffmpeg...");
      RenderRepaintBoundary boundary = paintKey.currentContext!.findRenderObject()! as RenderRepaintBoundary;

      pixelRatio = pixelRatio ?? 1.0;
      print("Pixel Ratio: $pixelRatio");

      for (int i = 0; i <= numberOfFrames; i++) {
        Timeline.startSync("Render Video Frame");
        double t = (i.toDouble() / numberOfFrames.toDouble());
        // await timeline.animateTo(t, duration: Duration.zero);
        timeline.value = t;

        ui.Image image = await boundary.toImage(pixelRatio: pixelRatio);
        ByteData? rawData = await image.toByteData(format: ui.ImageByteFormat.png);
        var rawIntList = rawData!.buffer.asInt8List().toList();
        Timeline.finishSync();

        if (i % frameRate == 0) {
          print("${((t * 100.0) * 100).round() / 100}%");
        }

        process.stdin.add(rawIntList);

        image.dispose();
      }
      await process.stdin.flush();

      print("stopping ffmpeg...");
      await process.stdin.close();
      process.kill();
      print("done!");
    } catch (e) {
      print(e);
    } finally {
      await timeline.animateTo(beforeValue, duration: const Duration(milliseconds: 500), curve: Curves.easeInOutQuad);
      setState(() {
        isRendering = false;
      });
    }
  }

Answer 2

I used Erik's answer as a starting point for my own implementation and would like to add to his original answer.

After saving all the png images to the target location, I used the ffmpeg package for Flutter to create a video of all the images. Since it took a while to find the right settings to create a video that could also be played by QuickTime Player, I want to share them with you:

final FlutterFFmpeg _flutterFFmpeg =
    new FlutterFFmpeg(); // Create new ffmpeg instance somewhere in your code

// Function to create the video. All png files must be available in your target location prior to calling this function.
Future<String> _createVideoFromPngFiles(String location, int framerate) async {
  final dateAsString = DateFormat('ddMMyyyy_hhmmss').format(DateTime.now());
  final filePath =
      "$location/video_$dateAsString.mov"; // had to use mov to be able to play the video on QuickTime

  var arguments = [
    "-y", // Replace output file if it already exists
    "-r", "$framerate", // Your target framerate
    "-start_number", "1",
    "-i",
    "$location/frame_%d.png", // The location where you saved all your png files
    "-an", // Don't expect audio
    "-c:v",
    "libx264", // H.264 encoding, make sure to use the full-gpl ffmpeg package version
    "-preset", "medium",
    "-crf",
    "10", // Ranges 0-51 indicates lossless compression to worst compression. Sane options are 0-30
    "-tune", "animation",
    "-preset", "medium",
    "-pix_fmt",
    "yuv420p", // Set the pixel format to make it compatible for QuickTime
    "-vf",
    "pad=ceil(iw/2)*2:ceil(ih/2)*2", // Make sure that height and width are divisible by 2
    filePath
  ];

  final result = await _flutterFFmpeg.executeWithArguments(arguments);
  return result == 0
      ? filePath
      : ''; // Result == 0 indicates that video creation was successful
}

If you are using libx264, make sure that you follow the instructions for the flutter_ffmpeg package: You must use the full-gpl version, which includes the x264 library.

Depending on the length of your animation, the desired framerate, the pixel ratio and the device's memory, saving all frames before writing the files could cause memory issues. Therefore, depending on your use case, you may want to pause/resume your animation and write the files in multiple batches so that you don't risk exceeding the available memory.