How to access camera frames in flutter quickly

Question

I would like to implement near real-time OCR on the camera feed of my flutter app. To do this I would like to access the camera data in a speedy manner. As far as I can tell I have two options, and have hit roadblocks with both:

1. Take a screenshot of the CameraPreview by putting a RepaintBoundary around it and creating a RenderRepaintBoundary, and calling boundary.toImage(). The problem with this method is that the .toImage method only seems to capture the painted widgets in the boundary and not the data from the camera preview. Simmilar to the issue described here: https://github.com/flutter/flutter/issues/17687

2. Capture an image with controller.takePicture(filePath) from Camera 0.2.1, similar to the example docs. The problem here is that it takes super long before the image becomes available (2-3 seconds). I guess that this is because the file is saved to the disc on capture and then needs to be read from the file again.

Is there any way that one can directly access the picture information after capture, to do things like pre-process and OCR?

Answer 1

For "near real-time OCR", you need CameraController#startImageStream

example code

import 'package:camera/camera.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';

void main() => runApp(MaterialApp(home: _MyHomePage()));

class _MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<_MyHomePage> {
  dynamic _scanResults;
  CameraController _camera;

  bool _isDetecting = false;
  CameraLensDirection _direction = CameraLensDirection.back;

  @override
  void initState() {
    super.initState();
    _initializeCamera();
  }

  Future<CameraDescription> _getCamera(CameraLensDirection dir) async {
    return await availableCameras().then(
      (List<CameraDescription> cameras) => cameras.firstWhere(
            (CameraDescription camera) => camera.lensDirection == dir,
          ),
    );
  }

  void _initializeCamera() async {
    _camera = CameraController(
      await _getCamera(_direction),
      defaultTargetPlatform == TargetPlatform.iOS
          ? ResolutionPreset.low
          : ResolutionPreset.medium,
    );
    await _camera.initialize();
    _camera.startImageStream((CameraImage image) {
      if (_isDetecting) return;
      _isDetecting = true;
      try {
        // await doSomethingWith(image)
      } catch (e) {
        // await handleExepction(e)
      } finally {
        _isDetecting = false;
      }
    });
  }
  Widget build(BuildContext context) {
    return null;
  }
}

This functionality was merged to https://github.com/flutter/plugins but it was not well documented.

Ref:

• https://github.com/flutter/flutter/issues/26348
• https://github.com/flutter/plugins/pull/965
• https://github.com/bparrishMines/mlkit_demo/blob/master/lib/main.dart#L43
• https://youtu.be/OAEWySye0BQ?t=1460