Running Flutter Integration Tests in AWS Device Farm, Saucelabs, Firebase Test Lab etc

Question

How do I run flutter integration tests (flutter_driver) non locally within an infrastructure like AWS Device Farm, Saucelabs, Firebase Test Lab?

Running flutter_driver tests on my local machine works smoothly and easily however cloud based mobile testing services don't have the required dependencies installed to utilize Flutter CLI commands.

There are a few places in the Flutter repo that indicate this is possible, however I'm having a hard time finding exactly what needs to happen to make this possible (I have limited experience with Android & iOS integration testing).

A few references I've found:
A comment discussing an integration test in the flutter_gallery: https://github.com/flutter/flutter/issues/18879#issuecomment-400816050

A github issue to add a Firebase Test Lab integration test: https://github.com/flutter/flutter/issues/11718

What do I need to add to my flutter app to allow for remote integration testing inside AWS Device Farm, Saucelabs, Firebase Test Lab etc?

Thanks for your help.

Answer 1

Although we don't officially support the Flutter framework and Flutter tests on AWS Device Farm, we do have a way for you to run these tests using our "custom environment mode" and by pre-selecting a different, supported framework. My advice would be to do the following:

• I have a very minimal dummy test for Appium Python, a supported test framework located at https://s3-us-west-2.amazonaws.com/aws-devicefarm-support/test_bundle_slim.zip. Download this from there, and, using the command line tool zip, add your local tests to this zip file, with a command such as "zip -r ~/Downloads/test_bundle_slim.zip flutter-tests/"
• Go to AWS Device Farm, setup an automation test run, upload your app, then get to the screen where you select your test type. Choose Appium Python, upload your zip file, and AWS Device Farm will accept the test because of the original dummy Appium Python files we've left in the zip file.
• Select "customize your test environment" below there, and a new window below with a YAML test spec file will appear. Put the following lines in the pre-test section of your test spec in place of the existing code used to start an Appium server:

{code}

  - >-
    if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ];
    then
        # Run EC2 setup code here
        curl https://storage.googleapis.com/flutter_infra/releases/beta/linux/flutter_linux_v0.11.3-beta.tar.xz -o flutter_linux_v0.11.3-beta.tar.xz
        tar xf flutter_linux_v0.11.3-beta.tar.xz
    fi

    if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ];
    then
        # Run Mac setup code here
        curl https://storage.googleapis.com/flutter_infra/releases/beta/macos/flutter_macos_v0.11.3-beta.zip -o flutter_macos_v0.11.3-beta.zip
        unzip -qq flutter_macos_v0.11.3-beta.zip
    fi

  - mv flutter ~/flutter
  - echo "export PATH=$PATH:$HOME/flutter/bin">> ~/.bash_profile
  - export PATH=$PATH:$HOME/flutter/bin
  - flutter devices

{code}

• Likewise, go to the test section of the test spec file and remove the python test command "bin/py.test" and replace it with your Flutter test command, such as "flutter drive --target=./flutter-tests/main.dart
• Save and close this test spec file, and finish setting up the automation run by selecting your devices and setting up their state.

Good luck testing on AWS Device Farm! I have confirmed that these steps do work to setup Flutter properly on our testing platform.

Answer 2

Note: this addresses the assumptions that lead to your question. Not a direct answer to your question.

It is possible to run Flutter integration tests on the cloud using emulators only (no actual devices). The following article describes how to set it up. The article includes a working example on Travis-CI.

Flutter unit, widget and integration testing with IOS and Android emulators on Travis-CI

Source code on GitHub, with a link to the latest build, including integration test results running on both iOS and Android, can be found here. Feel free to clone and use as a starting point for your own project.

Running integration tests on actual devices is probably something you want to do if you are building a complex app with wide distribution (a lot of users), etc...