How to fetch gcloud Firebase test lab Android Espresso results after testing from CLI?

Question

We're using Travis CI in combination with Fastlane to do our automated builds and nightly UI Espresso tests. For the instrumentation tests we are trying to use Firebase Test lab using the gcloud command line tools.

I'm trying to get the test results in an easy to work with format so that I can report internally to Slack, for example how many tests passed or failed and which ones.

./google-cloud-sdk/bin/gcloud beta firebase test android run \
--type instrumentation \
--app ourapp/app/build/outputs/apk/mock/debug/app-mock-debug-local.apk \
--test ourapp/app/build/outputs/apk/androidTest/mock/debug/ourapp-mock-debug-androidTest.apk \
--device-ids hammerhead \
--os-version-ids 22 \
--locales en \
--orientations portrait \
--project ourappgoogleprojectid \
--timeout 15m

This all executes the tests fine and prints two links to the raw results in the Google Cloud Storage buckets with some random links and also some logs like this:

Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-vx5ak1y4tt3sw-yrzyhxjh4r1r6/]

13:25:59 Test is Pending
13:26:06 Starting attempt 1
13:26:06 Test is Running
13:26:46 Logging into the device
13:26:53 Installing APK: com.ourcompany.ourapp.debug
13:27:13 Installing APK: com.ourcompany.ourapp.test
13:27:33 Running instrumentation test. Package: com.ourcompany.ourapp.test testrunner: android.support.test.runner.AndroidJUnitRunner orchestrator: false options: []
13:34:34 Instrumentation test has finished
13:34:34 Generating video
13:34:41 Retrieving performance samples
13:35:00 Retrieving test artifacts
13:35:13 Retrieving any crash results
13:35:20 Retrieving logcat
13:35:53 Done. Test time=416 (secs)
13:36:00 Test is Finished

Instrumentation testing complete.

Then finally it prints what I'm most interested in:

┌─────────┬───────────────────────────┬──────────────────────┐
│ OUTCOME │      TEST_AXIS_VALUE      │     TEST_DETAILS     │
├─────────┼───────────────────────────┼──────────────────────┤
│ Passed  │ hammerhead-22-en-portrait │ 60 test cases passed │
└─────────┴───────────────────────────┴──────────────────────┘

But there is nothing else from which I can get a good, workable, quick summary of the results.

I think I can only do it the following way:

Grep/awk/sed/whatever the bucket url that it prints in the output. But that url has some random ID's and whatnot. Then from the bash script in Travis somehow try to download the xml files in that bucket. But all the test result xml's are in their respective folders named after what device the tests ran on. For example hammerhead-22-en-portrait. Such a hassle in order to just get a simple test summary. All I need is the info in the table that it prints above.

Is there any gcloud command option I'm missing? The documentation is horrible. I was hoping there would be options to immediately download the proper xml files or for it to set an environment variable with the proper info or something.

Or is there a way to easily extract the values from that table?

EDIT: I just found out that Android Studio supports setting up a run configuration that automatically builds, uploads the apk's and executes the tests on a Firebase test lab matrix. You then get a nice test report in the 'Run' tab in Android Studio. From here you can export these test results to either HTML or XML. That XML is exactly what I want. Does somebody know what tools Android Studio uses to do this or how I can do the same using command line tools?

Answer 1

Here are some tips that should help with what you're trying to do.

Regarding the default random IDs in the results path: you can exercise more control over that path by using the --results-dir=my/results/path flag (just be sure to choose a unique value for each test you run). You can also specify --results-bucket=gs://my-bucket to control the top-level bucket where test results get stored. Note that if you use --results-bucket you need a GCP billing account and will incur Cloud Storage charges for your results; you can use the provided default bucket for no charge and the path to it should remain stable for scripting.

All of the flag options for the gcloud firebase test android run command are documented here or via the CLI by adding --help to any gcloud command or command snippet.

Using these flags, you should be able to directly grab the junit.xml file from the raw results using a gsutil cp <src> <dst> command.

If you want to parse the gcloud output directly, it is useful to know that the test result table is printed to stdout while all the status updates are printed to stderr. So you can isolate just the results by adding 1>results.out to your command. You can also control the format of the output table via the --format=<format> flag (available on all gcloud commands). For example, if you add --format=json 1>results.out to your command above, the output will look something like this:

[
  {
    "axis_value": "hammerhead-21-en-portrait",
    "outcome": "Failed",
    "test_details": "5 test cases failed, 187 passed"
  }
]

You can learn more about gcloud formats by running gcloud topic formats.