Generate test coverage with Jacoco and Circle CI fails

Question

Am trying to generate a test coverage in the Circle CI with this file config.yml, but the build fails and it says No connected devices. Below is the error generated on Circle CI :

And according to what I read Circle CI doesn't support Emulators currently.

Below is my config.yml file :

version: 2
jobs:
  build:
    working_directory: ~/ConvergeLevelApp
    docker:
      - image: circleci/android:api-25-alpha
    environment:
      JVM_OPTS: -Xmx3200m
      CC_TEST_REPORTER_ID: 403xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      ADB_INSTALL_TIMEOUT: 60
    steps:
      - checkout
      - restore_cache:
          key: jars-{{ checksum "build.gradle" }}-{{ checksum  "app/build.gradle" }}
      - run:
          name: Download Dependencies
          command: ./gradlew androidDependencies

      - save_cache:
          paths:
            - ~/.gradle
          key: jars-{{ checksum "build.gradle" }}-{{ checksum  "app/build.gradle" }}

      - run:
          name: Setup Code Climate test-reporter
          command: |
                  curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
                  chmod +x ./cc-test-reporter

      - run:
          name: Run Tests
          command: ./gradlew lint test

      - store_artifacts:
          path: app/build/reports
          destination: reports

      - store_test_results:
          path: app/build/test-results

      - run:
          name: Generate JaCoCo report
          command: ./gradlew jacocoTestReport
      - run:
          name: Upload coverage to code climate
          command: |
                  export JACOCO_SOURCE_PATH=app/src/main/java
                  ./cc-test-reporter format-coverage app/build/reports/coverage/debug/report.xml -t jacoco
                  ./cc-test-reporter upload-coverage

Answer 1

CircleCI does not currently support Android emulators. From my experience even when there is one that works, it will not work for very long. (Link from January 17, 2018).

With that being said, a good means by which you can still gather integration test coverage is by making use of Firebase Test Lab.

There are a mix of resources for how to actually run your tests using CircleCI and Firebase Test Lab.

Once you have setup the project to run your tests on Firebase Test Lab you should be able to edit the project as follows:

1. You will need to update your config.yml file's Firebase Testing task. Notice that I use the CIRCLE_BRANCH and the CIRCLE_BUILD_NUM to ensure the results are kept in a specific location in the result bucket per build. You will want to replace YOUR_LOCATION with your actual bucket location and DEVICE with the device that your are running the Firebase Tests on.

   You will also want to replace LOCATION_OF_ANDROID_TESTS with the Jacoco command used to run your instrumentation tests (you can find this by running your instrumentation tests locally and then looking for this name.) The gsutil commands are setup to pull the coverage file from the bucket where they are streamed and then delete the folder in the bucket. This is able to be accomplished faster if you use the gsutil -m -o command.

   It is worth mentioning that this will keep the results of failing builds in the bucket (so you might need to move that command out in order to make sure that your bucket doesn't get overstuffed with data).

   sudo pip install -U crcmod
   sudo gcloud auth activate-service-account --key-file=${HOME}/gcloud-service-key.json
   sudo gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
   sudo gcloud firebase test android run \
              --type instrumentation \
              --app PATH_TO_APK \
              --test PATH_TO_TEST_APK \
              --device DEVICE_OF_YOUR_CHOOSING \
              --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \
              --directories-to-pull=/sdcard \
              --results-dir=${CIRCLE_BRANCH}_${CIRCLE_BUILD_NUM}
   sudo gsutil -m cp -r -U gs://test-lab-YOUR_LOCATION/${CIRCLE_BRANCH}_${CIRCLE_BUILD_NUM}/DEVICE/artifacts/coverage.ec app/build/outputs/code_coverage/LOCATION_OF_ANDROID_TESTS/connected/coverage.ec
   sudo gsutil rm -r gs://test-lab-YOUR_LOCATION/${CIRCLE_BRANCH}_${CIRCLE_BUILD_NUM}
   

2. After (or before your choice) the integration tests have run you will want to run your unit tests.

3. After the unit tests have ran you will need to run a combine report Jacoco task in order to combine the unit test coverage reports with the integration test coverage reports.

Final Note: In order to have the reports output on the test device while on Firebase I had to add a debug manifest that includes the WRITE_EXTERNAL_STORAGE permission. This is because my application is not using this permission in release builds.If your application is using this permission already there is no need to add a debug manifest.

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

Answer 2

Recently CircleCI made available Android machine images, which can be used to run Android (UI) tests. Official documentation