Code coverage for Swift 3.0 project in Gitlab CI

Question

I have an iOS project written by Swift 3.0 with CocoaPods. I've configured Gitlab CI for this project and it works perfectly. This is my .gitlab-ci.yml file:

stages:
- build
build_project:
  stage: build
  script:
  - rm -rf Pods;rm -rf MyProject.xcworkspace;rm -rf MyProject.xcodeproj/project.xcworkspace
  - pod install
  - xcodebuild clean -workspace MyProject.xcworkspace -scheme MyProject | xcpretty
  - xcodebuild test -workspace MyProject.xcworkspace -scheme MyProject -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' | xcpretty -s
tags:
- ios
- lunchdot

I can't see code coverage for this project in my Gitlab repo. At the moment a coverage column for all my builds is empty. I tried to set Test coverage parsing in CI/CD Pipelines Gitlab settings, but it hadn't any effect because I don't know regex for Swift. Is it possible to set up code coverage for Swift project in Gitlab CI? How can I do this?

Answer 1

So, I tried a million ways to do it and the issue turned out to be xcpretty. After I removed it I started getting consistent results with my gitlab ci coverage data. If you still do not want to get tons of data you do not care about, you can use -quiet in your gitlab yams file. I will post it all though, so keep reading.

One still needs an external tool for coverage analysis - xcov seems to not be available anymore so I used slather. Worked like a charm.

These are the steps:

1) Get slather.

2) Get your .slather.yml file straightened out. Mine looks like the following (YourApp is the name of your app obviously):

# .slather.yml

coverage_service: cobertura_xml
xcodeproj: ./YourApp.xcodeproj
workspace: ./YourApp.xcworkspace
scheme: YourApp
source_directory: ./YourApp
output_directory: path/to/xml_report
ignore:
- "**/Pods/*"
- "thirdparty/*"
- "./Scripts/*"
- "**/3rdParty/*"
- "../**/Developer/*"
- "**/libraries/*"
- "Module-Core/*"
- "Module-Components/*"
- "**/AppUnitTests/*"

You can get the test output as html, xml, in codecov.io, etc, totally up to you. Check out the slather GitHub page to see the possible ways of doing that. But for the current issue, all we need is slather reporting in the command line so gitlab can pick it up. That is where properly setting up the gitlab yaml file comes in.

3) Set up the .gitlab-ci.yml file. Mine looks like this:

stages:
  - build

build_project_1:
  stage: build
  script:
    - xcodebuild clean -workspace YourApp.xcworkspace -scheme YourApp -quiet
    - xcodebuild test -workspace YourApp.xcworkspace -scheme YourApp -destination 'platform=iOS Simulator,name=iPhone 6,OS=10.3.1' -enableCodeCoverage YES -quiet
  after_script:
    - slather coverage -s

  only:
    - master

4) Next step is to:

• Go to your gitlab page/profile or whatever you call it

• Go to Settings and then Pipelines

• Scroll down to Test coverage parsing and add this regex expression there: \d+\%\s*$

And that is it. All you need to do is invoke a build.

Answer 2

Since Xcode 9.3 Apple provides xccov, so here is a solution using this, that does not rely on third party dependencies (and does not care if you use xcpretty, -quiet etc.)

Overview of xccovreport and xccov

When your Scheme has test coverage enabled (or when you pass -enableCodeCoverage YES to xcodebuild), an .xccovreport file is created. It contains the coverage percentages that you can see in Xcode UI.

The file is located in:

(for Xcode 9)
/Users/somename/Library/Developer/Xcode/DerivedData/MyApp-airjvkmhmywlmehdusimolqklzri/Logs/Test/E387E6E7-0AE8-4424-AFBA-EF9FX71A7E46.xccovreport

(for Xcode 10)
/Users/somename/Library/Developer/Xcode/DerivedData/MyApp-airjvkmhmywlmehdusimolqklzri/Logs/Test/Test-MyApp-2018.10.12_20-13-43-+0100.xcresult/action.xccovreport

(unless you specify a different folder in xcodebuild via -derivedDataPath)

Note: In Xcode 10, the .xccovreport file location is printed out in console after test finishes, but Xcode 9 does not do this. In any case, relying on this is probably not a good idea, as it might be silenced (e.g. by xcpretty)

The file isn't a plain text and to view it you have to call: xcrun xccov view <path_to_xccovreport_file>

Which will output the report. (You can pass --json for JSON report)

What we need to do

We want to be able to parse the file and print out the total percentage (so then GitLab can pick this up and use it in the dashboard).

There are a couple of challenges:
- Firstly we need to find out the file path of the xccovreport which contains random strings (in two places)
- Then we need to parse the file (using some regular expressions) and extract the total percentage.

The script

Here's what I am using (any improvement suggestions welcome, as I am not a bash expert):

#!/bin/bash

#1

# Xcode 10
TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
TEST_RESULTS_DIR=`ls -t $TEST_LOGS_DIR | grep "xcresult" | head -1`
TEST_COV_REPORT_FILENAME=`ls "$TEST_LOGS_DIR/$TEST_RESULTS_DIR/1_Test" | grep "xccovreport"`
TEST_COV_REPORT_FULL_PATH="$TEST_LOGS_DIR/$TEST_RESULTS_DIR/1_Test/$TEST_COV_REPORT_FILENAME"


# Xcode 9
# TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
# TEST_COV_REPORT_FILENAME=`ls $TEST_LOGS_DIR | grep "xccovreport"`
# TEST_COV_REPORT_FULL_PATH="$TEST_LOGS_DIR/$TEST_COV_REPORT_FILENAME"


# More general recursive search. Perhaps less likely to fail on new Xcode versions. Relies on filepaths containing timestamps that sort alphabetically correctly in time
# TEST_LOGS_DIR=`xcodebuild -project MyApp.xcodeproj -showBuildSettings | grep BUILD_DIR | head -1 | perl -pe 's/\s+BUILD_DIR = //' | perl -pe 's/\/Build\/Products/\/Logs\/Test/'`
# TEST_COV_REPORT_FULL_PATH=`find $TEST_LOGS_DIR -name '*.xccovreport' | sort -r | head -1`


#2
TOTAL_XCTEST_COVERAGE=`xcrun xccov view $TEST_COV_REPORT_FULL_PATH | grep '.app' | head -1 | perl -pe 's/.+?(\d+\.\d+%).+/\1/'`

#3
echo "TOTAL_XCTEST_COVERAGE=$TOTAL_XCTEST_COVERAGE"

What it does

#1 - gets the BUILD_DIR and then manipulates the path to get to the xccovreport file. Comment / uncomment the block for your version of Xcode.

#2 - We start with the full report as text. grep '.app' takes only the lines that contain .app. This is guaranteed to exist, because there is a line that reports the total coverage and contains MyApp.app. There will be multiple matches, but the first match will always be the overall total codecov score. So we use head -1 to take that first line of the grep result.

Now we have a line that looks like this:

MyApp.app                                      12.34% (8/65)

We use a perl regex to take only the “12.34%” part.

#3 - We simply print out the result (together with the variable name to make it easier to locate later in GitLab CI)

How to use

• Replace MyApp.xcodeproj with your correct value
• Make sure the correct logic is applied in step #1 (Xcode 9 / Xcode 10 / Generalized recursive search)
• Save to a printCodeCov.sh file in the root of your project.
• Make the file executable chmod +x printCodeCov.sh
• In your .gitlab-ci.yml file, add a line to the script that says - ./printCodeCov.sh
• In your GitLab Pipeline Settings, set the Test coverage parsing to TOTAL_XCTEST_COVERAGE=(.+)%

Notes

• This does not use the --json format of xccov. For that version, see below.
• This solution might be fragile, because of multiple assumptions about folder locations and report format
• I use perl instead of sed because the latter was too difficult (BSD/GNU differences, regex limitations etc).

---

JSON version

If you'd rather use the JSON report (from xccov), then in the script you need something like this:

# Xcode 10
COV_NUMBER_FRACTION=`xcrun xccov view --json $TEST_COV_REPORT_FULL_PATH | perl -pe 's/.+?\.app.+?"lineCoverage":([^,]+).+/\1/'`

# Xcode 9
# COV_NUMBER_FRACTION=`xcrun xccov view --json $TEST_COV_REPORT_FULL_PATH | perl -pe 's/.+"targets"[^l]+lineCoverage":([^,]+),.+/\1/'`

COV_NUMBER_PERCENTAGE=`bc <<< $COV_NUMBER_FRACTION*100`
TOTAL_XCTEST_COVERAGE=`printf "%0.2f%%\n" $COV_NUMBER_PERCENTAGE`