Code coverage

CI provider guides, Docker container coverage, and multi-report setup for DeepSource test coverage tracking.

This page covers CI-specific configuration for reporting test coverage to DeepSource. For initial setup steps, see the Track code coverage getting-started guide.

Supported formats

Analyzer	Supported formats
Go	cover.out (Generated)
Rust	GCOV, LCOV
Java	XML (Jacoco/Clover), LCOV
Scala	XML (Jacoco/Cobertura)
C#	XML (Cobertura)
JavaScript	XML (Cobertura), LCOV
PHP	XML (Cobertura)
Python	XML (Cobertura), LCOV
Ruby	JSON (SimpleCov), LCOV
C & C++	GCOV, LCOV
Swift	LCOV
Kotlin	XML (Jacoco, Kover)

Language guides

Python

Supported formats: XML (Cobertura), LCOV

Coverage.py

pip install coverage
coverage run tests.py
coverage xml

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml

Pytest

pip install pytest pytest-cov
pytest --cov=./ --cov-report xml

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml

Nose2

pip install nose2[coverage_plugin]>=0.6.5
nose2 --with-coverage --coverage-report xml

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml

Tox

Example .coveragerc:

[run]
branch = True
source = src
omit =
    .tox/*
    env/*

Example tox.ini for multiple Python versions:

[tox]
envlist = cov-init,py27,py36,py37,cov-report
skipsdist=True
skip_missing_interpreters=True

[testenv]
setenv =
    COVERAGE_FILE = .coverage.{envname}
deps =
    pytest
    pytest-cov
    coverage
commands =
    pytest --cov

[testenv:cov-init]
skipsdist = True
setenv =
    COVERAGE_FILE = .coverage
deps = coverage
commands =
    coverage erase

[testenv:cov-report]
skipsdist = True
setenv =
    COVERAGE_FILE = .coverage
deps = coverage
commands =
    coverage combine
    coverage report
    coverage xml

Go

Supported formats: cover.out (generated by go test)

The analyzer supports coverage profiles for all three modes — set, atomic, and count.

go test -coverprofile=cover.out

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key go --value-file ./cover.out

For multiple packages, combine coverage reports:

go test -coverprofile=cover.out ./somePackage
cat cover.out >> coverage.out

go test -coverprofile=cover.out ./someOtherPackage
cat cover.out >> coverage.out

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key go --value-file ./coverage.out

JavaScript

Supported formats: XML (Cobertura), LCOV

Jest

# Install jest
yarn add --dev jest
# OR
npm install --save-dev jest

# Run tests with Cobertura coverage
npx jest --coverage=true --coverageReporters=cobertura

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key javascript --value-file ./coverage/cobertura-coverage.xml

If the coverage file is generated at a path other than the root directory, pass that path to the --value-file flag.

Java

Supported formats: XML (Jacoco/Clover), LCOV

Jacoco

Add to your Maven pom.xml:

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.2</version>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>report</id>
            <phase>test</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
    </executions>
</plugin>

mvn test

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key java --value-file target/site/jacoco/jacoco.xml

For multi-module projects, use jacoco:report-aggregate to merge reports.

Clover

Add to your Maven pom.xml:

<build>
    <plugins>
        <plugin>
            <groupId>org.openclover</groupId>
            <artifactId>clover-maven-plugin</artifactId>
            <version>4.4.1</version>
            <executions>
                <execution>
                    <phase>verify</phase>
                    <goals>
                        <goal>instrument</goal>
                        <goal>aggregate</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

<reporting>
    <plugins>
        <plugin>
            <groupId>org.openclover</groupId>
            <artifactId>clover-maven-plugin</artifactId>
            <version>4.4.1</version>
        </plugin>
    </plugins>
</reporting>

mvn clean clover:setup test clover:aggregate clover:clover

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key java --value-file target/site/clover/clover.xml

Ruby

Supported formats: JSON (SimpleCov), LCOV

SimpleCov

gem install simplecov

Add to spec_helper.rb:

require 'simplecov'
SimpleCov.start

Add --require spec_helper.rb to .rspec
Run: bundle exec rake rspec

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key ruby --value-file ./coverage/.resultset.json

SimpleCov writes coverage results to .resultset.json. Upload this file to DeepSource.

Rust

Supported formats: GCOV, LCOV

cargo +stable install cargo-llvm-cov
cargo llvm-cov --lcov --output-path coverage.info

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key rust --value-file ./coverage.info

Kotlin

Supported formats: XML (Jacoco, Kover)

Jacoco

Add to your Maven pom.xml:

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.2</version>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>report</id>
            <phase>test</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
    </executions>
</plugin>

mvn test

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key kotlin --value-file target/site/jacoco/jacoco.xml

For multi-module projects, use jacoco:report-aggregate to merge reports.

Kover

Add to your top-level build.gradle.kts:

plugins {
     id("org.jetbrains.kotlinx.kover") version "0.7.2"
}

./gradlew koverXmlReport

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key kotlin --value-file build/reports/kover/xml/report.xml

Default Kover report location: build/reports/kover/xml/report.xml. Kover automatically runs your test suite — use -x test if you want to run tests separately.

Swift

Supported formats: LCOV

swift test --enable-code-coverage

# Generate LCOV report
llvm-cov export -format="lcov" </path/to/YourProjectPackageTests.xctest> -instr-profile </path/to/default.profdata> > info.lcov

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key swift --value-file info.lcov

C/C++

Supported formats: GCOV, LCOV

# Before building, add these flags:
# -fprofile-arcs -ftest-coverage
#
# For CMake:
# SET(CMAKE_CXX_FLAGS "-g -O0 -Wall -fprofile-arcs -ftest-coverage")
# SET(CMAKE_C_FLAGS "-g -O0 -Wall -W -fprofile-arcs -ftest-coverage")
# SET(CMAKE_EXE_LINKER_FLAGS "-fprofile-arcs -ftest-coverage")

# Generate coverage report
lcov –c –d . –o coverage.info

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key cxx --value-file ./coverage.info

C#

Supported formats: XML (Cobertura)

dotnet test --collect:"XPlat Code Coverage" --logger:"console;verbosity=detailed" --results-directory /tmp/test-results/

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key csharp --value-file /tmp/test-results/<test_guid>/coverage.cobertura.xml

Scala

Supported formats: XML (Jacoco/Cobertura)

Jacoco

Add sbt-jacoco to project/plugins.sbt:

addSbtPlugin("com.github.sbt" % "sbt-jacoco" % "<version>")

Run sbt jacoco to generate the report. Default location: /target/scala-{version}/jacoco/report.

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key scala --value-file target/scala-2.13/jacoco/report/jacoco.xml

PHP

Supported formats: XML (Cobertura)

PHPUnit

composer require --dev phpunit/phpunit
vendor/bin/phpunit --coverage-cobertura coverage.xml

curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh
export DEEPSOURCE_DSN=https://[email protected]
./bin/deepsource report --analyzer test-coverage --key php --value-file ./coverage.xml

Docker containers

Running tests inside a Docker container is a widely used practice in the software development community since it provides a consistent environment to run the application.

But, at the same time it also adds isolation and therefore, the test coverage reports generated inside the container are not accessible to the outside environment i.e. the CI systems on which the testing pipeline is running.

However, the following two methods can be used to report the test coverage data to DeepSource.

Inside the Docker container

For reporting test coverage to DeepSource from inside the container which runs tests, just pass some environment variables to the container using the --env/-e flag.

docker run -e DEEPSOURCE_DSN -e GITHUB_ACTIONS -e GITHUB_REF -e GITHUB_SHA ...

docker run -e DEEPSOURCE_DSN -e USER -e TRAVIS_PULL_REQUEST_SHA ...

# Export the latest git commit hash as an environment variable
export GIT_COMMIT_SHA=$(git --no-pager rev-parse HEAD | tr -d '\n')

# Pass the exported environment variable to the container in which tests
# need to be run
docker run -e DEEPSOURCE_DSN -e GIT_COMMIT_SHA ...

Outside the Docker container

The test coverage report can also be reported by copying it from the Docker container in which tests are run to a shared directory which the host can also access.

# Creating a directory to store test coverage data in the host
# This directory will be mounted in the docker container as well
mkdir shared_coverage

# Run the Docker container which runs tests
# The `-v` flag sets up a bindmount volume that links the ~/coverage directory
# from inside the container to the ~/shared_coverage directory on the host machine.
docker run --name=test -v "~/shared_coverage:$HOME/coverage" ...

# Report the test coverage report stored in the shared directory to DeepSource
./bin/deepsource report --analyzer test-coverage --key python --value-file
./shared_coverage/coverage.xml

In the Dockerfile of the container which runs test, make sure that the generated test coverage report is moved to the shared directory.

# In the container Dockerfile
mv coverage.xml ~/coverage

CI integration

If you're using a CI to run your tests, the recommended way to use the Test Coverage Analyzer is by automating the coverage reporting from your CI. This should be done in the scripts that are running your tests.

These are the things that you'd need to take care of:

Enable the test-coverage Analyzer in your repository settings.
Make sure that your CI is checking out the same commit and not making a merge commit. Failing to do this would cause the coverage report to be associated with the merge commit, and DeepSource would never pick it up for a run.
Install DeepSource CLI. This would be needed to report the coverage artifact.
DeepSource DSN for the repository. This is needed to identify the repository for which the coverage report is being sent. CLI looks for the DSN

Do not add the DEEPSOURCE_DSN variable as part of any publicly visible configuration file. It contains sensitive information.

The sections below contain boilerplate config for different CI providers. Please refer to the CLI documentation to learn more about CLI's report command used in the examples.

With Travis CI

On Travis CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN environment variable with the DSN copied above as its value.

Add this to .travis.yml:

after_success:
  # Phase: Tests have finished and a test coverage report is available

  # Install CLI
  - curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

  # From the root directory, run the report coverage command
  - ./bin/deepsource report --analyzer test-coverage --key python --value-file ./path/to/report

With Circle CI

On Circle CI, go to Settings > Environment Variables and add a DEEPSOURCE_DSN environment variable with the DSN copied above as its value.

Add the following step in .circleci/config.yml:

- run:
  name: Report results to DeepSource
  command: |
    # Tests have finished and a test coverage report is available

    # Install CLI
    curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

    # From the root directory, run the report coverage command
    ./bin/deepsource report --analyzer test-coverage --key python --value-file ./path/to/report

With GitHub Actions

On GitHub, navigate to the main page of the repository. Under your repository name, click "Settings". In the left sidebar, click Secrets.
- Type DEEPSOURCE_DSN in the "Name" input box.
- Add the value copied above.
When you checkout code, ensure that you use pull request HEAD commit instead of merge commit:

GitHub actions make a merge commit during checkout unless it is asked not to do it. If you're using GitHub actions, please make sure you make the following change in your checkout action.
```
  - uses: actions/checkout@v3
    with:
      ref: ${{ github.event.pull_request.head.sha }}
```
Add the following step in .github/workflows/main.yml:

  - name: Report results to DeepSource
    run: |
    # Tests have finished and a test coverage report is available

    # Install CLI
      curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

    # From the root directory, run the report coverage command
      ./bin/deepsource report --analyzer test-coverage --key python --value-file ./path/to/report

    env:
      DEEPSOURCE_DSN: ${{ secrets.DEEPSOURCE_DSN }}

With GitHub Actions CI using OIDC

You can also use OIDC to authenticate and send the coverage report to DeepSource, without using the DSN. Here are the steps to do that:

Make sure your GitHub Actions workflow is not making a merge commit during checkout. You can do this by using the ref input in the actions/checkout step:

- uses: actions/checkout@v3
  with:
    ref: ${{ github.event.pull_request.head.sha }}

Add required permission to get the OIDC token in your workflow:

permissions:
  id-token: write  # Required to get the OIDC token

Reference: GitHub Actions permissions

Pass --use-oidc flag to the deepsource report command while sending the coverage report:

This will use the workflow's OIDC token to get a temporary DSN for the repository and send the coverage report to DeepSource.

  - name: Report results to DeepSource
    run: |
    # Tests have finished and a test coverage report is available

    # Install CLI
      curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

    # From the root directory, run the report coverage command
      ./bin/deepsource report --analyzer test-coverage --key python --value-file ./path/to/report --use-oidc

With GitLab CI

Navigate to the project page of the repository on GitLab. Under project settings, in the sidebar, click on "CI/CD". Expand the variable section, and add the following:
- Type: "Variable"`
- Key: DEEPSOURCE_DSN
- Value: The DSN value copied above
- State: Protected (Yes)
- Masked: Yes
- Scope: All Environments

Add the following under the test job in .gitlab-ci.yml:

test:
  script:
    # Tests have finished and a test coverage report is available

    # Install CLI
    - curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

    # From the root directory, run the report coverage command
    - ./bin/deepsource report --analyzer test-coverage --key python --value-file ./path/to/report

With Heroku CI

Navigate to the app's Settings tab in the Heroku Dashboard and then add the Config Variables:
- KEY: DEEPSOURCE_DSN
- VALUE: The DSN value copied above

Run the following commands:

# Tests have finished and a test coverage report is available

# Install CLI
- curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

# From the root directory, run the report coverage command for DeepSource to analyze it
- ./bin/deepsource report --analyzer test-coverage --key python --value-file ./path/to/report

With Azure Pipelines

Set a secret variable for the repository's DSN to be used in the pipeline.
- Name: DEEPSOURCE_DSN
- Value: The DSN value for your repository on DeepSource

Add the following script in the pipeline(s) running tests for the repository:

# Add this after the script to run tests, which also generates a coverage report
- script: |
  # Install CLI
  curl -fsSL https://cli.deepsource.com/install | BINDIR=./bin sh

  # From the root directory, run the report coverage command for DeepSource to analyze it
  ./bin/deepsource report --analyzer test-coverage --value-file ./path/to/report

displayName: "Report coverage results to DeepSource"
env:
DEEPSOURCE_DSN: $(DEEPSOURCE_DSN)

In case of pull requests, Azure DevOps can make a merge commit that doesn't belong to the Repo's actual GIT tree. You can let DeepSource know about the correct commit OID of PR it should associate the coverage report to by setting the following environment variable.

env:
GIT_COMMIT_SHA: $(System.PullRequest.SourceCommitId)

Multiple reports

DeepSource supports merging coverage reports implicitly. If you have multiple CI pipelines generating partial coverage reports, send them as soon as they are generated under the same key name. DeepSource will combine all of them to prepare a final result. For example, if two CI pipelines test platform-specific parts of a module, you can report both the artifacts, and DeepSource will implicitly combine the results of the reports.

You'll notice a newly updated check every time you submit a new artifact. There's no time limit for sending multiple coverage reports.

The JaCoCo and Clover coverage formats report metrics for individual methods and may not contain data about the individual covered lines in the methods.

For such cases, DeepSource uses a max operation on the reported metrics to calculate the aggregate report. So if parts of certain methods are covered in different coverage reports, the reported line coverage for that method may be lower than the actual line coverage.

For example, if one report covers the top 25% lines for a method, and another report covers the bottom 25% lines for the same method, since there is no way of knowing if both the lines in the reports are different, DeepSource will report the coverage to be 25% even though it might be 50%. This is only a limitation for JaCoCo and Clover coverage formats at the moment.

On this page