We're hiring!
*

Rust: integrating LLVM source-base code coverage with GitLab

Guillaume Desmottes avatar

Guillaume Desmottes
March 24, 2021

Share this post:

Earlier this year, the Rust compiler gained support for LLVM source-base code coverage. This feature is called source-base because it operates on AST and preprocessor information directly, producing more precise coverage data compared to the traditional gcov coverage technique.

GitLab provides built-in integration of coverage information allowing for example reviewers to check if a MR is changing tested code or if it's increasing or decreasing the total coverage of the project. In this post we'll explain how to setup a CI job in a Rust project to feed source-base coverage information to GitLab.

Generating coverage profiles

The frst step is to add a new job to your CI pipeline, which will take care of generating the coverage reports. See the GitLab documentation if your project does not have any CI setup yet.

LLVM source-base code coverage instrumentation is currently only available in Rust nightly so our job will use an image providing this version of the compiler. It also needs the llvm-tools-preview component.

In order to generate the code coverage information, called raw profile, we need to set the environment variable RUSTFLAGS="-Zinstrument-coverage". By default, the profile is saved to a file called default.profraw. This will be a problem if we have multiple tests as each one will override the profile of the previous one. To avoid this, we'll also define LLVM_PROFILE_FILE with a generic pattern so each test will save its profile to its own file: LLVM_PROFILE_FILE="coverage-%p-%m.profraw".

Once we have setup these variables we just need to run the tests as usual. Here is how a coverage CI job would look like:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test

Generating coverage report

Now that the Rust compiler has generated the coverage profiles we can generate a report. This is done using grcov from Mozilla which is installed using cargo install.

We can then use it to generate a html report that we'll export as a job artifact. Here is an example of such report for the zbus crate.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
  artifacts:
    paths:
      - 'coverage'

We want our coverage report to cover only the code of our current crate, not its dependencies. This is achieved by passing the --ignore "*cargo*" flag to grcov to exclude code from the cargo registry. Depending of your exact setup you may have to adjust it. I'd suggest to first generate a report without any --ignore and then tweak it to exclude all the code not from your crate.

GitLab Test Coverage Visualization

The html report we just generated is very handy for manually checking which part of the code is covered by tests but is not usable directly by GitLab. To do so we need to feed it a cobertura report which is currently not supported by grcov.

Fortunately, it's possible to generate a lcov report and then convert it to cobertura as suggested in the ticket.

Passing this report to GitLab as a reports artifact will enable Test Coverage Visualization allowing reviewers to easily check if a MR is changing tested or untested code.


Here is the updated job:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Coverage parsing metric

Finally we want to tell GitLab the number of covered lines of code and the total number of lines so it can compute the coverage rate.

As grcov cannot easily produce this information yet we'll use the lcov tool:

    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov

The last step is to tell GitLab how to extract those numbers from the job logs. This is done in the CI/CD settings page, Test coverage parsing section, by setting this regular expression:

\s*lines\.*:\s*([\d\.]+%)


CI pipelines and merge requests will now display the coverage rate of the branch:

Conclusion

The flexibility of grcov and GitLab allowed our coverage job to provide:

  • a full html report as an artifact;
  • a cobertura report for GitLab integration;
  • coverage rate metrics to GitLab.

We had to work around some current grcov limitations by using external tools to convert and parse the reports. Once grcov will have gained support for these formats the whole process will become much more straightforward.

Here is the final job, you can also check the CI of zbus and gstreamer-rs for real-life examples.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Note that this job will build and download the reporting tools at each run. A future improvement for projects running a lot of coverage reports would be to build a seperate docker container with all the tooling preinstalled, as we did in zbus.

Comments (0)


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Visual-inertial tracking for Monado

05/04/2022

Monado now has initial support for 6DoF ("inside-out") tracking for devices with cameras and an IMU! Three free and open source SLAM/VIO…

Spotlight on Meson's full-featured developer environment

30/03/2022

When developing an application or a library, it is very common to want to run it without installing it, or to install it into a custom prefix…

How to write a Vulkan driver in 2022

23/03/2022

An incredible amount has changed in Mesa and in the Vulkan ecosystems since we wrote the first Vulkan driver in Mesa for Intel hardware…

Improving the reliability of file system monitoring tools

14/03/2022

Every file system used in production has tools to try to recover from system crashes. To provide a better infrastructure for those tools,…

PipeWire: A year in review & a look ahead

08/03/2022

The PipeWire project made major strides over the past few years, bringing shiny new features, and paving the way for new possibilities in…

Landing a new syscall, part 1: What is futex?

08/02/2022

Over the past 18 months, we have been on a roller-coaster ride developing futex2, a new set of system calls. As part of this effort, the…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2022. All rights reserved. Privacy Notice. Sitemap.