We're hiring!
*

Rust: integrating LLVM source-base code coverage with GitLab

Guillaume Desmottes avatar

Guillaume Desmottes
March 24, 2021

Share this post:

Reading time:

Earlier this year, the Rust compiler gained support for LLVM source-base code coverage. This feature is called source-base because it operates on AST and preprocessor information directly, producing more precise coverage data compared to the traditional gcov coverage technique.

GitLab provides built-in integration of coverage information allowing for example reviewers to check if a MR is changing tested code or if it's increasing or decreasing the total coverage of the project. In this post we'll explain how to setup a CI job in a Rust project to feed source-base coverage information to GitLab.

Generating coverage profiles

The frst step is to add a new job to your CI pipeline, which will take care of generating the coverage reports. See the GitLab documentation if your project does not have any CI setup yet.

LLVM source-base code coverage instrumentation is currently only available in Rust nightly so our job will use an image providing this version of the compiler. It also needs the llvm-tools-preview component.

In order to generate the code coverage information, called raw profile, we need to set the environment variable RUSTFLAGS="-Zinstrument-coverage". By default, the profile is saved to a file called default.profraw. This will be a problem if we have multiple tests as each one will override the profile of the previous one. To avoid this, we'll also define LLVM_PROFILE_FILE with a generic pattern so each test will save its profile to its own file: LLVM_PROFILE_FILE="coverage-%p-%m.profraw".

Once we have setup these variables we just need to run the tests as usual. Here is how a coverage CI job would look like:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test

Generating coverage report

Now that the Rust compiler has generated the coverage profiles we can generate a report. This is done using grcov from Mozilla which is installed using cargo install.

We can then use it to generate a html report that we'll export as a job artifact. Here is an example of such report for the zbus crate.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
  artifacts:
    paths:
      - 'coverage'

We want our coverage report to cover only the code of our current crate, not its dependencies. This is achieved by passing the --ignore "*cargo*" flag to grcov to exclude code from the cargo registry. Depending of your exact setup you may have to adjust it. I'd suggest to first generate a report without any --ignore and then tweak it to exclude all the code not from your crate.

GitLab Test Coverage Visualization

The html report we just generated is very handy for manually checking which part of the code is covered by tests but is not usable directly by GitLab. To do so we need to feed it a cobertura report which, at the time of writing, is not yet supported by grcov.

Fortunately, it's possible to generate a lcov report and then convert it to cobertura as suggested in the ticket.

Passing this report to GitLab as a reports artifact will enable Test Coverage Visualization allowing reviewers to easily check if a MR is changing tested or untested code.


Here is the updated job:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Coverage parsing metric

Finally we want to tell GitLab the number of covered lines of code and the total number of lines so it can compute the coverage rate.

As grcov cannot easily produce this information yet we'll use the lcov tool:

    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov

The last step is to tell GitLab how to extract those numbers from the job logs. This is done in the CI/CD settings page, Test coverage parsing section, by setting this regular expression:

\s*lines\.*:\s*([\d\.]+%)


CI pipelines and merge requests will now display the coverage rate of the branch:

Conclusion

The flexibility of grcov and GitLab allowed our coverage job to provide:

  • a full html report as an artifact;
  • a cobertura report for GitLab integration;
  • coverage rate metrics to GitLab.

We had to work around some current grcov limitations by using external tools to convert and parse the reports. Once grcov will have gained support for these formats the whole process will become much more straightforward.

Here is the final job, you can also check the CI of zbus and gstreamer-rs for real-life examples.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Note that this job will build and download the reporting tools at each run. A future improvement for projects running a lot of coverage reports would be to build a seperate docker container with all the tooling preinstalled, as we did in zbus.

Comments (2)

  1. Ben Widawsky:
    Dec 14, 2023 at 08:53 PM

    grcov does now support cobertura supports (landed shortly after this post actually)

    Reply to this comment

    Reply to this comment

    1. Mark Filion:
      Dec 15, 2023 at 02:09 PM

      Very true Ben, thanks for catching that!

      Reply to this comment

      Reply to this comment


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Automatic regression handling and reporting for the Linux Kernel

14/03/2024

In continuation with our series about Kernel Integration we'll go into more detail about how regression detection, processing, and tracking…

Almost a fully open-source boot chain for Rockchip's RK3588!

21/02/2024

Now included in our Debian images & available via our GitLab, you can build a complete, working BL31 (Boot Loader stage 3.1), and replace…

What's the latest with WirePlumber?

19/02/2024

Back in 2022, after a series of issues were found in its design, I made the call to rework some of WirePlumber's fundamentals in order to…

DRM-CI: A GitLab-CI pipeline for Linux kernel testing

08/02/2024

Continuing our Kernel Integration series, we're excited to introduce DRM-CI, a groundbreaking solution that enables developers to test their…

Persian Rug, Part 4 - The limitations of proxies

23/01/2024

This is the fourth and final part in a series on persian-rug, a Rust crate for interconnected objects. We've touched on the two big limitations:…

How to share code between Vulkan and Gallium

16/01/2024

One of the key high-level challenges of building Mesa drivers these days is figuring out how to best share code between a Vulkan driver…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2024. All rights reserved. Privacy Notice. Sitemap.