We're hiring!
*

Rust: integrating LLVM source-base code coverage with GitLab

Guillaume Desmottes avatar

Guillaume Desmottes
March 24, 2021

Share this post:

Earlier this year, the Rust compiler gained support for LLVM source-base code coverage. This feature is called source-base because it operates on AST and preprocessor information directly, producing more precise coverage data compared to the traditional gcov coverage technique.

GitLab provides built-in integration of coverage information allowing for example reviewers to check if a MR is changing tested code or if it's increasing or decreasing the total coverage of the project. In this post we'll explain how to setup a CI job in a Rust project to feed source-base coverage information to GitLab.

Generating coverage profiles

The frst step is to add a new job to your CI pipeline, which will take care of generating the coverage reports. See the GitLab documentation if your project does not have any CI setup yet.

LLVM source-base code coverage instrumentation is currently only available in Rust nightly so our job will use an image providing this version of the compiler. It also needs the llvm-tools-preview component.

In order to generate the code coverage information, called raw profile, we need to set the environment variable RUSTFLAGS="-Zinstrument-coverage". By default, the profile is saved to a file called default.profraw. This will be a problem if we have multiple tests as each one will override the profile of the previous one. To avoid this, we'll also define LLVM_PROFILE_FILE with a generic pattern so each test will save its profile to its own file: LLVM_PROFILE_FILE="coverage-%p-%m.profraw".

Once we have setup these variables we just need to run the tests as usual. Here is how a coverage CI job would look like:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test

Generating coverage report

Now that the Rust compiler has generated the coverage profiles we can generate a report. This is done using grcov from Mozilla which is installed using cargo install.

We can then use it to generate a html report that we'll export as a job artifact. Here is an example of such report for the zbus crate.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
  artifacts:
    paths:
      - 'coverage'

We want our coverage report to cover only the code of our current crate, not its dependencies. This is achieved by passing the --ignore "*cargo*" flag to grcov to exclude code from the cargo registry. Depending of your exact setup you may have to adjust it. I'd suggest to first generate a report without any --ignore and then tweak it to exclude all the code not from your crate.

GitLab Test Coverage Visualization

The html report we just generated is very handy for manually checking which part of the code is covered by tests but is not usable directly by GitLab. To do so we need to feed it a cobertura report which is currently not supported by grcov.

Fortunately, it's possible to generate a lcov report and then convert it to cobertura as suggested in the ticket.

Passing this report to GitLab as a reports artifact will enable Test Coverage Visualization allowing reviewers to easily check if a MR is changing tested or untested code.


Here is the updated job:

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Coverage parsing metric

Finally we want to tell GitLab the number of covered lines of code and the total number of lines so it can compute the coverage rate.

As grcov cannot easily produce this information yet we'll use the lcov tool:

    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov

The last step is to tell GitLab how to extract those numbers from the job logs. This is done in the CI/CD settings page, Test coverage parsing section, by setting this regular expression:

\s*lines\.*:\s*([\d\.]+%)


CI pipelines and merge requests will now display the coverage rate of the branch:

Conclusion

The flexibility of grcov and GitLab allowed our coverage job to provide:

  • a full html report as an artifact;
  • a cobertura report for GitLab integration;
  • coverage rate metrics to GitLab.

We had to work around some current grcov limitations by using external tools to convert and parse the reports. Once grcov will have gained support for these formats the whole process will become much more straightforward.

Here is the final job, you can also check the CI of zbus and gstreamer-rs for real-life examples.

coverage:
  image: "rustdocker/rust:nightly"
  stage: extras
  variables:
    RUSTFLAGS: "-Zinstrument-coverage"
    LLVM_PROFILE_FILE: "coverage-%p-%m.profraw"
  script:
    - rustup component add llvm-tools-preview
    - cargo test
    # generate html report
    - cargo install grcov
    - grcov . --binary-path ./target/debug/ -s . -t html --branch --ignore-not-existing --ignore "*cargo*" -o ./coverage/
    # generate cobertura report for gitlab integration
    - pip3 install lcov_cobertura
    - grcov . --binary-path ./target/debug/ -s . -t lcov --branch --ignore-not-existing --ignore "*cargo*" -o coverage.lcov
    - python3 /usr/local/lib/python3.5/dist-packages/lcov_cobertura.py coverage.lcov
    # output coverage summary for gitlab parsing
    - apt-get update && apt-get install -y lcov
    - lcov --summary coverage.lcov
  artifacts:
    paths:
      - 'coverage'
    reports:
      cobertura: coverage.xml

Note that this job will build and download the reporting tools at each run. A future improvement for projects running a lot of coverage reports would be to build a seperate docker container with all the tooling preinstalled, as we did in zbus.

Comments (0)


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Adding VP9 and MPEG2 stateless support in v4l2codecs for GStreamer

23/06/2021

Earlier this year, from January to April 2021, I worked on adding support for stateless decoders for GStreamer as part of a multimedia internship…

Bag of Freebies for XR Hand Tracking: Machine Learning & OpenXR

17/06/2021

In our previous post, we presented a project backed by INVEST-AI which introduces a multi-stage neural network-based solution. Now let's…

Testing cameras with lc-compliance on KernelCI

15/06/2021

Initiated as a joint effort by the Google Chrome OS team and Collabora, the recent KernelCI hackfest brought the addition of new tests including…

Zink: Summer 2021 update

14/06/2021

There's a lot that has happened in the world of Zink since my last update, so let's see if I can bring you up to date on the most important…

Open Source OpenGL ES 3.1 on Mali GPUs with Panfrost

11/06/2021

Panfrost, the open source driver for Arm Mali, now supports OpenGL ES 3.1 on both Midgard (Mali T760 and newer) and Bifrost (Mali G31, G52,…

Optimizing 3D performance with virglrenderer

17/05/2021

Collabora has been investing into Perfetto to enable driver authors & users to get deep insights into driver internals and GPU performance.…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2021. All rights reserved. Privacy Notice. Sitemap.

Collabora Limited is registered in England and Wales. Company Registration number: 5513718. Registered office: The Platinum Building, St John's Innovation Park, Cambridge, CB4 0DS, United Kingdom. VAT number: 874 1630 19.