Continuing our Kernel Integration series, we're excited to introduce DRM-CI, a groundbreaking solution that enables developers to test their graphics subsystem patches across numerous devices within the community's shared infrastructure. Released with Linux kernel 6.7, DRM-CI is a GitLab-CI pipeline that runs on the Freedesktop infrastructure. This setup allows for efficient kernel compilation and dispatches tests across multiple devices available in various community labs.

Linus Torvalds' recent critique of the Xe driver's compilation issues in Linux 6.8 underscores the critical need for thorough testing in kernel development.

Present Capabilities of DRM-CI

Currently, DRM-CI runs the igt-gpu-tools on a wide range of devices, including amdgpu, i915, mediatek, meson, msm, rockchip, and virtio-gpu. Additionally, support for vkms is in the works. These devices are stationed in various labs such as those of Collabora, Google, and the newly included Qualcomm lab. We welcome the integration of more labs - just contribute a patch!

The roadmap for DRM-CI includes enabling other devices, incorporating additional tests like kselftests, adding support for vgem driver, and implementing further automations. DRM-CI builds upon the groundwork laid by Mesa3D CI, including its GitLab YAML files and most of its setup, fostering collaboration and mutual strengthening.

Quick Tutorial on Getting Started with DRM-CI

Running tests is straightforward – push your code, hit a button, and the tests begin. Here's a quick setup guide from kernel docs:

Setup

1. Create or use your Linux tree at Freedesktop GitLab.
2. In your kernel repo's settings, switch the CI/CD configuration file to drivers/gpu/drm/ci/gitlab-ci.yml.
3. Join the drm/ci-ok group on Freedesktop GitLab for the necessary CI privileges.
4. Pushing to your repository will now trigger a CI pipeline, with results accessible at your repository's pipeline page.

Triggering Pipelines from Command Line

DRM-CI inherits its structure from the Mesa3D project, allowing the use of similar tools. The ci_run_n_monitor.sh script facilitates pipeline triggering without needing to navigate the web interface.

1. Clone Mesa: git clone --depth 1 https://gitlab.freedesktop.org/mesa/mesa.git
2. Generate a personal access token with API scope at Freedesktop GitLab and save it to ~/.config/gitlab-token.
3. From your Linux folder, run ../mesa/bin/ci/ci_run_n_monitor.sh with your specified target and project.

Example:

../mesa/bin/ci/ci_run_n_monitor.sh --target ".*" --force-manual --project linux # linux is the name of your repository on gitLab

Targeted Pipeline Triggers

Optimize resource usage by triggering only the tests relevant to your patch. Use the --target option to specify jobs, such as --target "amd.*|i915.*|msm.*" for certain device types.

Using Hash or Pipeline URL

You can also give a specific hash or pipeline URL to the script:

./bin/ci/ci_run_n_monitor.sh --target "i915.*" --force-manual --pipeline-url https://gitlab.freedesktop.org/helen.fornazier/mesa/-/pipelines/1043677

./bin/ci/ci_run_n_monitor.sh --target "android-virgl-llvmpipe" --force-manual --rev HEAD^

Gantt Chart Creation

Generate a Gantt chart for an in-depth analysis of execution duration using the ci_gantt_chart.sh script.

../mesa/bin/ci/ci_gantt_chart.sh https://gitlab.freedesktop.org/janedoe/linux/-/pipelines/1075920

Addressing Fails and Flakes

When encountering test failures that aren't due to infrastructure issues (like a device being down), one can inspect the job artifacts.

A key file, failures.csv, is generated listing all failing unit tests along with the reasons for their failures.

Example of failures.csv:

kms_addfb_basic@bad-pitch-65536,Fail
kms_addfb_basic@bo-too-small,Fail
kms_addfb_basic@too-high,Fail
kms_async_flips@async-flip-with-page-flip-events,Fail
kms_async_flips@crc,Fail
kms_async_flips@invalid-async-flip,Fail
kms_atomic_transition@plane-all-modeset-transition-internal-panels,Fail
kms_atomic_transition@plane-all-transition,Fail
kms_atomic_transition@plane-all-transition-nonblocking,Fail
kms_atomic_transition@plane-toggle-modeset-transition,Fail
kms_atomic_transition@plane-use-after-nonblocking-unbind,Fail

If your patch is the cause of a new failure, the first step should be to address this issue. However, if the failure isn't related to your patch, you can inform GitLab-CI about this expected failure to ensure your pipeline remains green.

Within the drivers/gpu/drm/ci/xfails/ directory, each job has associated files: -fails.txt, -flakes.txt, and -skips.txt. For a failure tagged as "Fail", add it to -fails.txt. If your patch unexpectedly fixes an issue, remove the respective test from -fails.txt.

In cases where a test is flaking (i.e., it inconsistently passes or fails), add it to -flakes.txt. GitLab-CI will then disregard its result.

Automating the Management of Fails and Flakes

The list of fails and flakes can be extensive, especially when enabling a new device. To assist with this, use the update-xfails.py script. It's advisable to retry failed jobs a few times to discern between fails and flakes. Run the script with the pipeline URL, and it will automatically update your xfails files based on the pipeline's results.

For example:

./drivers/gpu/drm/ci/xfails/update-xfails.py https://gitlab.freedesktop.org/janedoe/linux/-/pipelines/1017090

Tests Causing Device Crashes

If a particular test leads to an unrecoverable crash preventing the job from completing, you can list this test in the -skips.txt file (e.g., drivers/gpu/drm/ci/xfails/amdgpu-stoney-skips.txt). Tests mentioned here will not be executed, avoiding such crashes.

In addition, the skip list can be used to deliberately exclude tests that are irrelevant or unnecessary for certain drivers, thereby optimizing resource utilization.

Integrating GitLab-CI into Development Workflows

Developers can enhance their patch submission process by running pipelines and sharing the results link when submitting patches. This aids reviewers and maintainers in assessing the patch's impact, with all the advantages of pre-merge tests. However, to conserve resources, it's essential to coordinate with maintainers on their preferred testing protocols.

Expanding Beyond Graphics

Adapting the DRM-CI pipeline to other subsystems is feasible with a few modifications. The primary consideration is setting up dedicated GitLab-CI runners since Freedesktop's infrastructure is meant only for graphics.

In light of this, our team is developing a versatile and user-friendly GitLab-CI pipeline. This new pipeline is envisioned to function as a flexible interface for kernel maintainers and developers that can be evolved to connect with different test environments that can also be hooked with CI systems such as KernelCI. This approach aims to simplify the integration process, making GitLab-CI more accessible and beneficial to a broader range of developers.

Conclusion

A GitLab-CI workflow presents a significant stride in Linux kernel development, particularly for graphics subsystem patches. It not only streamlines the testing process but also fosters a culture of shared resources and community collaboration. As it continues to evolve, developers need to engage with these resources thoughtfully and responsibly, ensuring efficient and effective development cycles. Stay tuned for further updates and enhancements to DRM-CI, and don't hesitate to contribute to its growth! Happy coding!