For those who have been keeping an eye on the KernelCI project, Collabora has been a primary driving force, contributing significantly to the successful overhaul of the legacy infrastructure. Our commitment to KernelCI has spanned many years.

KernelCI aims at ensuring the quality, stability, and long-term maintenance of the Linux kernel. For the past two years we have been developing the new KernelCI Architecture. With such a complex project it can be hard sometimes to spot achievements. In this blog, we will highlight the milestones we reached in KernelCI that Collabora contributed to.

For general progress on KernelCI check out the recent updates on the KernelCI blog:    

• https://kernelci.org/blog/2025/06/23/june-2025-updates-linaro-and-elisa-joined-kernelci/
• https://kernelci.org/blog/2025/02/20/kernelci-february-2025-updates/

.kernelci.yml file - standard test plans

The KernelCI community is pushing forward the concept of having a standard test plan for kernel testing. At the moment we are calling it .kernelci.yml, taking an inspiration from a .gitlab-ci.yml file that can live in the codebase. The core idea is to allow developers and maintainers to specify how they want a test to be performed for their needs.  In our MVP, they can already select the branches and architectures to build against. Over time, we want to provide options to choose kernel config (or config fragments), hardware platforms to run the tests on, and build a mechanism to identify which tests should be run for a given folder/file or patchset.

KernelCI results in the cmdline

Since the start of the year, we have been building the  results subcommand on kci-dev to extract test results from the Dashboard API and expose them for cmdline interaction. The tooling is still growing, but it is already possible to view build, boot, and test results through it with several filtering options available. As the community picks it up for use, we are receiving feedback for extra features and support they need in kci-dev.

Maestro test results quality

Running a CI system with Kubernetes runners and Hardware-in-the-loop is a complex task where many things can go wrong. In the past few months we've been strengthening Maestro's reliability to reduce build and tests with errors, timeouts, and other infrastructure failures. We created tooling to validate if all expected data is landing on the KernelCI Dashboard. That tooling exposed a few gaps, and we've been fixing them one by one. On top of that, we are also adding retry capabilities to automatically retry jobs that failed due to infrastructure issues.

Email notifications

A key missing part of the new KernelCI Architecture was notifications. Not anymore. While our notification support is still basic, we are able to send regressions notifications for build and boot for many months now.  Also, we have been sending daily summary reports for many trees.

The notifications system still has a lot to evolve. We envision better ways to configure reports (recipients, select specific tests, or platforms, etc), including integration in the Dashboard frontend.

Test events for external CI systems

In KernelCI we have an ecosystem of CI/test systems that benefit from the Maestro's orchestration work.  Maestro keeps a list of trees to test and is also able to read .kernelci.yml files. Then, Maestro builds all the trees with the specified configs. In each of these steps Maestro raises events that systems can listen to and pick up new kernels to be tested.

The event mechanism is a key part of the KernelCI ecosystem, enabling players like: Texas Instruments, Microsoft, RISC-V International, and Qualcomm to listen to our events and connect themselves to KernelCI.

Revamping KernelCI common's database for test results

KCIDB is where the CI ecosystem sends their to test results for evaluation by the Linux kernel community. With the growth of KernelCI, the current KCIDB design failed to meet our needs and sank. Collabora has been working tirelessly with the rest of the community to transition to a new DB architecture. As part of the process, we wrote KCIDB-ng, a new API to receive submissions and ingest data into PostgreSQL. In parallel, we are working with the community to transition the DB to the Dashboard.

Soon you should see the results of the work with better query performance and many extensions and fixes to our test results data structure, facilitating the work of following up on the regressions!

Broad KernelCI storage

Managing test artifacts and kernel binaries across a distributed CI system presents unique challenges. To address this, Collabora has developed kernelci-storage, a robust file storage solution that provides the backbone for KernelCI's artifact management needs. The server features a multi-backend architecture with transparent caching capabilities - currently supporting Azure Blob Storage as the primary backend, while caching files locally for improved performance and reduced latency.

The system uses JWT token-based authentication and a straightforward REST API where developers can upload files via POST requests and download them through standard GET requests. This storage infrastructure has been crucial in supporting KernelCI's scaling efforts, allowing the system to handle the growing volume of test data while maintaining fast access times for developers and CI systems that depend on kernel builds, test logs, and other critical artifacts.

netdev-CI Integration

Collaboration between different kernel testing infrastructures is essential for comprehensive validation coverage. Collabora created a bridge between KernelCI's Maestro with the Netdev CI system, enabling seamless data consumption by netdev-CI.

The main goal of NIPA (netdev-CI system) is to minimize the amount of time netdev and BPF maintainers have to spend validating patches by automatically testing patches as soon as they hit the mailing list and reporting errors to patchwork. By integrating KernelCI's broader testing capabilities with NIPA's specialized networking focus, we're creating a more comprehensive testing ecosystem that allows networking-related test results from KernelCI's diverse hardware testing platforms to be consumed by NIPA. This provides networking maintainers with additional validation data beyond what their specialized testing infrastructure covers.

Conclusion

Throughout these milestones, Collabora's expertise and sustained commitment to the KernelCI project has been instrumental in transforming it from a legacy system into a modern, scalable testing infrastructure. From designing the new orchestrator with Maestro and developing robust storage solutions to creating seamless integrations with specialized CI systems like netdev-CI and helping build developer-friendly tooling like kci-dev, Collabora has consistently delivered the foundational components that make KernelCI's ecosystem approach possible. Our contributions span across every layer of the stack - from low-level infrastructure and API design to user-facing features like email notifications and command-line interfaces.

If your organization is looking to integrate comprehensive upstream kernel testing into your products or development workflow, Collabora's proven track record with KernelCI demonstrates our ability to deliver production-ready solutions for complex CI/testing challenges. Whether you need help setting up automated kernel validation, integrating with existing CI systems, or building custom testing infrastructure, our team has the expertise to support your needs. Contact us to discuss how we can help bring the same level of kernel testing excellence that powers KernelCI to your projects.