We're hiring!
*

Kernelci.org automated bisection

Guillaume Tucker avatar

Guillaume Tucker
January 16, 2018

Share this post:

The next step in Linux kernel testing: automatically finding commits that cause failures to boot.

About kernelci.org

The kernelci.org project aims at continuously testing the mainline Linux kernel, from stable branches to linux-next on a variety of platforms. When a revision fails to build or boot, kernel developers get informed via email reports. A summary of all the results can also be found directly on the website.

Automated bisection

When a kernel revision fails to boot, while it is reported in the emails it is not always obvious what caused the problem. Development branches get typically merged with many commits on them, and only one boot test is run after the merge. So the initial information is that the main branch used to work, and after these many commits got merged it started failing. The actual problem can be very hard to track down.

For a given set of bad and good revisions, it is possible to run more boot tests while using Git’s bisection feature to determine which one to test next until there is only one left. Ideally, this should be the one that caused the breakage. However there are many subtleties that complicate things, for example there may be several changes introducing different problems especially if the initial range of revisions is very wide. Also, failures to build some revisions or false positives from the boot tests can mislead the bisection logic and land on a change that is not the actual breaking one.

So, does it work?

There is currently an experimental feature to automatically run a bisection for each boot regression found on kernelci.org. This will be started with the known good/bad revisions, on a given platform, in a given lab, with a given config. It is already starting to show some useful results, for example:

The main challenge is to bring the results to a high level of quality before actively publishing them. False positives in this area can be very harmful: if the bisection finds a change that is not responsible for the breakage, reporting it can be counterproductive. Developers may spend time chasing a red herring and lose trust in the reports. For this reason, each valid bisection result is currently manually verified, curated and shared on mailing lists or by contacting individuals directly.

Next steps

We’re now in a maturing phase, identifying issues with the bisection tool and improving it until it’s ready for production. This will initially target only boot tests on mainline and stable branches. Future improvements can include extending it to more kernel trees, bisecting linux-next against mainline and covering more functional tests beyond booting to a prompt.
 


Visit Guillaume's blog.

Comments (0)


Add a Comment






Allowed tags: <b><i><br>Add a new comment:


Search the newsroom

Latest Blog Posts

Adding VP9 and MPEG2 stateless support in v4l2codecs for GStreamer

23/06/2021

Earlier this year, from January to April 2021, I worked on adding support for stateless decoders for GStreamer as part of a multimedia internship…

Bag of Freebies for XR Hand Tracking: Machine Learning & OpenXR

17/06/2021

In our previous post, we presented a project backed by INVEST-AI which introduces a multi-stage neural network-based solution. Now let's…

Testing cameras with lc-compliance on KernelCI

15/06/2021

Initiated as a joint effort by the Google Chrome OS team and Collabora, the recent KernelCI hackfest brought the addition of new tests including…

Zink: Summer 2021 update

14/06/2021

There's a lot that has happened in the world of Zink since my last update, so let's see if I can bring you up to date on the most important…

Open Source OpenGL ES 3.1 on Mali GPUs with Panfrost

11/06/2021

Panfrost, the open source driver for Arm Mali, now supports OpenGL ES 3.1 on both Midgard (Mali T760 and newer) and Bifrost (Mali G31, G52,…

Optimizing 3D performance with virglrenderer

17/05/2021

Collabora has been investing into Perfetto to enable driver authors & users to get deep insights into driver internals and GPU performance.…

Open Since 2005 logo

We use cookies on this website to ensure that you get the best experience. By continuing to use this website you are consenting to the use of these cookies. To find out more please follow this link.

Collabora Ltd © 2005-2021. All rights reserved. Privacy Notice. Sitemap.

Collabora Limited is registered in England and Wales. Company Registration number: 5513718. Registered office: The Platinum Building, St John's Innovation Park, Cambridge, CB4 0DS, United Kingdom. VAT number: 874 1630 19.