Automated Testing Summit 2025

KernelCI continues to evolve as the community-driven CI system for the upstream Linux kernel. In this talk, we’ll share recent developments in test coverage, lab infrastructure, and tooling — including progress with KCIDB and improvements to result tracking and visualization. We'll highlight new collaborations, improvements in reliability, and the ongoing challenges we face in scaling automated testing for a project as complex and diverse as the Linux kernel. Whether you're running a test lab, maintaining a kernel tree, or building QA tools, this session will give you a clear picture of where KernelCI stands today and where we’re headed next.

We can all agree that it is important to test the kernel. There are a number of tools for code coverage, dynamic and static analysis to chose from to test the kernel. Linux kernel provides two testing frameworks to write and run tests. A majority of kernel tests arr written using these two frameworks. Having a rich set of tools and frameworks is great. However it can be challenging to know which one to chose. In this talk, Shuah will give an overview of tools and frameworks and their differences, and how to use them to achieve the your kernel testing objectives and goals.

Kernel CI Maestro pipeline module streamlines scheduling of kernel build jobs, and distribution of testing across participating labs. A key limitation in Maestro test distribution flow was in the inability of distributing tests to lab services running internal test flows behind corporate firewalls. In recent enhancement to Maestro, new REST APIs were introduced that enable labs to subscribe to build events and proactively submit test results -- directly addressing noted limitations. This talk will showcase how we at Texas Instruments were able to integrate these APIs in our internal testing flow, to provide access to test results for all supported TI platforms via the Kernel CI dashboard. Attendees will gain a thorough understanding of the new API functionality, the recommended integration workflow for participating labs, and the substantial advantages of direct test data submission to Kernel CI, leading to improved collaboration and accelerated identification of Linux kernel regressions.

This session will explore how the Microsoft validates upstream Linux kernels using KernelCI. We'll cover the test frameworks and methodologies used to ensure stability and reliability, as well as strategies used to scaling validation across the Azure cloud. The session will also highlight opportunities to expand test coverage and improve upstream kernel quality through KernelCI and cloud integration.

The KernelCI community is pushing forward the .kernelci.yml file concept. The goal is that maintainers start adding a file to their tree listing all the testing they require for their subsystem. When Ci systems read a tree, it automatically understand what is needed for the testing procedure. This file will evolve in a step by step manner always considering the speed for the community and the CI systems to adopt each new spec feature. The .kernelci.yml discussion intersects with the Test Requirements discussions happening in the ELISA community for example. So it would be great to have a chat with everyone about it. During this session we want discuss these ideas and hear opinions from the audience. More info: https://lore.kernel.org/kernelci/20250428083706.GA691378@rocinante/T/#u

In the general case, coverage is performed by instrumenting code at relevant points and adding sections at the linker stage to maintain data being accumulated by instrumentation. This technique reduces the effects of instrumentation on runtime behavior to a practical limit. In practice, testing proceeds by running the instrumented application, and then, typically at program exit, looking for an instrumentation file that contains counters and other data generated by the instrumentation. This method of instrumentation and data collection works well for standalone applications. It presents a significant challenge when attempting to apply it to an operating system kernel like Linux. This talk will provide a detailed overview of a successful collaboration led by Steve VanderLeest, PhD, Boeing Technical Fellow, in cooperation with the University of Illinois Urbana-Champaign, to add LLVM code coverage capability to the Linux kernel. Chuck will provide additional background including details on testing and refinement efforts, demonstrate capability, and discuss the follow-on work in progress to get the capability accepted into the mainline Linux kernel.

We have a slew of ways to test the Linux kernel: selftests, kunit, and then we have a slew of subsystem specific tests. Intel 0-day has also done a fantastic job at helping find bugs. So has syzkaller. Some subsystems like filesystems and memory management have really complex test frameworks though and have falling behind in automation. Is it possible to automate testing of complex subsystems? Should we? And what are the implications if we're successful?

To provide perspective, it takes roughly 10 years to stabilize a new Linux filesystem. But can we do better? The kdevops project was started with the goal of first of addressing automation of testing of complex subsystems such as filesystems to help reduce the amount of time it takes to stabilize new filesystems or new filesystem features. The
project aimed at supporting local virtualization, bare metal, and all cloud provider support. Seven years later since the project got started, with the help of a lot of community collaboration the project is now integral part not only of testing pipelines but also development workflows. The kdevops project now enables continuous integration for different subsystems starting with:

* Linux modules
* Linux radix tree
* Linux filesystems: xfs, btrfs, ext4, tmpfs
* Linux network filesystems: NFS
* Linux blktests
* Linux selftests

A dashboard of results is now also updated automatically based on automatic tests: https://kdevops.org

What have we learned from all this effort so far? And what lies ahead for the roadmap? If you want to contribute and help how do you do that?

Does your organization build embedded operating systems for hardware devices? Do you still test builds on hardware manually? Jumpstarter, an open-source project started by Red Hat, connects your software factory to your hardware, modernizing embedded software development. Developed in collaboration with a leading automotive manufacturer, Jumpstarter bridges the gap between embedded hardware and testing CI/CD workflows, no matter the use case. In this session, we deep-dive into automated software testing on edge devices using standardized CI/CD pipelines like Tekton or GitLab. We use your Device Under Test for tasks like flashing, booting, accessing serial, audio, and video, stress testing, recovery operations, and even over-the-air updates. Let Jumpstarter handle the hard problem of interfacing with your hardware while you focus on writing better tests for your use case. The presentation will include a live demo and will share deployment instructions, workflow examples, and real-world use cases from RedHat scaling automated testing and improving standards for open source quality assurance.

Let's get together at the end of the day to chat about highlights of the day, bring up topics that were not discussed and look at next steps for the community and the Automated Testing Summit.