Automated Testing Summit 2025

Benchmark test results are difficult to interpret in an automated fashion. They often require human interpretation to detect regressions because they depend on a number of variables, including  kernel configuration, hardware capabilities, the running software environment and the specific workload. Reference values and results analysis criteria (that are needed to automatically detect regressions) have to be customized for these different variables.  Due to these variables between test systems, managing this test-related data upstream is difficult. Tim proposes standardizing the syntax and format of this data, including adding new syntax to KTAP to support a consistent format for benchmark values in KTAP/kselftest test output.

Other issues with upstream benchmark testing will be discussed, such as 1) where to store meta-data and reference value files, 2) a meta-data matching system to automatically manage associating reference values with test results, and 3) how to deal with "noise" in benchmark results. A sample boot-time regression test, addressing these issues, will be presented. These issues will need to be addressed in order to support shared use of benchmark tests upstream, and a stronger ability to detect and report performance regressions both to upstream Maintainers and within the wider kernel development community.