Change Detection for Continuous Performance Engineering: Otava performs statistical analysis of performance test results stored in CSV files, PostgreSQL, BigQuery, or Graphite database. It finds change-points and notifies about possible performance regressions.
You can also read "8 Years of Optimizing Apache Otava: How disconnected open source developers took an algorithm from n3 to constant time" - https://arxiv.org/abs/2505.06758v1
If you've had the problem it solves you don't really need an explanation beyond "Change Detection for Continuous Performance Engineering" I think.
Basically if I'm reading it correctly the problem is you want to automate detection of performance regressions. You can't afford to do continuous A/B tests. So instead you run your benchmarks continuously at HEAD producing a time series of scores.
This does the statistical analysis to identify if your scores are degrading. When they degrade it gives you a statistical analysis of the location and magnitude of the (so something like "mean score dropped by 5% at p=0.05 between commits X and Y").
Basically if anyone has ever proposed "performance tests" ("we'll run the benchmark and fail CI if it scores less than X!") you usually need to be pretty skeptical (it's normally impossible to find an X high enough to detect issues but low enough to avoid constant flakes), but with fancy tools like this you can say "no to performance tests, but here's a way to do perf analysis in CI".
IME it's still tricky to get these things working nicely, it always requires a bit of tuning and you are gonna be a bit out of your depth with the maths (if you understood the inferential statistics properly you would already have written a tool like this yourself). But they're fundamentally a good idea if you really really care about perf IMO.
it is basically a “performance regression detector” It looks at a time series of benchmark results (e.g. test runtime, latency, memory usage across commits) and tries to answer one question: did something actually change, or is this just noise?
in a performance critical project I am working on, I had a precommit hook to run microbenchmarks to avoid perf regressions but noise was a real issue. Have to to try it to be sure if it can help but seems like a solution to a problem I had.
If your benchmarks are fast enough to run in pre-commit you might not need a time series analysis. Maybe you can just run an intensive A/B test between HEAD and HEAD^.
You can't just set a threshold coz your environment will drift but if you figure out the number of iterations needed to achieve statistical significance for the magnitude of changes you're trying to catch, then you might be able to just run a before/after then do a bootstrap [0] comparison to evaluate probability of a change.
I would be interested in a layperson summary of how this code deals with whacky distributions produced by computer systems. I am too stupid to understand anything more complex than introductory SPC which struggles with non-normal distributions.
I think I could use a slightly more detailed explanation of what it is and how it works at the high level. The website doesn't fully explain it.
e.g. The about page, as of this writing, does not say anything about the project.
https://otava.apache.org/docs/overview
Change Detection for Continuous Performance Engineering: Otava performs statistical analysis of performance test results stored in CSV files, PostgreSQL, BigQuery, or Graphite database. It finds change-points and notifies about possible performance regressions.
You can also read "8 Years of Optimizing Apache Otava: How disconnected open source developers took an algorithm from n3 to constant time" - https://arxiv.org/abs/2505.06758v1
and "The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System" - https://arxiv.org/abs/2003.00584 (and I guess this blog post https://www.mongodb.com/company/blog/engineering/using-chang... )
(https://en.wikipedia.org/wiki/Change_detection explains what's change detection)
that really doesn't explain it very well
If you've had the problem it solves you don't really need an explanation beyond "Change Detection for Continuous Performance Engineering" I think.
Basically if I'm reading it correctly the problem is you want to automate detection of performance regressions. You can't afford to do continuous A/B tests. So instead you run your benchmarks continuously at HEAD producing a time series of scores.
This does the statistical analysis to identify if your scores are degrading. When they degrade it gives you a statistical analysis of the location and magnitude of the (so something like "mean score dropped by 5% at p=0.05 between commits X and Y").
Basically if anyone has ever proposed "performance tests" ("we'll run the benchmark and fail CI if it scores less than X!") you usually need to be pretty skeptical (it's normally impossible to find an X high enough to detect issues but low enough to avoid constant flakes), but with fancy tools like this you can say "no to performance tests, but here's a way to do perf analysis in CI".
IME it's still tricky to get these things working nicely, it always requires a bit of tuning and you are gonna be a bit out of your depth with the maths (if you understood the inferential statistics properly you would already have written a tool like this yourself). But they're fundamentally a good idea if you really really care about perf IMO.
it is basically a “performance regression detector” It looks at a time series of benchmark results (e.g. test runtime, latency, memory usage across commits) and tries to answer one question: did something actually change, or is this just noise?
in a performance critical project I am working on, I had a precommit hook to run microbenchmarks to avoid perf regressions but noise was a real issue. Have to to try it to be sure if it can help but seems like a solution to a problem I had.
If your benchmarks are fast enough to run in pre-commit you might not need a time series analysis. Maybe you can just run an intensive A/B test between HEAD and HEAD^.
You can't just set a threshold coz your environment will drift but if you figure out the number of iterations needed to achieve statistical significance for the magnitude of changes you're trying to catch, then you might be able to just run a before/after then do a bootstrap [0] comparison to evaluate probability of a change.
[0] https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
It's hilarious that the about page doesn't tell you anything about the project
I would be interested in a layperson summary of how this code deals with whacky distributions produced by computer systems. I am too stupid to understand anything more complex than introductory SPC which struggles with non-normal distributions.
Change Detection for tests. The algorithm: https://otava.apache.org/docs/math