Performance Engineering

Performance Agent: Building a Trustworthy .NET Performance Regression Gate

articleOctober 7, 2026By Ali Haghighi

BenchmarkDotNet is very good at measuring code.

The harder question is what to do with the result.

If a benchmark changes by 4%, is that a regression, normal run-to-run variation, or a comparison between environments that should never have been compared in the first place?

I built Performance Agent around that problem.

The design rule is simple:

AI proposes. Measurements decide.

The core workflow does not need an LLM. It turns benchmark evidence into a deterministic PASS, FAIL or INCONCLUSIVE verdict. Optional AI analysis can explain the evidence, but it cannot change the verdict or the process exit code.

The problem I wanted to solve

A raw benchmark result is not yet a release decision.

A useful regression gate needs to answer several questions before declaring a change safe or unsafe:

• What baseline are we comparing against?
• What performance budget was configured?
• Are the two runs actually comparable?
• Is the observed difference larger than the uncertainty in the measurements?
• Has the baseline drifted over time?
• Can CI and other tools consume the result without parsing human-oriented console output?

That pushed the project away from "another BenchmarkDotNet wrapper" and toward an evidence and decision layer around BenchmarkDotNet.

Keep the verdict outside AI

The command-line workflow is intentionally deterministic:

benchmark evidence
      |
      v
environment validation
      |
      v
statistical comparison
      |
      v
performance budget
      |
      v
PASS / FAIL / INCONCLUSIVE

The CLI exposes stable JSON output and meaningful exit codes:

0 = PASS
1 = FAIL
2 = INCONCLUSIVE or invalid evidence

That boundary matters because a CI gate should not depend on whether a model phrases an answer confidently.

AI is optional and advisory. It can summarize the measurements, explain a change and suggest what to inspect next. It cannot override the measured verdict.

Refusing to guess is a feature

One of the design decisions I care about most is the explicit INCONCLUSIVE state.

Performance measurements are noisy. Two runs may also differ in runtime, operating system, architecture, processor count or GC configuration.

When the evidence is not strong enough, returning PASS would create false confidence and returning FAIL would create noise.

The tool therefore checks environment compatibility and the uncertainty around the mean before issuing a trusted verdict. If the evidence does not justify a binary decision, it says so.

Preserve the history behind the decision

A regression tool becomes less useful if the baseline can quietly move until gradual degradation looks normal.

Performance Agent keeps immutable archived runs and distinguishes two baseline concepts:

• Current: the baseline used for normal comparison.
• Anchor: the original reference point that remains available even when Current is reset.

This makes baseline changes explicit and preserves enough provenance to understand how a decision was reached.

Validate the tool with a known-bad change

I did not want to validate a regression gate only with its own unit tests.

The separate performance-agent-realworld-test repository contains an independent BenchmarkDotNet project and a CI workflow that builds Performance Agent from source.

The test compares a StringBuilder implementation with a deliberately inefficient string += version, and also runs a no-change control.

On one GitHub-hosted runner, the regression produced:

regression check (exit 1)

Mean:
3,558.11 ns -> 289,144.7 ns
+8,026.35%
budget: +5%
FAIL

Allocation:
59,200 B -> 13,038,000 B
+21,923.65%
budget: +10%
FAIL

Overall: FAIL

The no-change control stayed inside the budget:

no-change control (exit 0)

Mean:
3,558.11 ns -> 3,534.2 ns
-0.67%
PASS

Allocation:
59,200 B -> 59,200 B
0%
PASS

Overall: PASS

The absolute timings are not the important part. They vary between runners. The important part is whether the tool reaches the correct decision from the evidence it was given.

The validation found a bug in the validator

The first real-world run exposed a more interesting failure than the deliberate performance regression.

The benchmark host inherited the caller's working directory. BenchmarkDotNet locates projects by name, so when another project with the same name was nearby, perfagent run could silently measure the wrong code.

The result was dangerous because it looked valid.

The regression test initially reported PASS because both sides had measured the slow version.

That issue was fixed in PR #119, and the independent test repository now uses a directory layout that protects against the same mistake.

That failure reinforced the main principle behind the project: a polished verdict is worthless if the evidence feeding it is wrong.

Benchmark validity is part of the product

The tool also warns about benchmark patterns that can make results misleading, including:

• trivial benchmark bodies;
• manual Stopwatch timing inside a BenchmarkDotNet benchmark;
• forced garbage collection inside the measured region.

Roslyn-based changed-code hints can map a Git diff to changed C# members and suggest which benchmarks are relevant.

The tool is careful not to overstate that feature. A changed member matching a benchmark does not prove the benchmark actually executes the changed code. Readiness reporting keeps that assumption visible instead of silently converting it into certainty.

A small architecture with explicit boundaries

The main components are separated by responsibility:

PerformanceAgent.Core
  comparison, budgets, verdicts, history

PerformanceAgent.BenchmarkDotNet
  measurement adapter

PerformanceAgent.AI
  optional advisory analysis

benchmark-host
  isolated process for benchmark execution

CLI / local web UI
  user-facing surfaces over the same services

The local web UI is loopback-only. Reports can also be generated as JSON, Markdown and standalone HTML.

At the current scope, the repository has more than 300 unit, integration and end-to-end tests covering verdict logic and CLI contracts.

What the project deliberately does not do

A trustworthy tool also needs clear boundaries.

Performance Agent does not:

• generate benchmarks automatically;
• discover production hot paths;
• prove that a selected benchmark covers the changed code;
• eliminate machine-load noise;
• sandbox benchmark projects.

Benchmark projects are executable code and are treated as trusted input.

For the strongest comparison, baseline and candidate should still run in the same CI job and environment.

What I learned from building it

The most useful part of this project was not adding AI to performance engineering.

It was deciding where AI should not be allowed to make the decision.

Measurements can be noisy and incomplete, but they can be recorded, compared and challenged. A language model can help explain them. It should not turn uncertainty into certainty.

The resulting boundary is one I now reuse in other engineering systems:

collect evidence
-> validate comparability
-> make deterministic decisions where possible
-> preserve uncertainty where necessary
-> use AI to explain and propose the next investigation

That is a much stronger foundation than asking a model whether a benchmark "looks slower."

The source, documentation and implementation history are available in Performance Agent on GitHub.