You add [MemoryDiagnoser] and [Benchmark] to a class, run dotnet run -c Release, and BenchmarkDotNet hands you back a beautifully formatted table:
mean, median, standard deviation, allocated bytes per operation. It’s one of
the best measurement tools in any ecosystem.
And then what?
You read the numbers. You compare them, in your head, to what you remember from last week. Maybe you screenshot the table and paste it into a Slack message. Maybe you don’t, because nothing regressed today, so there’s nothing to say.
That’s the gap. BenchmarkDotNet is a measurement tool. It has no opinion about whether 4.5ms is acceptable or a regression. It doesn’t remember what the number was yesterday. It doesn’t fail your build. It doesn’t stop a pull request. Every one of those decisions is left as an exercise for the reader — which in practice means it doesn’t happen, until someone notices production got slower and goes spelunking through git blame.
A budget is not a measurement
The distinction I kept running into while working on CedarRecon, a reconciliation engine where I’m doing a lot of comparative benchmarking (dictionary-based vs. indexed classification strategies, that kind of thing): a measurement tells you what happened. A budget is a policy — a committed, reviewable statement that says “this operation must not take more than X, and here’s what we do if it does.”
Those are different artifacts with different lifecycles. A measurement is disposable — you take it, you look at it, you move on. A budget needs to be:
- Committed — visible in a pull request diff, not a number someone remembers
- Reviewable — a change to the budget is itself a decision someone signs off on, the same way you’d review a change to a config file
- Enforced — a CI failure, not a Slack message someone might read
BenchmarkDotNet gives you excellent measurements. It was never trying to be the second thing.
Why not just eyeball it in CI logs?
The honest answer is: you can, for a while, on a small project with one or two people watching closely. It stops working the moment either of those conditions changes.
I found this out concretely, not hypothetically. While dogfooding an early build of a tool I’ve been writing specifically to close this gap (more on that below), I ran a real benchmark suite from CedarRecon, deliberately introduced a regression into one method to confirm the tool would catch it — and it did, correctly, twice. But on the same run, an unrelated method I hadn’t touched showed a 28% regression at the largest input size.
If I’d been eyeballing a CI log, that’s the kind of number that either gets ignored (“probably just noise, I’m busy”) or gets chased for an hour before someone realizes the machine was probably running warm. What actually happened: I reran just that benchmark in isolation, and it came back clean — comfortably within a few percent of baseline, no regression at all. It was noise. A shared CI runner, or a laptop that hadn’t idled down between runs, produced one bad sample.
That’s not a knock on BenchmarkDotNet — its statistics are good. It’s a demonstration of why “look at the number and decide” doesn’t scale as a process, even when the underlying measurement tool is solid. You need something that applies a consistent rule every time, and — just as importantly — something that treats a single suspicious result as a hypothesis to confirm, not a verdict to act on immediately.
What a performance contract actually needs
Turning “here are some numbers” into “here is an enforced budget” means a few concrete things have to exist that BenchmarkDotNet doesn’t try to provide:
A committed baseline. Not a number in someone’s head — a file in the repo. JSON, reviewable, diffable. When a baseline changes, that’s a deliberate act someone did on purpose, visible in the PR.
A threshold policy. How much regression is acceptable before it’s a failure? 5%? 15%? Does a tiny absolute change (1ns → 1.2ns) count, even though it’s “20%”? These are decisions, and they should live in configuration, not in someone’s judgment call at 4pm on a Friday.
A pass/fail exit code. The single most useful thing a CI step can produce. Not a report someone might read — a build that goes red.
No SaaS, no hosted database. This one’s a deliberate design choice more than a hard requirement, but it matters to me: baselines and policies as plain repo-local JSON mean no external account, no network dependency to evaluate a benchmark, and — critically — the same command produces the same decision locally and in CI. Nothing hidden behind a dashboard only CI can reach.
Where this is going
That’s the shape of the tool I’ve been building — Cedar.BenchmarkGate, a local-first performance regression gate designed specifically for BenchmarkDotNet output. It doesn’t run your benchmarks. It doesn’t replace BenchmarkDotNet’s statistics. It reads the JSON BenchmarkDotNet already produces, compares it against a committed baseline under a policy you control, and returns an exit code CI can act on — plus a Markdown summary that explains why, not just that.
v0.1.0-alpha.1 is live now, built and dogfooded against CedarRecon’s real
benchmark suite — including that 28% false alarm, caught and correctly
dismissed by rerunning rather than reacting.
Next up: the architecture — normalized benchmark identities, why baselines and policies are deliberately separate documents, and the exit-code contract that makes this usable as an actual CI gate and not just a fancier report.