benchmark-gate validate

In part two, v0.2.0-alpha.1 added a stability gate and a policy file. Both are good things. Both are also new ways to write a file that’s subtly wrong — a threshold typo, an inverted warning/failure pair, a baseline entry missing its identity — and until now, the only way to find that out was to run check and read whatever exception came back first.

v0.3.0-alpha.1 adds benchmark-gate validate. It doesn’t evaluate anything. It just tells you, exhaustively, what’s wrong with a file — before that file ever gets near a real check run.

The problem with fail-fast

Here’s policy.json with two unrelated mistakes in it:

{
  "schemaVersion": 1,
  "stability": { "minimumMeasurements": 0, "maximumCoefficientOfVariation": 0.1 },
  "metrics": {
    "meanNanoseconds": { "direction": "lower-is-better", "warningPercent": 10, "failurePercent": 10 }
  }
}

minimumMeasurements: 0 is invalid. warningPercent equal to failurePercent is also invalid. Before this release, PolicyFile.Load threw on the first problem it found — you’d fix minimumMeasurements, re-run, and then find out about the threshold. Two mistakes, two round-trips.

validate finds both in one pass:

$ benchmark-gate validate —policy bad-policy.json bad-policy.json ERROR BGV105 /stability/minimumMeasurements: Value must be greater than zero; actual value was 0. ERROR BGV117 /metrics/meanNanoseconds: Metric ‘meanNanoseconds’ has warningPercent (10) >= failurePercent (10). warningPercent must be strictly less than failurePercent for the policy to be meaningful.

$ echo $LASTEXITCODE 12

Both errors, one invocation, no guessing what else might be wrong once you fix the first thing.

One command, three file types, together

benchmark-gate validate --policy policy.json
benchmark-gate validate --baseline baseline.json
benchmark-gate validate --results ./BenchmarkDotNet.Artifacts/results

All three flags are optional individually, but at least one is required, and you can combine them — validate your whole input surface in one call before wiring anything into CI:

$ benchmark-gate validate —policy bad-policy.json —baseline bad-baseline.json bad-policy.json ERROR BGV105 /stability/minimumMeasurements: Value must be greater than zero; actual value was 0. ERROR BGV117 /metrics/meanNanoseconds: Metric ‘meanNanoseconds’ has warningPercent (10) >= failurePercent (10). warningPercent must be strictly less than failurePercent for the policy to be meaningful.

bad-baseline.json ERROR BGV203 /benchmarks/1: Duplicate benchmark identity ‘MyApp.Bench.Sort|job=Default’.

Two independently broken files, two different validators (PolicyValidator and SnapshotValidator), one invocation — each grouped under its own source heading, with a blank line separating artifacts so a multi-file run stays scannable instead of turning into a wall of undifferentiated error text.

Console output groups by source file. Errors print in red when you’re looking at a real terminal; piped or redirected output stays plain, so scripts and CI logs get deterministic text.

Structured output for machines

--json writes a versioned report covering every artifact you validated in one call, instead of just printing to the console. Here’s a baseline with a duplicate benchmark entry:

benchmark-gate validate --baseline bad-baseline.json --json validation-report.json

{
  "schemaVersion": 1,
  "isValid": false,
  "errorCount": 1,
  "warningCount": 0,
  "artifacts": [
    {
      "kind": "Baseline",
      "source": "bad-baseline.json",
      "isValid": false,
      "errorCount": 1,
      "warningCount": 0,
      "diagnostics": [
        {
          "code": "BGV203",
          "severity": "Error",
          "title": "Duplicate benchmark identity",
          "path": "/benchmarks/1",
          "message": "Duplicate benchmark identity 'MyApp.Bench.Sort|job=Default'."
        }
      ]
    }
  ]
}

Top-level isValid/errorCount/warningCount let a CI step decide pass/fail without walking every artifact; each artifact still carries its own counts and full diagnostic list for anything that wants the detail — a future benchmark-gate explain BGV203 command, an IDE annotation, a Markdown summary, whatever ends up consuming this.

Where the checks actually live

This is the part I think is more interesting than the command itself: validate doesn’t contain a single validation rule. Every check it runs already existed — I extracted it from check’s own load path, rather than writing a second implementation next to the first.

PolicyFile.Load used to do this:

if (warningPercent >= failurePercent)
    throw new PolicyFileException(path, "warningPercent must be strictly less than failurePercent...");

Fail fast, one exception, done. That check now lives in PolicyValidator, Core-side, returning a ValidationDiagnostic instead of throwing — and PolicyFile.Load calls that same validator, same as it always did, just collecting every finding instead of stopping at the first one:

public static GatePolicy Load(string path)
{
    var document = Deserialize(path);
    var validation = PolicyValidator.Validate(document);

    if (!validation.IsValid)
        throw PolicyFileException.FromValidationResult(path, validation);

    return PolicyCompiler.CompileValidated(document);
}

validate calls the same PolicyValidator.Validate and just… returns what it finds, without compiling anything:

public static ValidationResult Validate(string path)
{
    var document = Deserialize(path);
    return PolicyValidator.Validate(document);
}

Same rule, same code path, two different callers with two different needs. check needs a compiled, guaranteed-valid GatePolicy or nothing. validate needs the findings themselves — including ones check would never show you, because a file with only warnings still loads fine as far as check is concerned.

The same split applies to baseline files (SnapshotValidator, which is what caught the duplicate identity above) and to BenchmarkDotNet result JSON (ObservationValidator, plus a second ObservationSetValidator for catching the same benchmark identity duplicated across two different result files in a directory — a genuinely different scope from “duplicated within one file,” and worth its own diagnostic).

Diagnostic codes are not implementation detail

Every finding has a stable, documented code — BGV1xx for policy, BGV2xx for baseline, BGV3xx for BenchmarkDotNet input. The codes are numbered by what they mean, not by the order I happened to write the checks in — BGV100/BGV101 are “missing schema version” and “unsupported schema version” as two separate codes, not one, because a missing field and an explicitly-wrong value have different fixes. Once a code ships, it means what it means; a future version can add codes, never repurpose one.

What v0.3.0 didn’t fix

BdnParameterStringParser — the thing that turns "N=1000000,Distribution=Canonical" into a parameter dictionary — still silently drops a malformed fragment ("N1000000", no =) instead of reporting it. I know, because I went looking for exactly this kind of gap while building the observation validator, and found it already documented in a two-year-old code comment as “a future version could surface this as a diagnostic.”

I didn’t fix it here, on purpose. By the time any validator runs, the parser has already thrown the malformed fragment away — there’s no evidence left to build a diagnostic from. Fixing it means changing what the parser returns, not adding a check on top of what it currently discards. That’s real, separate, tracked work, not a validate-command diagnostic I could bolt on this release.

What’s next

v0.4.0 — benchmark-gate compare. Pulls the delta calculation out of the evaluator: compare describes what changed between a baseline and a current run, independent of any policy; check applies policy to that same comparison instead of computing it inline. Full sequencing is in ROADMAP.md.

dotnet tool install --global Bijecta.BenchmarkGate.Tool --version 0.3.0-alpha.1

Repo, README, and the full roadmap: github.com/Bijecta/BenchmarkGate.