Controlled benchmarks

What “controlled” means here

Every compared variant receives the same prompt, each repetition starts from a fresh copy of the fixture, and one deterministic command evaluates the result. A setup names several model and thinking variants plus a repetition count, so the intended difference is the configured route rather than the task or starting files.

Two study shapes

A controlled A/B setup is used when two or more routes should attempt the same coding or review task and be compared on one executable score; the document-to-text capability corpus is used when every catalog model should extract text from the same collection of document hard cases and be checked against explicit text oracles.

What the numbers claim—and what they do not

These are local, small, dated observations over versioned fixtures, useful for calibrating model recommendations for similar work. They do not establish provider-wide capability, rank models in general, predict production traffic, or make a failed infrastructure attempt into evidence that a model lacks a capability; the library may recommend from evidence, but it does not route work.

How the shapes map to established suites

The harness borrows reproducible-task and deterministic-evaluation ideas from SWE-bench and HumanEval, while keeping its own synthetic fixtures and metrics. These runs are not submissions to either suite, and their numbers cannot be compared with those leaderboards.

Results

The tables below are rendered directly from website/data/benchmarks.json, the same generated projection checked in CI. A “winner” is the route selected by that setup’s declared success-and-tie-break rules for that run, not a universal best model.

Column legend

• Variant / modelThe configured route and canonical model observed in the raw run.
• SuccessSuccessful repetitions divided by all repetitions for that variant.
• Outcome scoreThe mean setup-defined evaluation score; consult the setup before comparing unlike studies.
• Avg. tokensThe mean total of the four recorded token components per repetition.
• Avg. durationThe mean wall-clock attempt duration; ranges appear when a variant was repeated.
• Cost / successObserved run cost divided by successful outcomes; unknown when there was no success or price.
• PassedCorpus cases whose completed extraction satisfied that document oracle.
• InfrastructureCases where the CLI or host failed before a capability result existed.
• Observed levelDemonstrated, partial, not demonstrated, or not attempted for this exact corpus run.

How to add a study

For A/B work, add a schema-backed definition under benchmarks/setups/ and its minimal fresh seed under benchmarks/fixtures/; for document extraction, add the corpus definition and canonical fixtures under benchmarks/document-to-text/. Run the matching command from the repository root, inspect the timestamped raw result and sidecar under benchmarks/results/, and append a corrected follow-up run instead of editing published evidence in place.

# Controlled A/B setup
dotnet run --project src/TokenEconomy.Benchmarks -- run benchmarks/setups/<setup-id>.json

# Document-to-text capability corpus
dotnet run --project src/TokenEconomy.Benchmarks -- document-to-text benchmarks/document-to-text/<corpus-id>.json

# Refresh and verify the website projection
python3 scripts/generate-website-data.py
python3 scripts/generate-website-data.py --check
dotnet test tests/TokenEconomy.Tests/TokenEconomy.Tests.csproj --filter FullyQualifiedName~WebsiteTokenUsageDataTests

The complete setup fields, oracle checks, failure classification, and publication checklist live in the benchmark protocol.

Snapshot provenance

The public projection is generated from append-only repository evidence and carries its own UTC generation timestamp. CI runs the same generator with --check and stops deployment if any evidence-derived body differs from the committed JSON.