How well does a model review code?

A finding must explain a defect

Keep the finding tied to the code that was reviewed: the revision, exact location, triggering condition and observable consequence. Store the original observation separately from its assessment and later fix.

Quality Studio retains native findings and their lifecycle state. Keep independent assessments in a separate evidence record. Token Economy reads sourced measurements to compare review choices and account for consumption.

Use a small set of explicit metrics

In these formulas, dismissed means independently assessed as incorrect. Keep valid-but-deferred, duplicate, out-of-scope and unverified findings separate. Report duplicate comments as an additional review burden.

Method references: SWE-PRBench distinguishes confirmed, plausible and fabricated findings; MCR-Bench tracks identity and lifecycle across review rounds. SARIF defines portable result identities and locations.

Report severe-defect recall and results on clean changes separately. Keep the model, reasoning setting, review harness, repository snapshot and assessment policy fixed for a comparison. A model’s confidence statement is not an independent assessment.

API-equivalent cost under subscriptions provides a relative price-weighted consumption measure; actual allowance and billing follow the provider’s own rules.

Published review studies

The measurements below evaluate review findings directly. Each table keeps its publisher’s protocol, metric and configuration. Compare rows inside the same study; differences across dated studies can also come from the test corpus or review harness.

Download review definitions, measurements and local coverage (JSON) · Research methods and sources

Read review evidence in C#

The library exposes published review measurements through BenchmarkEvidenceCatalog and operational Quality Studio evidence through ReviewEvidenceReport. Neither API invokes a model or verifies a finding for you.

using TokenEconomy;

var catalog = BenchmarkEvidenceCatalog.Default;
foreach (var type in catalog.Types.Where(t =>
    t.CapabilityClass == BenchmarkCapabilityClass.CodeReview))
{
    foreach (var result in catalog.ResultsFor(type.Id).Where(r =>
        r.ModelId == KnownModels.ClaudeFable51 &&
        r.PublishedAt <= new DateOnly(2026, 9, 12)))
    {
        Console.WriteLine($"{type.Name}: {result.Score} {type.Unit}");
        Console.WriteLine(result.SourceUrl);
    }
}

This returns separate Fable 5.1 recall and precision measurements for both published review configurations. Score keeps the original scale. ReasoningEffort remains Unspecified because these internal Low/High configurations do not identify the provider’s effort setting. Context retains the harness, sample and qualifications.

Complete C# example with return fields → · Full API explanation

Import operational review evidence

QualityStudioReviewRunImporter reads a Token Economy schema-v1 drop. ReviewEvidenceAggregator returns counts, assessment coverage, confirmation rate and gate failures; ReviewQualityFor(model) exposes the resulting per-model summary. Unknown outcomes remain null, and fixtures do not contribute.

Complete read-only import and aggregation example → · Input contract, outputs and conflict handling

TaskClass.SourceCodeReview selects the source-review task category. Its maintained policy fit is distinct from TaskClass.Review, whose Quality Studio suitability requires retained operational evidence. A general coding benchmark does not populate that review-quality field.

Benchmark return fields and provenance · Selection status and evidence maturity

Connect findings to Quality Studio

Use the existing Quality Studio API to read a run and its findings. Preserve native IDs and revisions when handing evidence to another host. The published research above can inform which candidates to evaluate; it does not replace the local review record.

Complete C# reader for the native Quality Studio report → Pass the host URL, repository ID and run ID as arguments. It performs one GET, reports exported finding-entry counts and preserves the original JSON, including optional evidence fields. Configure the client authentication required by your host.

Structured findings in the report

The JSON export now preserves captured observations[].sourceRevision and reviewer, plus each finding’s anchors, evidenceItems and reproduction. These use Quality Studio’s existing native types. Captured source and observed/unverified evidence stay inspectable; a missing executed thinking level is not filled from the request.

Updated readers accept older v1 reports without these optional fields. Consumers using an older closed schema or strict DTO must adopt the updated contract when reading the new fields. Quality Studio research and integration notes.

The native report and the Token Economy review drop are different contracts. Preserve the report’s bytes or hash, native run/revision and observation/finding identities. Its route fields describe the request; actual execution must come from observed provenance. A report can be replaced by a later revision, and later finding decisions are not an immutable assessment history.

Keep accepted, waived, resolved and suppression separate from confirmed/dismissed claims. A finding can disappear from a later review without proving that it was fixed. The planned TE-38 v2 bridge requires independent assessments and retained evidence cutoffs; that exporter is not yet available.

Quality Studio run contract · Finding evidence and assessment design

AGT supplies the task and attempt context: what was requested, which route ran and whether the delivery was accepted. Quality Studio supplies code findings. Keep those identities linked without interpreting every task rejection as a verified defect or every completed task as a clean review.

What local review evidence currently supports

The current v1 gate uses at least 20 operational runs, 20 assessed findings and 70% outcome coverage before deriving a review compatibility signal. Cohorts retain model, effort, CLI and review aspect. That signal measures confirmation among assessed findings; recall requires a separate oracle.

A useful next evaluation includes faulty and clean changes, independent adjudication, severity and exact-span checks, repeated runs and complete usage. Save the task-specific input before launch. The historical evaluation guide explains how to avoid using later outcomes as estimate inputs.