Estimate the work before routing it.

Start with an intake worksheet

This example describes a bounded feature spanning two components. The five supplied scores have concrete reasons; empirical uncertainty comes from the estimator.

using TokenEconomy;

// An illustrative AGT intake, saved before the first implementation run.
var card = new ComplexityCard
{
    TaskKey = "DEMO-42",
    Prompt = "Add a saved provider filter to the task list.",
    Project = "Agent Studio",
    Area = "task-list",
    TaskType = "feature",
    AcceptanceCriteria = ["The chosen provider survives reload.",
                          "Reset restores the unfiltered list."],
    ReferencedFiles = ["frontend/task-list.ts", "frontend/filter-store.ts"],
    ReferencedSubsystems = ["task-list", "filter-state"],
    ExpectedChangedLines = 120,
    RoutingSignals = new ComplexityRoutingSignals
    {
        CorrectnessRisk = 12,      // Reversible behavior; acceptance checks exist.
        ExpectedScope = 8,         // 51-200 expected lines / two components.
        ContextDemand = 8,         // One adjacent state contract to read.
        TaskUncertainty = 6,       // A specified feature.
        QuotaAndCostHeadroom = 5   // Current run has comfortable headroom.
    }
};

var estimate = new TaskComplexityEstimator().Estimate(card);
Console.WriteLine($"{estimate.Score}/100 -> {estimate.Level}");
Console.WriteLine($"Confidence: {estimate.Confidence:F2}");
Console.WriteLine(estimate.ScoreEvidence);
foreach (var dimension in estimate.Dimensions)
    Console.WriteLine($"{dimension.Name}: {dimension.Evidence}");
// 49/100 -> Standard
// Confidence: 0.55 (the no-history cap; not a success probability)

var estimates = new InMemoryTaskComplexityEstimateStore();
estimates.Upsert(estimate); // Replace with durable host storage in production.

Download the complete example · Pass the saved estimate into task admission

Read Score, Level and HardFloors for routing. Each criterion retains its points and evidence. TokenForecast, DurationForecast and ReissueForecast expose LowerBound, Expected, UpperBound and Evidence; duration is a TimeSpan, and reissues are an expected count. ConfidenceEvidence explains the confidence value, HistoricalEvidence reports measurement coverage, and Neighbours identifies comparable cards.

Where the values come from in AGT

At task admission, fetch the task detail and save an immutable intake snapshot with its capture time and source references. A later edit to the card must not change the input used to judge the earlier estimate.

AGT’s live and archive lists identify cards. The detail endpoint supplies their text and classification; GET /api/tasks/{key}/runs supplies recorded runs. The existing report tool reads GET /api/tasks, GET /api/tasks/archive and GET /api/tasks/{key}. These are AGT endpoints, separate from this in-process library.

Download a complete C# reader for the AGT task API → It reads the authored body and classification, leaves unknown scope fields unset, runs the estimator and saves the intake plus estimate to a new JSON file. Call it before admission; add reviewed scope and risk evidence from the worksheet.

Missing information stays missing. Leave optional fields unset and expose the missing intake evidence. Automatic fallback rules are deliberately simple, including English keyword matching. For German or ambiguous requests, use the structured worksheet and explicit reasons; do not assume the prompt parser understood the task.

A shared 100-point rubric

Use the same anchors across tasks. Each value is a number of policy points, not a percentage or an estimated token count.

The sum maps to 0–20 Trivial, over 20–50 Standard, over 50–69 Demanding, and over 69–100 Critical. Hard floors apply afterwards without changing the score: a small security-boundary edit still requires Critical; a public protocol change requires at least Demanding.

The current default ladder is Luna/Medium, Terra/Medium, Sol/Medium, Sol/XHigh. The band and correctness floor guide the router; they do not prove that a particular model is empirically best. Routing contract · Model benchmarks

What makes one task more complex than another?

Change one intake fact at a time. These examples use no historical cohort and the same comfortable headroom.

For historical neighbours, the estimator weights matching project (20%), area (24%), task type (16%), subsystem overlap (12%), file-extension overlap (10%), prompt length similarity (8%) and forecast-feature similarity (10%). It retains up to 20 cards above similarity 0.41; the closest five influence forecasts, weighted by similarity squared.

This is a transparent heuristic, not semantic understanding. Review the returned Neighbours. For model efficiency comparisons, additionally hold task class, scenario, model/effort, acceptance oracle and telemetry definition constant. The estimator does not automatically control all these factors for you.

Confidence describes input and history support. It is not a calibrated probability of success. Forecast bounds are heuristic envelopes, not 95% prediction intervals. An optional LlmComplexityAssessment can affect forecasts and confidence; it cannot rewrite the canonical score or hard floors.

Turn completed AGT attempts into comparison data

Import an exported task-storage directory with AgentStudioTaskStorageImporter, then call ComplexityHistory.FromRunRecords. Records are deduplicated by task key and attempt; the result has one sample per card.

var store = new InMemoryAgentStudioRunStore();
var importer = new AgentStudioTaskStorageImporter();
importer.ImportDirectory(exportedTaskStoragePath, store);

var coverage = ComplexityBacktester.MeasureCoverage(store.Records);
var history = ComplexityHistory.FromRunRecords(store.Records);
var estimate = new TaskComplexityEstimator().Estimate(intakeCard, history);

exportedTaskStoragePath is your read-only AGT export; intakeCard is the frozen card shown above. The importer reads task.json records. It does not automatically turn a live API detail response into the storage format or reconstruct an old prompt version.

For a historical backtest, replace each sample’s card with its retained pre-launch intake. The conversion helper uses imported card fields and cannot verify when they were authored. If an old task has no such snapshot, label it a retrospective reconstruction and keep it out of claims about pre-launch accuracy.

Recovering a historical intake from AGT

promptHistory contains task extensions, not every revision of the original body. A hash cannot recover text that is no longer present. Missing original inputs exclude a card from a strict pre-launch backtest.

Record actual input, output, cache-read and cache-write tokens; started and completed times; actual model/effort; acceptance/review result; semantic reissue reason; and whether all attempts were retained. Tokens measure consumption. Runtime measures work duration. Review and acceptance measure success. A transport retry is not a semantic failure.

Compare estimates with later outcomes

1. Freeze a cutoff. Training cards must be complete, with outcomes already observable before it.
2. Evaluate later cards using their original intake snapshots. Keep every attempt of a card in the same split.
3. Use ComplexityBacktester.RunHeldOut(training, evaluation). It rejects overlapping task keys. The caller must enforce time order.
4. Compare predicted vs measured tokens, reissues and success within comparable task groups. Always publish coverage and sample counts.

Download the complete AGT history backtest → It imports exported runs, verifies snapshot timing, excludes incomplete attempt exports and overlapping time windows, prints coverage, and reports per-card estimates alongside observed values. It does not modify AGT.

dotnet new console --framework net10.0
dotnet add package TokenEconomy
# Replace Program.cs with history-backtest.cs, then:
dotnet run -- ./agt-export ./intake-snapshots.json 2026-08-01T00:00:00Z

The separate intake-snapshots.json is a collector-owned array of card, capturedAtUtc, firstLaunchAtUtc, completedAtUtc, and completeAttemptHistory. The downloaded example defines that record. These are retained facts, not dates to infer from the current task state.

Do not train a risk score to reproduce expensive runs blindly. Costs also reflect model choice, loops, missing context and failed attempts. Use mispredicted cards to review specific rubric anchors and data coverage; use controlled model benchmarks before changing routing thresholds.

AGT evidence: measured limits of the current estimate

September 2026: new archive audit

We read all 174 Agent Studio cards among the newest 200 archive entries. Eleven contained token usage and projected runs. None of those eleven retained first-run context; three had an authored body whose hash and audit timestamp authenticated it before the first recorded dispatch. In eight cases, token-entry count minus one disagreed with the number of additional runs.

Replaying today’s estimator on those three authenticated bodies, without calibration history, gives:

These default forecasts are not calibrated to the observed AGT workload. Confidence is 0.38 in all three cases. Reported totals include cache reads and may span multiple runs; complete attempt linkage and semantic reissues are unverified. This is a current-policy replay, not the original historical prediction or a validated accuracy result.

New audit and API field map · Anonymized replay data · Coverage data

July 2026: the earlier 30-card replay

The checked-in July replay contains 30 real cards. It reports a token median absolute percentage error of 78.6% and rank correlation 0.473. This suggests some ordering signal with weak absolute forecasts in that sample.

Its reported band agreement is 13.3%, but the comparison uses a consumption-derived band. It also uses token-entry count minus one as a retry proxy and the span between token entries as duration; single-entry cards have zero span. These are not semantic retries or complete execution durations. The replay is leave-one-card-out, not a temporal holdout with proven original intake.

30-card data · Methods and limits · Estimator source · Backtest source

Use AGT as operational calibration evidence. Model benchmarks and matched local task trials remain the primary evidence for comparing model capability.