CodingAgentRunner

Documentation

The public API is small: five types cover most run-control work — CliRunner, CliOptions, CliRunRequest, CliRunInfo, and CliRunEvent. Metrics and rendering are opt-in layers on top of the same event stream. Everything below is example-first and accurate to the current API.

Quickstart

Wire the runner once, resolve a driver by CLI type, and start a run. RunId is your correlation key — you bring it; the library won't generate one for you. Every event, log line, and Stop call is addressed by it.

using CodingAgentRunner;

var runner = new CliRunner(new CliOptions());   // wire once
var driver = runner.Get("claude");              // a driver per CLI

var (run, error) = await driver.StartAsync(new CliRunRequest
{
    RunId = "task-1",
    Prompt = "Fix the failing build.",
    WorkingDirectory = repo,
});

A full request — model, reasoning, permission, context

var (run, error) = await driver.StartAsync(new CliRunRequest
{
    RunId            = "task-42",
    Prompt           = promptText,
    WorkingDirectory = repo,
    Model            = "claude-opus-4-8",
    ThinkingLevel    = CliThinkingLevels.XHigh,   // normalized against the model
    PermissionMode   = CliPermissionModes.Yolo,   // unattended; never prompts
    // ContextMode defaults to 'clean' — an isolated per-run CLI home.
    // Use 'shared' to run against the operator's normal signed-in CLI home.
}, ct);

Check availability across CLIs

foreach (var d in runner.Drivers)
    Console.WriteLine($"{d.CliType}: {(d.IsAvailable() ? "ok" : "not installed")}");

var (available, version, path) = runner.Get("codex").TestCliPath();
if (runner.TryGet(userChoice, out var chosen)) { /* … */ }

CLI setup & diagnostics

Installing and signing in stays your job — the library runs CLIs that are already set up on the machine. InspectEnvironment() tells you what's missing and which commands fix it: per CLI the install state, the version, whether a credential source exists, and — as data (CliSetupInfo) — the install commands, sign-in steps and docs URL. It also checks Node.js/npm, which the npm-distributed CLIs need.

using CodingAgentRunner.Diagnostics;

var report = runner.InspectEnvironment();
if (!report.AnyReady)
    Console.WriteLine(report.ToText());          // per-CLI state + the fixing commands

CliEnvironmentStatus codex = report.For("codex")!;
if (!codex.Installed)
    logger.LogWarning("Codex missing. Install: {Cmd}", codex.Setup.RecommendedInstallCommand);
if (codex.Credentials == CredentialSignal.NotFound)
    logger.LogWarning("Codex not signed in. {Step}", codex.Setup.LoginSteps[0]);

What ToText() renders — a machine with nothing installed

claude      NOT INSTALLED (probed 'claude')
            install: npm install -g @anthropic-ai/claude-code
            NOT SIGNED IN — Run `claude` in a terminal; the first run opens a browser sign-in …
            docs: https://code.claude.com/docs/en/setup
…
node        installed v24.18.0
npm         installed 11.16.0

Installing and signing in, per CLI. Installs are scriptable; the sign-in is a one-time browser flow per machine, with a non-interactive alternative for headless/CI setups. A credential probe reports a credential source, not a validated token — an expired login still fails at run time.

Agent	Install (scriptable)	Sign in (once per machine)	Headless / CI
Claude Code	npm install -g @anthropic-ai/claude-code	run claude, browser sign-in	claude setup-token → CLAUDE_CODE_OAUTH_TOKEN, or ANTHROPIC_API_KEY
OpenAI Codex	npm install -g @openai/codex	codex login (ChatGPT account)	codex login --device-auth or --with-api-key, or copy ~/.codex/auth.json
Google Gemini	npm install -g @google/gemini-cli	run gemini, Google sign-in	GEMINI_API_KEY, or a service account
Google Antigravity	irm https://antigravity.google/cli/install.ps1 | iex (Windows) · install.sh (macOS/Linux)	run agy once (prints URL + code on headless)	no documented token export

The full guide — credential file locations, seeding a login onto another machine, and why the library deliberately does not auto-install — is in docs/cli-setup.md.

Run identity: RunId, SessionId, continuation

Three ids, three jobs. You assign the RunId; the CLI assigns a SessionId (surfaced in a SessionStarted event); to continue a conversation you start a new run that carries the captured session id back as ResumeSessionId.

Id	Assigned by	Spans	Used for
RunId	you	one run (one spawn)	addressing events / logs / Stop / output
SessionId	the CLI	a conversation (many runs)	continuing the conversation
ResumeSessionId	you (a captured SessionId)	—	"continue this session" — set = continue, null = fresh

// Run 1 — capture the CLI's SessionId.
string? sessionId = null;
driver.OnRunEvent += (runId, e) =>
{
    if (e is CliRunEvent.SessionStarted s) sessionId = s.SessionId;
};
await driver.StartAsync(new CliRunRequest
{
    RunId = "task-1", Prompt = "Draft the parser.", WorkingDirectory = repo,
});

// Run 2 = continuation — NEW RunId, SAME session. One field is the whole signal.
if (driver.IsCompatibleSessionId(sessionId))
    await driver.StartAsync(new CliRunRequest
    {
        RunId = "task-1-followup", Prompt = "Now add error recovery.",
        WorkingDirectory = repo,
        ResumeSessionId = sessionId,
    });

The event stream

CliRunEvent is a closed sum type, so a switch over it is checked for exhaustiveness by the compiler. Two events are engine-universal (RunStarted first, RunEnded last); the rest come from the per-CLI adapter, so treat CLI-specific events defensively. RunEnded is the single terminal event and its Outcome is three-valued.

driver.OnRunEvent += (runId, e) =>
{
    switch (e)
    {
        case CliRunEvent.SessionStarted s:    Log($"session {s.SessionId}");            break;
        case CliRunEvent.OutputDelta d:       Console.Write(d.Text);                  break;
        case CliRunEvent.ToolStarted t:       Log($"[tool] {t.ToolName} {t.Argument}"); break;
        case CliRunEvent.ToolCompleted c:     Log($"[tool done] {c.ToolName} error={c.IsError}"); break;
        case CliRunEvent.RateLimitObserved r: ShowUsage(r.Window, r.Status, r.ResetsAt); break;
        case CliRunEvent.TurnCompleted t:     Log($"turn done: {t.UsageSummary}");     break;
        case CliRunEvent.TurnFailed f:        Log($"turn failed: {f.Reason}");         break;
        case CliRunEvent.Interrupt i:         Log($"interrupt {i.Reason} (fatal={i.IsFatal})"); break;
        case CliRunEvent.RunEnded end:                                // one terminal, 3-valued
            Log(end.Outcome switch
            {
                RunOutcome.Completed => $"done (exit {end.ExitCode})",
                RunOutcome.Stopped   => $"stopped: {end.Reason}",   // deliberate — not an error
                RunOutcome.Failed    => $"failed: {end.Reason}",
                _ => "?",
            });
            break;
    }
};

Group	Events	Notes
Engine (every CLI)	RunStarted, RunEnded	Always first / last. RunEnded.Outcome ∈ Completed/Stopped/Failed.
Common core (Claude, Codex, Gemini, Antigravity)	SessionStarted, OutputDelta, ToolStarted/ToolCompleted, TurnCompleted/TurnFailed, Unknown	The reliable baseline a consumer can depend on. Antigravity reuses Gemini's adapter.
CLI-specific (handle defensively)	SessionInitializing (Claude), TurnStarted (Codex), Heartbeat (Codex), RateLimitObserved (Claude), PlanUpdated (Claude/Codex)	"May not arrive" — don't depend on them.
Interrupt signal	Interrupt	A classifier-raised stop condition with a typed InterruptReason (EnvironmentBlocker, QuotaExhausted, Sentinel, SelfReference, NeedsInput, SilentCompletion) and an IsFatal flag — end a run stuck against a wall before the watchdog's silence budget runs out.
Reserved	NeedsInput, ApprovalRequested	Defined in the contract; no adapter emits them today.

Interrupt classification — a stop-worthy line becomes a typed event

An IInterruptClassifier runs per output line. When it recognizes a stop-worthy condition it emits a typed CliRunEvent.Interrupt(InterruptReason, Detail, IsFatal) into the same stream — for cases like EnvironmentBlocker, QuotaExhausted, Sentinel, SelfReference, NeedsInput, and SilentCompletion. The library only raises the event; you keep Stop() authority and decide how to act on IsFatal. The mechanism is wired into the engine read-loop and a starter EnvironmentBlocker classifier ships, but every built-in descriptor leaves its InterruptClassifier unset, so no built-in CLI emits Interrupt until you opt in.

driver.OnRunEvent += (runId, e) =>
{
    if (e is CliRunEvent.Interrupt i)
    {
        Log($"interrupt {i.Reason}: {i.Detail}");
        if (i.IsFatal) driver.Stop(runId, RunStopReason.UserStop); // your call
    }
};

Raw output, live and after the run

// live: every raw line, tagged stdout/stderr
driver.OnOutput += (runId, line) => sink.Append($"{line.Stream}: {line.Text}");

driver.OnFinished += (runId, info) =>
{
    Console.WriteLine($"{info.RunId}: {info.Status} (exit {info.ExitCode}, {info.DurationSeconds:F1}s)");
    var lines = driver.GetOutput(runId);   // from the buffer…
    driver.Forget(runId);                  // …release it; GetOutput then reads from disk
};

Configuration

Everything on CliOptions is optional and defaults sanely; leave a path unset and the runner finds the CLI by name on PATH.

var options = new CliOptions
{
    ClaudePath = @"C:\tools\claude\claude.exe",   // else: 'claude' from PATH
    CodexPath  = "codex",
    EnvironmentOverrides = new Dictionary<string, string> { ["HTTP_PROXY"] = proxy },
};
var runner = new CliRunner(options, logger);

Git-guard — the host owns version control

// Default: the agent may not run commit/push/reset/… (the host owns git).
var guarded = new CliOptions
{
    GitGuard = new GitGuardOptions
    {
        ForbiddenCommands = ["commit", "push", "reset", "checkout"],
        EnvPrefix = "MY_APP",
    },
};
// …or deliberately let the agent use git:
var trusting = new CliOptions { AllowAgentGitMutation = true };

Registration in DI (ASP.NET)

builder.Services.AddSingleton(sp => new CliRunner(
    new CliOptions(),
    sp.GetRequiredService<ILoggerFactory>().CreateLogger("cli")));

// in an endpoint / service:
var driver = runner.Get(request.Cli);

Capabilities & overridable defaults

Ask what a CLI can do instead of switching on its type. CliCapabilities answers per CLI: SupportsCleanContext, SupportsResume, EmitsHeartbeatDuringThinking (true for Codex's reasoning models), and the available ThinkingLevels with a DefaultThinkingLevel. A shared context is still a normal supervised run; only the CLI home changes from a temporary clean home to the user's real signed-in state. Use those flags to keep a selection UI honest across CLIs rather than special-casing each one.

var caps = driver.Capabilities(model);   // CliCapabilities, resolved per model
if (caps.SupportsCleanContext)
    request = request with { ContextMode = CliContextModes.Clean };
if (!caps.SupportsResume)
    request = request with { ResumeSessionId = null };   // this CLI can't resume; start fresh

// Codex reasoning models stay silent but emit Heartbeat; the watchdog already
// treats Heartbeat as activity, so EmitsHeartbeatDuringThinking just informs budgets.

Overridable defaults — resolve by specificity

A default resolves from most specific to least: CLI ▸ model ▸ thinking level, falling back to a global default. A CliScope (CliType, optional Model, optional ThinkingLevel) selects the layer; Set(scope, value) is the one-line override. The library ships the real per-CLI/model tables it needs — CliThinkingLevels and CliReasoningFlags — not a pre-seeded battery of every default.

// A default silence budget in seconds; override for one CLI, then one model.
var silenceBudget = new CliDefault<int>(300);
silenceBudget.Set(new CliScope("codex"), 600);                       // Codex: 10 min
silenceBudget.Set(new CliScope("codex", Model: "gpt-5.5"), 1200);    // that model: 20 min

// Resolution walks CLI ▸ model ▸ thinking level, then the global fallback.
int seconds = silenceBudget.Resolve(new CliScope("codex", Model: "gpt-5.5"));   // 1200

Watchdog & plan tracking

The engine already tracks each run's phase and last activity. Attach the built-in watchdog in one line instead of wiring your own timer; it stops a hung run as RunStopReason.Watchdog — a deliberate stop, not a crash. Budgets are per phase (a model still "thinking" inside a tool call is fine for minutes; no handshake right after spawn is a fast failure).

// autoStop: stop a hung run itself; dispose to detach.
using var watchdog = RunWatchdog.Attach(driver, WatchdogPolicy.Default, autoStop: true);

watchdog.OnHung += (runId, phase, silence) =>
    Log($"{runId} hung in {phase} after {silence:F0}s of silence");

// Tune the policy if the defaults don't fit:
var policy = WatchdogPolicy.Default with { QuietSeconds = 20, TickSeconds = 5 };

Plan tracking — TodoWrite / update_plan

driver.OnRunEvent += (id, e) =>
{
    if (e is CliRunEvent.PlanUpdated p)
        foreach (var item in p.Items)                 // PlanFrameItem: Id, Title, Status
            board.Upsert(item.Id, item.Title, item.Status); // pending / active / done
};

Quota (optional add-on)

The module brings the mechanism — escalation caching, persistence, cap-enforcement — plus two built-in probes. ClaudeOAuthUsageProbe calls the usage endpoint the Claude Code CLI itself uses (via the sign-in token it stores) and returns real server-side percent and reset times for the 5-hour, weekly and model-scoped windows. CodexSessionLogProbe reads the freshest rate_limits entry from Codex's session rollout logs — no process spawned, no quota spent; the data is as old as the last Codex run, so pair it with the free event harvest below. Gemini is deprecated (no probe); Antigravity exposes no quota surface to probe today. The IQuotaProbe contract stays open for your own probes.

var quota = new QuotaService(
    probes: [new ClaudeOAuthUsageProbe(), new CodexSessionLogProbe()],
    options: new QuotaCacheOptions
    {
        DefaultTtl = TimeSpan.FromMinutes(10),
        EscalationTiers =
        [
            new QuotaEscalationTier(90, TimeSpan.FromMinutes(2)),    // ≥90% → every 2 min
            new QuotaEscalationTier(97, TimeSpan.FromSeconds(30)),   // ≥97% → every 30 s
        ],
    });

QuotaReport report = quota.GetWithBackgroundRefresh();   // cached now; refreshes stale entries
foreach (var s in report.Snapshots)
    Console.WriteLine($"{s.CliType}: {s.MaxUsedPct:F0}% (plan {s.Plan})");

Bring your own probe, harvest events for free, cap a CLI

// A probe is one delegate behind the contract (HTTP call, CLI scrape, …).
IQuotaProbe claudeProbe = new DelegateQuotaProbe("claude", async ct =>
    new QuotaSnapshot
    {
        CliType = "claude", Plan = "Max",
        Windows = [ new QuotaWindow { Label = "5-hour", UsedPct = pct, ResetAt = reset } ],
    });

// A run already emits RateLimitObserved — feed it in for free, no probe spawn.
driver.OnRunEvent += (runId, e) => quota.Observe("claude", e);

// Declare a cap once; check a cheap gate before picking up work.
quota.Cap("claude", stopAtPercent: 97);
QuotaGate gate = quota.Gate("claude");
if (!gate.Allowed) Skip(reason: gate.Reason, retryAfter: gate.RetryAfter);

Wait for a nearby quota reset — optional, off by default

A quota-limit failure normally follows the application's existing failure or CLI-fallback route. Set WaitOnQuota.Enabled to add one branch: the runner refreshes quota for that CLI and waits only when the probe returns a plausible future reset within Threshold (30 minutes by default). It stops the exhausted process, emits QuotaWaitStarted with the reason and reset time, waits asynchronously, emits QuotaWaitEnded, and restarts the same request. Unknown quota, a failed probe, or a later reset leaves the existing route unchanged.

var quota = new QuotaService(
    probes: [new ClaudeOAuthUsageProbe(), new CodexSessionLogProbe()]);

var runner = new CliRunner(new CliOptions
{
    WaitOnQuota = new WaitOnQuotaOptions
    {
        Enabled = true,
        Threshold = TimeSpan.FromMinutes(30),
        QuotaService = quota,
    },
});

runner.Codex.OnRunEvent += (runId, e) =>
{
    if (e is CliRunEvent.QuotaWaitStarted wait)
        ShowStatus($"Waiting for quota reset at {wait.ResetAt:t}");
    else if (e is CliRunEvent.QuotaWaitEnded)
        ClearStatus();
};

Metrics & optional rendering

The recorder rebuilds metrics from the same typed events your UI, watchdog or quota layer already receives — no second data source. The optional rendering package is separate: core never references it, so non-UI consumers don't pay for Markdown parsing or Markdig.

using CodingAgentRunner.Metrics;
using CodingAgentRunner.Rendering;

var recorder = new RunMetricsRecorder();
driver.OnRunEvent += (_, e) => recorder.Observe(e);

driver.OnFinished += (runId, _) =>
{
    RunMetrics metrics = recorder.Build();
    Console.WriteLine($"first output: {metrics.TimeToFirstOutputMs / 1000.0:F1}s");
};

IReadOnlyList<RenderedLine> lines = MarkdownRenderer.ToLines(agentMarkdown);
string html = string.Concat(lines.Select(HtmlRenderer.SpansToHtml));

Injected link policy — one renderer, any app's link rules

The renderer emits link spans that carry a raw target and a LinkKind (Url, FilePath, TaskRef, Anchor); it does not decide what the href becomes. You inject one LinkResolver delegate that turns a LinkSpec into a ResolvedLink (href, target, rel, data-attributes), so the same renderer serves any app's link policy. The default is LinkExtractor.WebDefault, and LinkExtractor.IsSafeUrl enforces an http/https/mailto allowlist — it rejects javascript: and data: targets.

// Decide per link what the href and target become.
LinkResolver resolve = spec => spec.Kind switch
{
    LinkKind.FilePath => new ResolvedLink(Href: $"vscode://file/{spec.RawTarget}"),
    LinkKind.TaskRef  => new ResolvedLink(Href: $"/tasks/{spec.RawTarget}"),
    _                 => LinkExtractor.WebDefault(spec),   // http/https/mailto only
};

string html = string.Concat(lines.Select(l => HtmlRenderer.SpansToHtml(l, resolve)));

Optional benchmarks

The benchmark project covers the host-side hot paths: adapter parsing, usage parsing, and optional rendering. It's not part of the normal test suite or release gate — run it manually when you want overhead and allocation numbers. It measures the library, not agent quality or CLI wall-clock time.

# optional: all benchmarks (full statistical run)
dotnet run -c Release --project benchmarks/CodingAgentRunner.Benchmarks -- --filter '*'

# one class
dotnet run -c Release --project benchmarks/CodingAgentRunner.Benchmarks -- --filter '*AdapterParsing*'

# fast smoke check
dotnet run -c Release --project benchmarks/CodingAgentRunner.Benchmarks -- --filter '*' --job dry

CLI performance data contract

CLI performance numbers should come from real CLI executions, not from the host-side BenchmarkDotNet project. Store raw runs as JSON, then render aggregate rows by scenario, CLI, model, thinking level, context token bucket and measured token usage.

{
  "schemaVersion": 3,
  "generatedAt": "2026-09-12T08:00:00Z",
  "runner": {
    "packageVersion": "0.0.0-local",
    "gitCommit": "replace-with-source-commit"
  },
  "environment": {
    "os": "Windows 11",
    "cpu": "replace-with-cpu",
    "memoryGb": 32,
    "network": "office-wifi",
    "workspace": "fixture-or-repo-hash"
  },
  "runs": [
    {
      "runId": "2026-06-26-codex-default-medium-simple-001",
      "scenario": {
        "id": "simple-question.001",
        "complexity": "simple",
        "promptHash": "sha256:replace-with-prompt-hash",
        "fixture": "none",
        "checks": ["answer-key"],
        "sourceTests": [
          {
            "path": "tests/CodingAgentRunner.Tests/Metrics/RunMetricsTests.cs",
            "name": "Recorder_DerivesTtfo_Ttsi_TurnWallClock_TokensPerSec_FromTheStream"
          }
        ]
      },
      "variant": {
        "cli": "codex",
        "cliVersion": "replace-with-cli-version",
        "model": "GPT 6 Astra",
        "modelId": "gpt-6-astra",
        "thinkingLevel": "medium",
        "contextMode": "clean",
        "contextBucket": "small"
      },
      "measurements": {
        "timeToFirstCliFrameMs": 0,
        "timeToFirstOutputDeltaMs": 0,
        "timeToFirstRenderedOutputMs": null,
        "wallClockMs": 0,
        "toolCalls": 0,
        "filesChanged": 0,
        "contextTokens": 22657,
        "inputTokens": 0,
        "cachedInputTokens": 0,
        "cacheCreationInputTokens": 0,
        "outputTokens": 0,
        "reasoningTokens": 0,
        "totalTokensUsed": 0
      },
      "measuredAt": "2026-09-12T08:00:00Z",
      "outcome": {
        "status": "completed",
        "passed": true,
        "exitCode": 0,
        "rubricScore": 1.0
      },
      "artifacts": {
        "transcript": "artifacts/cli-performance/simple-question.001/run.jsonl",
        "diff": "artifacts/cli-performance/simple-question.001/run.patch",
        "testLog": "artifacts/cli-performance/simple-question.001/test.log"
      }
    }
  ]
}

Group	Why it matters
environment	Separates machine, OS, network and fixture effects from model/CLI behavior.
scenario	Makes simple prompts, repo-reading prompts, small fixes and complex tasks comparable over time. Include sourceTests so each scenario links back to the source-level contract it exercises.
variant	Captures the exact CLI, model, thinking level, permission mode, context mode and context token bucket used for the run.
measurements	Records first CLI frame, total wall-clock, tool churn, touched files, context tokens, cached tokens, output tokens, reasoning tokens and total tokens used.
provenance + models	Dates every measurement environment, records the exact CLI versions, and separates current model ids from retained obsolete ids.
outcome	Prevents a fast failure from looking like a good result; pass rate belongs beside duration.
artifacts	Lets readers audit the transcript, generated patch and validation output behind every aggregate row.

Recommended aggregate key: scenario.id + variant.cli + variant.model + variant.thinkingLevel + variant.contextBucket. Publish n, median, p90, min/max, time to first CLI frame, pass rate, median contextTokens, token sub-counts and the top failure reasons for each aggregate. Keep unavailable current-model runs as explicit pending records; never fill missing measurements with synthetic timing values.

Push events vs pull streaming

Two shapes for the same events. The event handlers are best for multiplexing many concurrent runs through one handler (route by evt.RunId). StreamAsync is the ergonomic choice for a single run — await foreach, natural control flow, built-in cancellation. They coexist.

// PULL (single run): one await foreach — done.
await foreach (var e in driver.StreamAsync(request, ct))
{
    switch (e)
    {
        case CliRunEvent.OutputDelta d:  Console.Write(d.Text); break;
        case CliRunEvent.ToolStarted t:  Log(t.ToolName);       break;
        case CliRunEvent.RunEnded end:   return end.Outcome;     // terminal ends the stream
    }
}
// A spawn failure throws; cancelling ct stops the run AND ends the enumeration.