Runtime reliability for self-hosted agents

Your agent tells you it won.
Sancho reads the tape.

Silent failure happens at every scale — the frontier just clears the loud bar, so its failures stay silent while every runtime dashboard glows green. Sancho catches both mid-run and fixes them in place. Run the cheap model. Keep the speed. Lose the failures.

Every runtime signal was green. The agent had lied.

We gave a frontier agent 17 real bugs fixed after its training cutoff — impossible to memorize. It declared success on all 17. Checkers, never the model's self-report, caught what the runtime couldn't: it was wrong on 7. No loop, no crash, no malformed call, no latency spike. Only reading what actually ran caught it.

Derailed Sancho intervened Recovered
Frontier agent
claimed done
17/17
Declared success on every post-cutoff bug.
Actually wrong
every signal green
7/17
The scored tasks were the frontier's own failures — a ceiling, not a rate. Every cheap detector misses a silent false-success by construction.
Caught + repaired
+ sancho
pass
One wrong “done” caught blind, repaired to pass the hidden tests — no answer key.

17 real bugs merged after the model's training cutoff — impossible to memorize. Verdicts from checkers, never model self-report; generation sealed from scoring. The 7 the frontier scored wrong were its own failures, so 41% is a selection-biased ceiling, not a general rate. Exploratory (T2), N=17, single seed. Pre-registered multi-seed confirmation in progress — every number replays from a manifest hash.

It isn't one vendor

Five frontier models. Every one lied about the same thing.

We ran five frontier models as coding agents on fresh, contamination-controlled bug-fix tasks — Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, Grok 4.5, and DeepSeek v4. Every vendor silently claimed success on a task it had actually failed: each 1 of 14, combined 5 of 70 ≈ 7% (exploratory, single seed). And all five tripped on the same archetype — the partial fix that passes the obvious test and fails the hidden second contract test.

Five frontier vendors
run as agents
5
Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro · Grok 4.5 · DeepSeek v4 — on fresh, contamination-controlled tasks.
Silent false-success
every vendor, each 1/14
5/70
≈ 7% (exploratory, single seed). Not one outlier — a shared, systematic blind spot.
The same failure
one archetype
1 test
The partial fix that passes the obvious test and fails the hidden second contract test.

Five frontier coding agents on unbiased, contamination-controlled bug-fix tasks (not frontier-selected): each 1 of 14 scoreable, combined 5/70 ≈ 7%. Verdicts from execution checkers, never model self-report. Exploratory (T2), single seed; a balanced multi-seed run supersedes.

In our newest held-out measurement on real repository tasks, 3 of 10 “done” claims from a frontier coding agent were false (exploratory, N=10, single agent).

Shrink the model, and the failure flips — loud.

Hand those same 7 bugs — the ones the frontier claimed done and got wrong — to a model that fits a 16 GB gaming GPU (Qwen2.5-Coder-7B). It fails all 7 too. But where the frontier lied quietly, the small model breaks in the open: malformed tool calls by turn 3. The cheap detectors that miss the frontier's silent lie catch this one instantly.

Malformed by turn 3 Sancho caught it Rerouted
16 GB local agent
failed loud
7/7
Malformed tool calls by turn 3 — failed all 7, and never once claimed done.
Silent false-success
the frontier: all 7
0/7
Zero silent lies. A small model fails in the open — nothing hides from the runtime.
Caught + rerouted
+ sancho
caught
The cheap detectors flag the collapse in ~3 turns and step in — no broken result ships.

The 7 post-cutoff bugs the frontier claimed done and got wrong — each with a verified fail→pass test. Native tool format, Qwen2.5-Coder-7B (Q8_0), single seed, N=7. “Malformed” = never emits a valid tool call and can't recover — the kind of failure every cheap detector sees. Exploratory (T2).

Silent-dominated at the top of the ladder, loud-dominated at the bottom. Sancho catches the whole ladder.

Whatever you run, Sancho meets it there

Silent failure occurs at every scale; frontier failure is silent-dominated because the frontier clears the loud bar. The failure you hit depends on the rung you run — and both rungs are first-class.

Running local or small models?

On a 7B–35B, quantized, budget-conscious stack the failures are mostly loud — and Sancho catches them live, in-line, before they burn your budget. On a 16 GB-class model we see malformed tool calls by turn 3 (live-proven through sancho serve shadow detection at severity=distress — a real run, qualitative), plus runaway loops and budget-grind.

malformed by turn 3 runaway loops budget-grind

Small doesn't mean always-honest, either: even full precision is not a clean actor — our F16 baseline false-claimed success on 4/14 tasks (exploratory, N=1/task). Loud vs silent is a spectrum, not a wall. And when a config collapses, the ladder can climb back: in exploratory runs, runtime detection + escalation recovered a fully-collapsed 2-bit config (0/14) to 15/15 clean (exploratory, N=1/task).

Run the cheapest model that works — because Sancho tells you the moment it doesn't.

Running frontier APIs?

Frontier models clear the loud bar, so the failure that's left is silent — a confident, wrong “done” with every runtime signal green. We ran five vendors as coding agents and every one silently claimed a task it had failed: combined 5 of 70 ≈ 7% (exploratory, single seed), all on the same partial-fix archetype.

silent false-success passes the obvious test fails the hidden one

Cheap detectors can't see it by construction — only reading what actually ran can. That's the verifier's job: execution verification, never the model's self-report. See the cross-vendor finding →

Keep the frontier's speed — and know when its “done” is real.

Same ladder, different rung: every model size has a characteristic failure mode, and Sancho catches each one — loud failures with in-line detectors, silent ones with execution verification.

Catch the failure. Fix it in place. Escalate only if it must.

An OpenAI-compatible proxy between your agent framework and your model servers. Change one base_url — no agent code changes. Silent failure occurs at every scale; smaller stacks fail loud first — malformed calls, loops — while the frontier clears the loud bar and its failures are silent-dominated: a confident, wrong “done.” Sancho reads the token stream and catches both, then climbs the cheapest ladder that completes the task.

Catch Fix Escalate
One line to install
# before
base_url = "http://vllm:8000/v1"

# after — sancho rides alongside
base_url = "http://sancho:8080/v1"
Catch — distress it reads
malformed tool calls degenerate loops budget burn logprob anomaly latency anomaly + silent false-success
Fix → Escalate — cheapest first
01
nudge
Corrective prompt — hands the agent the exact failing check to fix in place.
cheapest
02
climb
Precision climb — same model, higher fidelity.
mid
03
switch
Escalate to a stronger model — on the roadmap (the M2 auto-router). Last resort, hard collapse only.
most
The Flight Recorder — every episode, on the record

An append-only log of every run: each tool call, every distress signal Sancho caught, and every intervention it made — replayable, and exportable to your observability stack. It's the black box for your agents: when a result comes out wrong, you can see exactly what happened, the moment Sancho flagged it, and what it did about it. That record is how you prove a workflow can be trusted.

every tool call distress signals interventions + reroutes verifier verdicts OpenTelemetry export
Evidence-carrying verdicts

Every refutation ships the evidence that produced it: the failing probe, the execution transcript, and a manifest hash of the exact run. Nothing rests on our word — verdicts replay on your own machine with sancho verify --replay. A verdict you can re-run, not a score you have to trust.

failing probe execution transcript manifest hash sancho verify --replay

Big model, small model, or in between — the job gets done, and you can trust it's actually done. Sancho → trust your workflows.

The Sancho Reliability Index

Which agents quietly fail?

A public, vendor-independent leaderboard that ranks agents and models on the failure ordinary benchmarks miss: silent false-success — how often a model reports a job done when it isn't. Every score comes from deterministic, execution-checked runs — never the model's own self-report — so you can finally choose a model by how honestly it fails, not just how well it demos. Never pay-to-place. First rows: what removing memorization does to a frontier agent.

Agent × benchmarkClaimed doneResolvedSilent false-success
Frontier · post-cutoff · unmemorizable · N=1717/1759%41%
Frontier · SWE-bench Lite · memorizable · N=66/667%33%
Qwen3.6-27B · post-cutoff · N=71/70/7fails loud
Qwen3.6-35B-A3B · post-cutoff · N=70/70/7fails loud
Qwen2.5-Coder-7B · post-cutoff · N=70/70/7fails loud
Get notified at launch Exploratory (T2) · single seed · verdicts from execution checkers, never model self-report; generation sealed from scoring. Frontier scored on all post-cutoff bugs (N=17) & SWE-bench Lite (N=6); local models on the 7 post-cutoff bugs with a verified fail→pass test (N=7), native tool format. The 7 the frontier scored wrong were its own failures, so 41% is a selection-biased ceiling, not a general rate. Local models fail loud — malformed calls or budget exhaustion the cheap detectors catch — so we report them as loud failures, not a silent rate. Full model × quant × runtime Index lands with the Session-B preprint.

How we keep ourselves honest

The whole point is a number you can trust — so the method is fixed before the data lands, and the misses get published too.

Measurement discipline
  • Thresholds are pre-registered before we see the data.
  • Held-out sets stay sealed until scoring.
  • A six-class lie taxonomy — partial fix, symptom mask, test-gaming, wrong locus, regression, spec-reinterpretation — with per-class reporting coming.
  • We publish our null results.
Roadmap
  • Auto-escalation router · M2
    Picks the cheapest rung that completes the task, automatically.
  • Per-class reliability reporting
    Each lie class scored at pre-registered statistical power.
  • Learned lie-detection layer · in production now
    Trained on a 10,000-item synthetic adversary corpus.
  • Repair loop
    Refuted work re-attempted with the evidence in hand.

The squire becomes the governor

Everything a single box needs is free, forever. You pay when agents become something your business depends on across a fleet.

Squire
Serves one knight
Free · Apache-2.0
  • Full detection + reroute ladder
  • Shadow mode + one-command activate
  • Flight Recorder + OTel export
  • Single node, your GPUs
Governor Pro
Governs the estate
Per GPU / month
  • The verifier — catches silent derails
  • Fleet policies + SLO reporting
  • Community-calibrated routing law
Barataria
The island of cheap models
Enterprise
  • Compliance pack (EU AI Act)
  • Air-gapped install + on-prem verifier
  • Private calibration on your workloads

Ride along early

Early access is small on purpose: self-hosted teams with agents in production get priority. One email when the repo, the Index, and the finding go live.

Check your inbox — confirm your email and you're on the list.

No spam. One email when the repo, the Index, and the finding go live.