Silent failure happens at every scale — the frontier just clears the loud bar, so its failures stay silent while every runtime dashboard glows green. Sancho catches both mid-run and fixes them in place. Run the cheap model. Keep the speed. Lose the failures.
“I have completed the migration and verified that all tests pass.”
We gave a frontier agent 17 real bugs fixed after its training cutoff — impossible to memorize. It declared success on all 17. Checkers, never the model's self-report, caught what the runtime couldn't: it was wrong on 7. No loop, no crash, no malformed call, no latency spike. Only reading what actually ran caught it.
17 real bugs merged after the model's training cutoff — impossible to memorize. Verdicts from checkers, never model self-report; generation sealed from scoring. The 7 the frontier scored wrong were its own failures, so 41% is a selection-biased ceiling, not a general rate. Exploratory (T2), N=17, single seed. Pre-registered multi-seed confirmation in progress — every number replays from a manifest hash.
We ran five frontier models as coding agents on fresh, contamination-controlled bug-fix tasks — Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, Grok 4.5, and DeepSeek v4. Every vendor silently claimed success on a task it had actually failed: each 1 of 14, combined 5 of 70 ≈ 7% (exploratory, single seed). And all five tripped on the same archetype — the partial fix that passes the obvious test and fails the hidden second contract test.
Five frontier coding agents on unbiased, contamination-controlled bug-fix tasks (not frontier-selected): each 1 of 14 scoreable, combined 5/70 ≈ 7%. Verdicts from execution checkers, never model self-report. Exploratory (T2), single seed; a balanced multi-seed run supersedes.
In our newest held-out measurement on real repository tasks, 3 of 10 “done” claims from a frontier coding agent were false (exploratory, N=10, single agent).
Hand those same 7 bugs — the ones the frontier claimed done and got wrong — to a model that fits a 16 GB gaming GPU (Qwen2.5-Coder-7B). It fails all 7 too. But where the frontier lied quietly, the small model breaks in the open: malformed tool calls by turn 3. The cheap detectors that miss the frontier's silent lie catch this one instantly.
The 7 post-cutoff bugs the frontier claimed done and got wrong — each with a verified fail→pass test. Native tool format, Qwen2.5-Coder-7B (Q8_0), single seed, N=7. “Malformed” = never emits a valid tool call and can't recover — the kind of failure every cheap detector sees. Exploratory (T2).
Silent-dominated at the top of the ladder, loud-dominated at the bottom. Sancho catches the whole ladder.
Silent failure occurs at every scale; frontier failure is silent-dominated because the frontier clears the loud bar. The failure you hit depends on the rung you run — and both rungs are first-class.
On a 7B–35B, quantized, budget-conscious stack the failures are mostly loud — and Sancho catches them live, in-line, before they burn your budget. On a 16 GB-class model we see malformed tool calls by turn 3 (live-proven through sancho serve shadow detection at severity=distress — a real run, qualitative), plus runaway loops and budget-grind.
Small doesn't mean always-honest, either: even full precision is not a clean actor — our F16 baseline false-claimed success on 4/14 tasks (exploratory, N=1/task). Loud vs silent is a spectrum, not a wall. And when a config collapses, the ladder can climb back: in exploratory runs, runtime detection + escalation recovered a fully-collapsed 2-bit config (0/14) to 15/15 clean (exploratory, N=1/task).
▸ Run the cheapest model that works — because Sancho tells you the moment it doesn't.
Frontier models clear the loud bar, so the failure that's left is silent — a confident, wrong “done” with every runtime signal green. We ran five vendors as coding agents and every one silently claimed a task it had failed: combined 5 of 70 ≈ 7% (exploratory, single seed), all on the same partial-fix archetype.
Cheap detectors can't see it by construction — only reading what actually ran can. That's the verifier's job: execution verification, never the model's self-report. See the cross-vendor finding →
▸ Keep the frontier's speed — and know when its “done” is real.
Same ladder, different rung: every model size has a characteristic failure mode, and Sancho catches each one — loud failures with in-line detectors, silent ones with execution verification.
An OpenAI-compatible proxy between your agent framework and your model servers. Change one base_url — no agent code changes. Silent failure occurs at every scale; smaller stacks fail loud first — malformed calls, loops — while the frontier clears the loud bar and its failures are silent-dominated: a confident, wrong “done.” Sancho reads the token stream and catches both, then climbs the cheapest ladder that completes the task.
An append-only log of every run: each tool call, every distress signal Sancho caught, and every intervention it made — replayable, and exportable to your observability stack. It's the black box for your agents: when a result comes out wrong, you can see exactly what happened, the moment Sancho flagged it, and what it did about it. That record is how you prove a workflow can be trusted.
Every refutation ships the evidence that produced it: the failing probe, the execution transcript, and a manifest hash of the exact run. Nothing rests on our word — verdicts replay on your own machine with sancho verify --replay. A verdict you can re-run, not a score you have to trust.
Big model, small model, or in between — the job gets done, and you can trust it's actually done. Sancho → trust your workflows.
A public, vendor-independent leaderboard that ranks agents and models on the failure ordinary benchmarks miss: silent false-success — how often a model reports a job done when it isn't. Every score comes from deterministic, execution-checked runs — never the model's own self-report — so you can finally choose a model by how honestly it fails, not just how well it demos. Never pay-to-place. First rows: what removing memorization does to a frontier agent.
| Agent × benchmark | Claimed done | Resolved | Silent false-success |
|---|---|---|---|
| Frontier · post-cutoff · unmemorizable · N=17 | 17/17 | 59% | 41% |
| Frontier · SWE-bench Lite · memorizable · N=6 | 6/6 | 67% | 33% |
| Qwen3.6-27B · post-cutoff · N=7 | 1/7 | 0/7 | fails loud |
| Qwen3.6-35B-A3B · post-cutoff · N=7 | 0/7 | 0/7 | fails loud |
| Qwen2.5-Coder-7B · post-cutoff · N=7 | 0/7 | 0/7 | fails loud |
The whole point is a number you can trust — so the method is fixed before the data lands, and the misses get published too.
Everything a single box needs is free, forever. You pay when agents become something your business depends on across a fleet.
Early access is small on purpose: self-hosted teams with agents in production get priority. One email when the repo, the Index, and the finding go live.
No spam. One email when the repo, the Index, and the finding go live.