Quantized models don't fail loudly — they loop, fumble tool calls, and claim victories that never happened. Sancho rides alongside every run: it detects distress mid-episode and reroutes before the task fails. Run the cheap model. Keep the speed. Lose the failures.
$ pip install sancho && sancho serve sancho: shadow mode — watching, not touching ▲ 14 distress events this week 9 format-class → recoverable by corrective prompt 3 hard collapse → would have escalated 2 silent false-successes flagged estimated recovered tasks: 8 $ sancho activate ✓ reroute ladder armed — zero code changes
We benchmarked the same model at different quantizations inside a deterministic agent harness — every verdict from checkers, never model self-report, every run reproducible from a manifest hash. Perplexity saw nothing. The agent loop saw this:
| Configuration | Agentic tasks clean | What happened |
|---|---|---|
| 35B MoE · 2-bit quant, alone | 0 / 14 | Total collapse — loops, malformed tool calls |
| Same model · near-lossless, alone | 10 / 14 | Better — but it lied about one success |
| 2-bit quant + Sancho | 15 / 15 | Distress caught mid-run, rerouted, recovered |
Exploratory pilot data (N=1 per task), measured at a pinned runtime on two independent backends. Pre-registered, multi-seed confirmation in progress — every number on this page will reproduce with one command.
An OpenAI-compatible proxy between your agent framework and your model servers. Change one base_url — no agent code changes.
Malformed tool calls, degenerate loops, budget burn, logprob anomalies — runtime signals, scored for severity as the tokens stream.
Format sloppiness or cognitive collapse? One cheap constrained turn classifies the failure — because the fix depends on the disease.
Corrective prompt first, precision climb next, model escalation last — the cheapest intervention that completes the task, chosen live.
Which models see giants? A public leaderboard scoring model × quant × runtime on agentic reliability — derail rate, silent-derail rate, clean rate — from deterministic, reproducible runs. Vendor-independent. Never pay-to-place. Launching with the preprint.
Get notified at launchEverything a single box needs is free, forever. You pay when agents become something your business depends on across a fleet.
Early access is small on purpose: self-hosted teams with agents in production get priority. One email when the repo, the Index, and the finding go live.
Request early accessSwap this button for your form endpoint (Buttondown / Formspree) before launch.