Benchmarks
The numbers, and exactly how we got them.
Measured runs, each published with its method and its caveats. Where a benchmark hasn't been run yet, this page says so instead of showing you a chart.
Measured · 2026-07-27
Token savings vs a careful file+grep baseline
Three real coding tasks, same repo (commit 66ff0c7), three gather styles. Tokens = ceil(chars / 4); hybrid RAG off. The machine was under heavy load at run — latency was degraded, but token counts are load-independent.
Naive dump
Whole listed files + full contents of up to 30 grep-hit files — no paste cap.
Careful typical
Whole hot files + first 4k of up to 12 extra grep hits, 48k-char cap.
Ganvil MCP
Real tools: search + context_pack + outline + get_symbol (48k cap), via stdio.
| Task | Careful tokens | Ganvil MCP tokens | Careful ÷ MCP | % less |
|---|---|---|---|---|
| dedup | 12,000 | 3,305 | 3.6× | 72% |
| handle_msg | 12,000 | 3,476 | 3.5× | 71% |
| claim | 12,000 | 2,333 | 5.1× | 81% |
| Total | 36,000 | 9,114 | 4.0× | 75% |
Averages ~75% less input context vs a careful file+grep gather (per-task 71–81%). Against a naive whole-file dump it's ~11.8× fewer / ~91% less. Live-model generation wasn't re-run in this batch; prior live API runs showed prompt tokens down without a clear gen-time win on short completions.
Corpus: the Ganvil repository itself at commit 455bb84c5c57 (135 files), imported via gan import and served by gan mcp from the imported store. Run 2026-07-08.
A driver speaks MCP over stdio and issues a realistic mix of ten tool calls, recording each response's size. Each call is paired with a counterfactual: the bytes the equivalent read-grep-read workflow ingests, measured from the actual repository. For callers, that means the summed size of every file containing the queried name — which is what "grep, then read the hits" costs. Tokens estimated at 4 characters per token on both sides.
The counterfactuals are deliberately conservative. They exclude wrong-file reads, retries, and re-reads across turns, and they assume the grep workflow finds the right files on the first pass. Real agents don't.
| MCP GANVIL CALL | Response tokens | Counterfactual tokens | Saving |
|---|---|---|---|
| outline | 1,738 | 4,814 — read the file | 2.8× |
| get_symbol (one function body) | 889 | 4,705 — read its file | 5.3× |
| callers (resolve_rev, depth 1) | 141 | 14,820 — grep + read 8 hit files | 105× |
| impact (transitive + tests) | 2,012 | 15,210 — grep + read hit files | 7.6× |
| search (ranked, degraded mode) | 406 | ~5,700 — grep + read candidates | 14× |
| structural_grep (.unwrap() in ganvil-core) | 2,012 | ~22,200 — grep + read to filter false positives | 11× |
| context_pack (task bundle, 2k budget) | 2,012 | 26,420 — read top 6 relevant files | 13× |
| symbol_diff (HEAD vs worktree) | 14 | ~2,000 — read the text diff | 143× |
| blame_symbol | 74 | ~5,000 — read log -p of the file | 68× |
| log_search | 283 | ~1,500 — read full log output | 5.3× |
| Total | 9,581 | 102,410 | ~10.7×* |
*Session total for this counterfactual method only (tool surface vs assumed perfect read-grep-read). Product tagline uses the more conservative ~4× / ≈75% less input context from end-to-end gather benches—not “10× faster code generation.”
EFFICACY
Real-world savings sit above these figures.
Counterfactuals exclude retries and wrong-file reads; deterministic tool output turns repeat queries into prompt-cache hits; and context_pack replaces multiple LLM retries from your coding agent during exploration with a single call.
Headline claim we stand behind: ~4× less input (≈75%).
On end-to-end gather paths (MCP pack vs dump+grep), context tokens drop on the order of ~2–6×, often ~4×. That is about 75% less input context—not a promise that model decode wall-clock is 4× faster.
Per-tool table below is a different (stricter) method.
Each row compares one MCP response to a read-grep-read counterfactual. Some tools look extreme (callers, symbol_diff) because the counterfactual is many whole files. We do not market “10× codegen speedup” from those outliers alone.
What ~75% less input means for you.
Result: you write way more code with the same plan quota, or you reduce API costs for the same throughput—because less of every turn is spent paying to re-read the repo. How much depends on how input-heavy your sessions are (output-heavy jobs save less on total bill).
What we do not claim (yet).
We do not claim full E2E “write code 10× faster” wall-clock. Live short completions often show similar gen time; prefill savings matter more on large prompts. See method + caveats above.