Measured, not promised.
About half the cost per task — and cheaper on every task we ran.
312 live runs with Claude Opus 5.5 and hidden tests the agent never saw. The context engine and the prompt cache cut the cost of an accepted task by 49% — and every number on this page comes straight from the run logs.
The headline,
cost per accepted task by arm
Arm C — the context engine plus the prompt cache — costs about half of the baseline on both sets, at the same acceptance.
Cost per accepted task, by arm
Total measured cost ÷ runs accepted by hidden tests. Same model, same machine, same tasks.
Main set: 240 runs, $8.37 total. Harder set: 72 runs, $3.22 total. Arm R was not run on the harder set.
Data table
| Arm | Main set | Harder set |
|---|---|---|
| A · Baseline | $0.0405 | $0.0543 |
| B · + context engine | $0.0376 | $0.0513 |
| C · + context engine + cache | $0.0205 | $0.0284 |
| R · + research helpers | $0.0409 | not run |
Every task got cheaper — main set
Mean cost per run of each task (3 repeats), baseline A against arm C. Sorted by baseline cost.
- A · Baseline
- C · context engine + cache
Data table
| Task | Set | A mean | C mean | Change |
|---|---|---|---|---|
| h9_py_long_repair | held-out | $0.0641 | $0.0422 | -34% |
| h11_py_rounding | held-out | $0.0605 | $0.0344 | -43% |
| h12_node_format_bytes | held-out | $0.0524 | $0.0245 | -53% |
| h3_node_empty_roman | held-out | $0.0481 | $0.0415 | -14% |
| h5_py_word_count | held-out | $0.0474 | $0.0224 | -53% |
| h8_node_missing_test | held-out | $0.0470 | $0.0259 | -45% |
| h6_py_public_type | held-out | $0.0469 | $0.0205 | -56% |
| h2_py_large_repo | held-out | $0.0424 | $0.0121 | -71% |
| t7_node_feature | tuning | $0.0400 | $0.0226 | -44% |
| h4_node_median | held-out | $0.0395 | $0.0163 | -59% |
| t8_py_missing_test | tuning | $0.0389 | $0.0183 | -53% |
| t4_py_kv | tuning | $0.0355 | $0.0149 | -58% |
| t6_py_empty_fizzbuzz | tuning | $0.0343 | $0.0241 | -30% |
| h1_py_leap | held-out | $0.0340 | $0.0092 | -73% |
| t2_py_discount | tuning | $0.0324 | $0.0207 | -36% |
| t1_node_slug | tuning | $0.0322 | $0.0164 | -49% |
| h7_node_config | held-out | $0.0302 | $0.0174 | -42% |
| t3_py_config | tuning | $0.0297 | $0.0099 | -67% |
| h10_py_concurrent_edit | held-out | $0.0282 | $0.0089 | -69% |
| t5_py_add | tuning | $0.0267 | $0.0083 | -69% |
Every task got cheaper — harder set
Mean cost per run of each task (3 repeats), baseline A against arm C. Sorted by baseline cost.
- A · Baseline
- C · context engine + cache
Data table
| Task | Set | A mean | C mean | Change |
|---|---|---|---|---|
| x4_py_vat_symptom | harder | $0.0914 | $0.0381 | -58% |
| x3_node_semver | harder | $0.0772 | $0.0480 | -38% |
| x8_py_roman_strict | harder | $0.0516 | $0.0329 | -36% |
| x7_py_compat_rename | harder | $0.0483 | $0.0243 | -50% |
| x1_py_lru | harder | $0.0452 | $0.0242 | -47% |
| x2_py_csv | harder | $0.0435 | $0.0198 | -54% |
| x6_node_debounce | harder | $0.0423 | $0.0257 | -39% |
| x5_py_intervals | harder | $0.0349 | $0.0145 | -58% |
Same prompt size,
most of it read back from the cache
The context engine keeps the prompt small; the cache means the stable part is paid for once and read back at a twentieth of the price.
Where each run's input tokens come from
Mean input tokens per run, split into fresh input, tokens written to the cache, and tokens read back from it.
- Fresh input
- Cache write (billed 1.25×)
- Cache read (billed 0.05×)
Prompt tokens include cached reads and writes; they are split here, never added twice. Billing ratios from ADR 0010.
Data table
| Set · arm | Input | Fresh | Cache write | Cache read |
|---|---|---|---|---|
| Main · A | 6,584 | 6,584 | 0 | 0 |
| Main · B | 6,339 | 6,339 | 0 | 0 |
| Main · C | 6,341 | 7 | 1,378 | 4,955 |
| Main · R | 6,803 | 6,803 | 0 | 0 |
| Harder · A | 8,648 | 8,648 | 0 | 0 |
| Harder · B | 8,390 | 8,390 | 0 | 0 |
| Harder · C | 8,866 | 8 | 1,657 | 7,200 |
What one request costs, cache by cache
Six live requests to anthropic/claude-opus-5.5 via OpenRouter (2026-09-26), measured with the provider's own cost field.
A cache marker is required — there is no implicit cache. Writes bill at 1.25×, reads at 0.05×; the measured costs match those ratios.
Data table
| Scenario | Prompt tokens | Cache read | Cache write | Cost |
|---|---|---|---|---|
| No cache marker (same request twice) | 3,625 | 0 | 0 | $0.0146 |
| Marked, cold | 4,931 | 0 | 4,927 | $0.0248 |
| Marked, warm (same request) | 4,931 | 4,927 | 0 | $0.0012 |
| Later content changed, prefix the same | 4,931 | 1,767 | 3,160 | $0.0163 |
| Prefix changed (full miss) | 4,938 | 0 | 4,934 | $0.0248 |
| Breakpoint on a tool result, then continue | 6,001 | 5,982 | 0 | $0.0014 |
Nothing got worse:
every run accepted, fewer turns, similar time
Accepted by hidden tests: 312 of 312 runs
Each square is one run (three repeats per task and arm). Filled = the hidden tests passed on the final tree.
- Accepted
Every arm passed every task, so these sets cannot separate the arms on quality. We don't claim the arms are equal in general — only that none was worse than the baseline here.
Data table
| Group | Arm | Accepted |
|---|---|---|
| Main set · tuning (8 tasks) | A | 24/24 |
| Main set · tuning (8 tasks) | B | 24/24 |
| Main set · tuning (8 tasks) | C | 24/24 |
| Main set · tuning (8 tasks) | R | 24/24 |
| Main set · held-out (12 tasks) | A | 36/36 |
| Main set · held-out (12 tasks) | B | 36/36 |
| Main set · held-out (12 tasks) | C | 36/36 |
| Main set · held-out (12 tasks) | R | 36/36 |
| Harder held-out set (8 tasks) | A | 24/24 |
| Harder held-out set (8 tasks) | B | 24/24 |
| Harder held-out set (8 tasks) | C | 24/24 |
Turns per run — the context engine needs fewer
Share of the 60 main-set runs per arm that finished in 1, 2, 3 or 4 model turns. Mean on the right.
- 1 turn
- 2 turns
- 3 turns
- 4 turns
Published: average turns 3.05 → 2.50 with the context engine (B vs A, −18%). Means here are computed from the evidence log.
Data table
| Arm | 1 | 2 | 3 | 4 | Mean turns |
|---|---|---|---|---|---|
| A · Baseline | 0 | 6 | 45 | 9 | 3.05 |
| B · + context engine | 0 | 36 | 18 | 6 | 2.50 |
| C · + context engine + cache | 3 | 32 | 18 | 7 | 2.48 |
| R · + research helpers | 0 | 10 | 41 | 9 | 2.98 |
Time to finish, every run
Wall-clock seconds per run, one dot per run (jittered vertically). Ticks show the published median and p90.
- One run
- Median
- p90
Speed was not the goal of the context engine or the cache; medians stay within a few seconds across arms.
Data table
| Set · arm | Runs | Median s | p90 s | Fastest s | Slowest s |
|---|---|---|---|---|---|
| Main · A | 60 | 17 | 33 | 8.8 | 46.6 |
| Main · B | 60 | 16 | 31 | 6.8 | 38.0 |
| Main · C | 60 | 17 | 30 | 6.9 | 37.5 |
| Main · R | 60 | 17 | 34 | 7.8 | 42.0 |
| Harder · A | 24 | 19 | 41 | 11.3 | 57.1 |
| Harder · B | 24 | 17 | 35 | 8.8 | 39.1 |
| Harder · C | 24 | 16 | 40 | 8.8 | 41.6 |
How we measured
pre-registered, then run
The protocol and its thresholds were written down before any result was seen; results are reported as they came out.
Baseline
The P5 baseline agent, unchanged: no context engine and no cache markers.
+ context engine
A Rust engine parses the project with tree-sitter, builds a symbol graph and sends the model only the definitions a task needs.
+ context engine + cache
B, plus provider cache markers on the stable prefix (tools + system) and a moving breakpoint on the latest message.
+ research helpers
A plus research tools the model may call. It never called them in 60 runs, so no benefit could be measured — it stays off.
- Model
- anthropic/claude-opus-5.5 through OpenRouter, in every arm
- Machine
- One Mac (macOS 26.6, arm64); same tools and verifier for every run
- Main set
- 20 tasks — 8 tuning, 12 held-out — × 4 arms × 3 repeats = 240 runs (2026-09-27)
- Harder set
- 8 new held-out tasks (LRU eviction, RFC 4180 CSV, SemVer precedence, a symptom-only bug in a 200-file repo …) × 3 arms × 3 = 72 runs
- Acceptance
- Hidden tests the agent never sees, run on the final tree in a container with no network
- Task validity
- Hidden tests fail on every original project and pass on every reference solution
- Cost
- Total measured cost ÷ accepted runs, from the provider's own cost field
- Rule, fixed before running
- An arm can't become the default if it is worse than A on more than one held-out task
- Spend
- $8.37 for the main set, $3.22 for the harder set
What we did not measure
A benchmark is only as honest as its edges. These are ours.
- Any other product. There is no comparison with Cursor, Claude Code, Codex or anyone else — only our own baseline.
- Any other model. Every run used Claude Opus 5.5; other models were not measured here.
- Quality differences. Every arm passed every task (312 of 312), so these sets can't tell the arms apart on quality.
- Large, multi-step work. Tasks are small and medium; a much bigger set would be needed to see quality gaps.
- RAM and CPU during live agent runs. Only the core engine's memory was measured, in a local soak with no model calls.
- Statistical confidence of the success rate — undefined at this sample size.