The harness matters too
Coding benchmarks usually compare models. In practice, the harness around a model—its prompts, tools, context management, and stopping rules—also affects whether an agent completes a task.
FrontierHarness Eval measures that difference. We compared 9 production coding harnesses across 12 configurations and 30 agentic tasks, holding the model, tasks, and runtime constant. Every run used Kimi K3 and executed on Runta.
Results
Explore the full leaderboard at harness.runta.com.
| Category | Harness | Result |
|---|---|---|
| Highest pass rate | Codex | 66.7% pass rate |
| Best value | Pi | 60.0% pass rate · $2.43/pass |
| Cost leader | Exo Harness | $1.05/pass |
| Speed leader | DSH Minimal | 5m 41s |
Across all 360 cells: 209 successes, 151 failures. Field-wide pass rate is 58.1%; field-wide token-weighted cache hit rate is 92.4%.
Exo Harness ($1.05 per pass) and Pi ($2.43) were the least expensive. DSH Minimal had the lowest median runtime at 5m 41s. DSH Creator finished near the top on pass rate, cost, and time (63.3%, $3.28 per pass, and 6m 44s). Claude Code reached a similar pass rate but cost 5.3x the field median per pass. As discussed below, that result may reflect the interaction between the harness, model, and gateway rather than the harness alone.
The results show three useful patterns:
- Quality and cost can diverge. Claude Code passes 19 tasks, but reaches $18.34 in median effective cost per pass.
- Cache hit rate is not cost. A cached 300-turn failure can still burn more than a short cache miss.
- Vendor lineage did not produce an advantage here. Kimi Code, made by the same lab as the model, tied for seventh at 56.7%.
| Harness (configuration) | Pass rate | Median cost per pass | Cache, median cell | Median time |
|---|---|---|---|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| DSH Creator | 63.3% | $3.28 | 84.3% | 6m 44s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| DSH PTC | 60.0% | $4.58 | 87.2% | 7m 44s |
| DSH Standard | 60.0% | $3.46 | 86.5% | 6m 17s |
| Oh My Pi | 56.7% | $4.75 | 82.2% | 6m 46s |
| Kimi Code | 56.7% | $3.65 | 88.0% | 7m 56s |
| DSH Minimal | 56.7% | $4.72 | 84.6% | 5m 41s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
Why we built FrontierHarness
Runta is an execution layer for agents. To improve agent efficiency, we need to measure not only whether a model can complete a task, but how the harness turns model output into useful work.
Model evaluations establish a baseline. The harness supplies the tools and environment, and influences both task performance and cost.
Three questions shaped the evaluation:
- What does a task actually cost? Tokens and steps, measured directly from each trajectory.
- What does a failure cost? Agents spend the most money on the problems they cannot solve. A cost metric that only counts successes hides the majority of the bill.
- How efficiently does a harness use tools? Effective tool calls reduce unnecessary steps, improve accuracy, and save tokens.
How we built FrontierHarness
Benchmark scope
The benchmark covers 30 tasks (21 Terminal-Bench tasks and 9 DeepSWE tasks) across 9 harnesses: Claude Code, Codex, DeepSeek Harness, Exo Harness, Hermes, Kimi Code, Oh My Pi, OpenCode, and Pi. These run in 12 configurations for 360 total evaluations, with one canonical result for every task and harness-configuration pair. Outcomes use deterministic verifier-based pass/fail scoring, and first-turn cache reads are repriced consistently across harnesses to make costs comparable.
The execution model
Every harness and every task environment is installed once and captured as a golden Runta checkpoint. Each of the 360 trials is a fresh restore from that checkpoint: identical vCPU, memory, disk size, disk contents, and memory state. No trial inherits filesystem drift, a warmed page cache, or a partially-installed dependency from the trial before it.
A few rules make the numbers comparable:
- Benchmark tasks are never run before the formal evaluation. Implicit prefix caching means a debug run leaves the trajectory warm for hours. All debugging happens on a separate smoke task set.
- All 12 configurations run the same task in the same job, so comparisons within a task use the same environment.
- One attempt per cell. Retries are warmer by definition, and best-of-N selection would bias quality upward.
Why Kimi K3
We wanted a model that was not native to one of the leading harnesses in the comparison.
Using Claude would favor Claude Code’s native environment; using a GPT model would do the same for Codex. Kimi K3 gave us a more neutral starting point, although it does not remove every compatibility effect.
Kimi K3 is a frontier-class coding model with published token pricing and implicit prompt caching. We served it through Fireworks so every harness reached the same third-party inference backend.
All nine harnesses connect through a single gateway that speaks both the OpenAI Responses API and the Anthropic Messages API, so each harness talks its native protocol to identical weights.
Why K3 gives Kimi Code no home-field advantage
Kimi Code is built by the same lab as Kimi K3, but it did not have a native-path advantage in this evaluation.
Kimi Code’s kimi provider type returned a 404 against our gateway, so we configured it as type = "anthropic" and routed it over the Messages API. Its stream output also omits usage data, which we recovered from usage.record events in the session wire log.
Kimi Code tied for seventh at 56.7%, with a $3.65 median cost per pass and a 7m 56s median runtime. Its 88.0% median cache hit rate tied for the best in the field, but that did not translate into a higher pass rate or lower cost.
Finding: Claude Code’s cache hit rate is two different numbers
The leaderboard shows a median cache hit rate, but that number does not describe where the total token spend went.
Claude Code’s cache hit rate is 25.0% token-weighted and 67.8% by median cell. Every other harness in the field sits between 91.6% and 99.1% weighted. The dashboard plots the median; the total token bill is driven by the weighted number. Nothing on a normal leaderboard reconciles the two, and for Claude Code the gap is 2.7x.
The median, 67.8%, describes a typical cell and can be compared with the $18.34 median cost per pass. On that measure, Claude Code’s cache rate was 1.3x lower than Codex’s 88.0%. The token-weighted figure, 25.0%, describes the total input-token volume. It is heavily affected by the most expensive task, where Claude Code cost $64.36 with a 15.7% cache hit rate.
The weighted number is also extremely concentrated. Three of Claude Code’s nineteen successful cells carry 88% of its token weight, and one (python-statemachine-state-data-scoping, 24.6M input tokens at a 15.7% hit rate) carries 68% on its own. The 25.0% figure is approximately “the cache rate on one very large cell,” which is why it should not be read as a description of typical Claude Code behavior.
Why is the weighted rate so low?
One likely contributor is harness-model fit. Claude Code is designed around Anthropic models and their explicit cache-control semantics. K3 exposes an implicit prefix cache with different behavior, so the same caching strategy may not transfer cleanly. This is a property of the complete harness-model configuration, not necessarily a defect in Claude Code.
The trajectories also point to gateway behavior. In one archived Claude Code session, the cache read stayed near 18.4K tokens—the size of the system prompt—on most calls, while the growing conversation was billed at the fresh rate. On roughly one call in five, the same append-only prefix reached a 100% hit rate:
call fresh cacheRead total_in hit%
1 1174 18432 19606 94.0
4 19381 18432 37813 48.7
5 0 38345 38345 100.0 <-- same conversation, full prefix hit
8 23092 18688 41780 44.7
13 0 45277 45277 100.0
25 35474 18944 54418 34.8
26 0 55514 55514 100.0
That variation within one append-only conversation is difficult to explain as a stable property of the harness alone. It suggests that the Messages API path through our gateway contributed to the result.
We cannot yet separate the harness’s contribution from the gateway’s. The $18.34 and $64.36 figures describe the exact configuration we tested, not Claude Code in isolation. Separating those effects is the first item on the v1.1 list.
An example task
datacurve/python-statemachine-state-data-scoping is a feature request against fgmacedo/python-statemachine at a pinned base commit. Official DeepSWE pass rate: 38%. Median historical trajectory: 117 agent steps.
Solving this means touching the metaclass, the callback injection machinery, history restoration, pickling, the SCXML parser, and the diagram renderer, then keeping the existing suite green. Seven of the 12 configurations passed.
| Harness | Result | Cost | Turns | Cache | Time |
|---|---|---|---|---|---|
| Pi | pass | $2.50 | 90 | 98.2% | 39m |
| Codex | pass | $5.97 | 187 | 99.1% | 37m |
| Hermes | pass | $6.70 | 156 | 99.1% | 29m |
| Kimi Code | pass | $9.24 | 209 | 99.3% | 38m |
| DSH Standard | pass | $10.14 | 227 | 99.3% | 47m |
| DSH Minimal | pass | $10.49 | 259 | 99.1% | 44m |
| Claude Code | pass | $64.36 | 381 | 15.7% | 60m |
| Exo Harness | fail | $1.46 | 51 (cap) | 96.0% | 40m |
| Oh My Pi | fail | $2.71 | 86 | 98.3% | 42m |
| DSH Creator | fail (timeout) | $3.88 | 109 | 98.8% | 60m |
| DSH PTC | fail (timeout) | $12.95 | 285 | 99.5% | 60m |
| OpenCode | fail (timeout) | n/a | 0 | n/a | 60m |
This task illustrates how similar outcomes can have very different costs.
Pi and Claude Code both passed, with a 26x difference in price. Claude Code used 4.2x as many turns and recorded a 15.7% cache hit rate, compared with Pi’s 98.2%. The difference belongs to the complete harness-model-gateway configuration tested here; the current data cannot isolate each component’s contribution.
DSH PTC spent $12.95 on a failed run, more than six of the seven configurations that passed. DSH Creator failed at $3.88 while DSH Minimal passed at $10.49. Both used the same harness, model, and runtime, but different presets. Creator reached the 60-minute limit, so prompt and tool configuration affected both pacing and outcome.
Limitations and future work
Limitations
FrontierHarness v1.0 focuses on coding and terminal-based tasks. It does not yet measure computer-use agents or performance in domains such as mathematics, business, legal, health, biology, and other areas of knowledge work. Broader task coverage is needed before drawing conclusions beyond software engineering.
The benchmark also treats each harness as shipped. For opinionated harnesses such as Claude Code, we cannot modify internal caching behavior to improve cache hit rates or normalize caching across gateways. The reported cost gap therefore reflects the full harness-model-gateway combination, but it cannot isolate how much of that gap would remain if every harness were equally optimized for K3’s caching semantics.
Two further limits are worth stating explicitly. Turn limits are each harness’s own defaults rather than a common cap, so harnesses that stop early trade pass rate for cost in ways the leaderboard does not separate.
Future work
Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.
Conclusion
The harness is an important variable that model benchmarks usually leave unmeasured. FrontierHarness v1.0 compares that layer across one model, one task set, one runtime, and nine harnesses, while including the cost of failed runs.
The pass-rate differences are too small to establish that one harness is generally better than another. The cost differences are much larger. In the most extreme case, prompt-cache behavior accounted for much of the additional cost—an effect a model-only benchmark would miss. Future runs will test more models and separate harness behavior from protocol and gateway effects.