The Agent Harness Is a Performance Variable, Not Just a Wrapper

Teams often compare coding agents as if model name were the independent variable and everything around it were plumbing. It is not. Context selection, tool schemas, compaction, retries, repository memory, verification, and permission boundaries all change what the model sees and what actions survive.

OpenForgeRL treats the inference harness as part of the trainable and measurable system rather than invisible plumbing.

TL;DR

• Agent performance belongs to a model–harness–environment system.
• A stronger score may come from more tokens, hidden retries, better tools, or looser permissions.
• Harness comparisons must hold model, task, environment, budget, and completion criteria constant.
• Evaluate regressions per harness release, not only model release.

Recent research frames code itself as an agent harness and trains harness-native agents across environments. See Code as Agent Harness and OpenForgeRL. The useful takeaway is methodological, not a new leaderboard.

Measure these layers separately:

Layer	Example variable
Context	file selection, summaries, compaction threshold
Tools	schema quality, latency, error detail, parallelism
Loop	retry policy, planning, steering, stop condition
Memory	repository instructions, prior runs, learned facts
Verification	tests, lint, browser inspection, independent review
Authority	network, credentials, approvals, writable scope

A harness that runs three hidden retries has a different budget from one that exposes the first attempt. A harness with unrestricted internet may solve more tasks while violating the intended security envelope. Report both capability and authority.

For a fair experiment, pin the model snapshot and reasoning effort, container image, repository commit, task text, time/token/cost budget, network policy, and pass criteria. Save event traces and patches, then classify failures by context, reasoning, tool, environment, verification, or policy.

Run paired trials after every harness change. A “minor” compaction or file-search update can improve one repository and regress another without any model release.

Why the wrapper changes the answer

The survey provides a second research perspective on executable, verifiable, and stateful harness design alongside OpenForgeRL.

A model does not inspect a repository directly. The harness decides which files become tokens, how tool results are summarized, whether errors are visible, and when old context disappears. It also chooses the action vocabulary. A precise search_symbol tool and a generic shell tool induce different plans even when they can eventually retrieve the same bytes.

Loop policy matters just as much. Does the system plan before editing? Can it run tests after a failure? Does it see the full error or a truncated tail? When does it ask for approval? What counts as completion? A harness that stops after producing a patch is solving a different task from one that must pass tests and inspect the rendered page.

These choices interact. More context can help repository understanding but bury the relevant constraint. Aggressive compaction saves tokens but may discard an earlier user correction. Parallel tool calls reduce latency but can produce results against different workspace states. Hidden retries improve headline success while making cost and reproducibility harder to interpret.

Specify the experimental unit

Before comparing systems, write down what is held constant and what is allowed to vary. The cleanest experimental unit is one task attempt in a fresh, pinned environment. Randomize task order, use multiple trials where sampling is nondeterministic, and prevent one attempt’s generated files or memory from leaking into the next.

The result record should include:

• model identifier, reasoning setting, and sampling parameters;
• harness commit and full configuration;
• repository commit, container image, and dependency state;
• initial instructions and all human steering;
• network, secret, filesystem, and approval policy;
• tool calls with latency, errors, and truncated output markers;
• token, time, compute, and monetary budgets;
• final patch, tests, side effects, and independent grading result.

If one harness receives repository-specific instructions and another does not, report that as a treatment variable. If a person rescues a stalled run, count the intervention and its time. “Human in the loop” is not a constant unless the loop is specified.

Use a scorecard that resists benchmark theater

Pass rate is important, but it hides different operating profiles. Add at least these dimensions:

Metric	What it reveals
Verified task success	Whether the requested behavior actually works
Unsupported-change rate	Scope creep and accidental edits
Human intervention	Operational burden hidden by final success
Cost and latency distribution	Tail behavior, not just averages
Security-policy compliance	Whether capability stayed inside authority
Reproducibility	Whether another run/reviewer can explain the result
Recovery quality	Behavior after tool, test, or network failure

Grade with tests that were not exposed to the agent when possible, plus targeted review for security and maintainability. Public tests alone encourage patches that satisfy visible examples while violating the underlying requirement. Conversely, avoid opaque grading that cannot tell the harness team why it failed.

Report denominators and uncertainty. Ten handpicked tasks cannot support a universal claim. Segment by task type—bug fix, feature, refactor, frontend, dependency update, operations—because a harness optimized for fast code search may not be best at browser verification or long-running incident work.

Attribute failures to the right layer

Failure labels are imperfect, but they make iteration more disciplined:

• Context failure: the needed instruction or file was absent, stale, or lost during compaction.
• Reasoning failure: evidence was present, but the conclusion or plan was wrong.
• Tool failure: the schema, implementation, latency, or error handling prevented progress.
• Environment failure: dependencies, credentials, services, or fixtures were unavailable.
• Verification failure: the system stopped without testing the relevant behavior.
• Authority failure: policy blocked a necessary action or permitted an unsafe one.
• Interaction failure: unclear steering, approval timing, or status communication caused divergence.

Do not use these labels to excuse failures. Use them to select the next intervention. Adding a larger model will not fix a tool that returns an empty error. Adding more retries will not fix a test environment that is permanently missing a service.

Review a sample of successful runs too. Some “passes” rely on accidental environment state, broad credentials, or tests that do not cover the requirement. Near misses often reveal harness regressions earlier than the aggregate pass rate.

Harness optimization has a security frontier

Capability can rise when the harness gains internet access, writable scope, stored credentials, or automatic approvals. That is not a free performance improvement. Plot success against an authority envelope and compare systems at equivalent permissions.

Use task-scoped credentials and explicit network destinations. Record denied actions as data: repeated attempts to leave the allowed scope may indicate poor planning or a missing legitimate capability. Never “fix” a benchmark by granting an administrator token unless the real deployment would do the same.

Verification also has authority. Running arbitrary repository tests may execute untrusted code. A strong harness separates reasoning, execution, and secrets; inspects project instructions; and escalates consequence-changing actions. Security policy is part of the benchmark configuration, not environmental noise.

How to improve a harness without fooling yourself

Create a regression suite from real failures, but remove customer secrets and preserve the causal structure. For every change, state the expected mechanism: better symbol search should reduce context misses, while richer tool errors should improve recovery. Run paired trials on both the target failures and a broad holdout set.

Version prompts, schemas, compaction logic, repository-memory rules, and graders like code. Roll out gradually and retain trace comparison. A gain that doubles token use or human interventions may still be worthwhile, but the tradeoff should be visible.

Beware optimization against one model snapshot. Tool descriptions and context strategies that suit one model may regress another. Maintain compatibility tests across the models you actually support, and treat model upgrades as joint system releases.

Frequently asked questions

Is the harness more important than the model?

There is no universal ranking. Weak reasoning cannot always be orchestrated away, and a capable model can still fail inside a poor harness. Measure the deployed combination.

Can benchmark results from two vendors be compared?

Only when task versions, budgets, authority, environment, verification, and human intervention are comparable. Otherwise the numbers describe different experiments.

What is the first harness metric to add?

Verified success with complete cost, intervention, and trace data. It gives a baseline from which context, tool, and loop changes can be evaluated honestly.

Verdict

To place these measurements in an architecture, read why production agents need a runtime layer. The authority dimension should be evaluated against explicit production agent guardrails, not treated as an unreported benchmark advantage.

The harness is part of the product’s intelligence in practice, even when it contains no learned weights. That does not make model quality irrelevant; it makes single-variable marketing comparisons unreliable.

Choose a harness by the work it completes correctly inside your authority and budget envelope. The best result is not the patch that passes once—it is the system whose success and failure you can explain, reproduce, and improve.