AIPOCH: What a Scientific Agent Harness Must Make Auditable

A research agent can produce a polished report while quietly losing the code version, source snapshot, failed experiments, and branch where its conclusion changed. AIPOCH’s recent open-source releases emphasize subagent delegation, artifacts, branching, queues, and token traces. The important question is whether those features create reproducibility rather than a more elaborate chat history.

AIPOCH exposes the relevant harness primitives; scientific credibility still depends on how runs, sources, and artifacts are recorded and replayed.

TL;DR

• A scientific harness should preserve evidence lineage, environment, decisions, failures, and artifacts.
• Branching is valuable only when parent state and divergence are explicit.
• Token dashboards measure resource use, not scientific quality.
• Every conclusion should be reproducible from immutable inputs and executable steps.

The AIPOCH project lists August releases for production subagent delegation, artifact navigation, session branching, queued follow-ups, and usage reporting. These map to real research needs, but each needs a stricter interpretation.

An artifact needs content hash, creator run, source inputs, code commit, environment lockfile, timestamps, and status. A branch needs a parent checkpoint and a reason for divergence. A subagent result needs its prompt, tools, citations, and rejected alternatives—not just a paragraph pasted into the parent.

Use an append-only evidence ledger:

claim -> source snapshot -> extraction
      -> code/notebook version -> run environment
      -> output artifact -> reviewer decision

Store failures too. Removing null results makes a research agent look cleaner while increasing publication bias. Cost telemetry can help compare workflows, but “fewer tokens” is not “better science.” Evaluate citation correctness, reproducibility rate, sensitivity to source changes, and reviewer disagreement.

Reproducibility starts before the model answers

The public repository makes orchestration inspectable; data, environment, and artifact provenance determine whether a run is reproducible.

Saving the final prompt and response is insufficient. Search results change, webpages are edited, APIs return new records, package resolvers select new builds, and hosted models are updated. A rerun needs immutable—or at least precisely identified—inputs. Snapshot permissible source material, record retrieval timestamps and queries, hash local files, pin code and dependencies, and identify the exact model and harness configuration used.

Some inputs cannot legally or practically be copied. In that case, store a durable citation, content hash, access conditions, extraction method, and enough metadata to detect that the source changed. Reproducibility does not mean publishing protected data. It means an authorized reviewer can reconstruct what the agent saw and distinguish source drift from reasoning drift.

The run manifest should be machine-readable:

run_id: exp-2026-08-22-014
question: "Does intervention X change outcome Y?"
parent_run: exp-2026-08-21-009
sources:
  - uri: https://example.org/dataset/v3
    sha256: "..."
code_commit: "4f2c..."
environment_lock: "sha256:..."
model: "provider/model-snapshot"
harness_version: "..."
tool_policy: "research-readonly-v2"
random_seeds: [17, 31, 43]

Without this layer, branching produces multiple conversations, not multiple comparable experiments.

Branches and subagents need scientific semantics

A branch should state which variable changes and which state remains fixed. “Try another approach” is not enough. Is the branch testing a different hypothesis, dataset filter, statistical method, prompt, or model? If several variables change at once, the resulting difference cannot be attributed cleanly.

Subagents can parallelize literature review, data validation, code inspection, and adversarial critique. They can also multiply correlated errors. Five agents using the same retrieval index and prompt assumptions are not five independent confirmations. Record shared dependencies, require each result to cite primary evidence, and assign at least one role to search for disconfirming evidence.

The parent should not silently merge conflicts. Preserve both claims, their sources, and the rule used to resolve them. A reviewer needs to see that two branches disagreed about an exclusion criterion, not merely the polished consensus paragraph produced afterward.

Queued follow-ups are operationally convenient, but they introduce state drift. Bind each queued instruction to the checkpoint it expects. If the dataset, branch, or code changes before execution, the harness should pause or create a new branch rather than applying an old instruction to new state.

Artifacts should be first-class, typed objects

“Artifact” can mean a PDF, notebook, chart, dataset, compiled binary, or narrative report. Treating all of them as generic attachments loses validation opportunities. Define types and required metadata. A chart should identify the data and plotting code; a dataset should carry schema and transformation lineage; a notebook should record execution order and environment; a report should map material claims to evidence.

Generated artifacts move through states: draft, validated, reviewed, superseded, and retracted. Never overwrite a validated artifact in place. Create a new version and preserve the relationship. A content hash detects byte changes, while semantic version notes explain why the change matters.

The UI should make lineage navigable in both directions. From a claim, reach the source extraction and run. From a dataset, find every chart and conclusion that depends on it. This is what enables targeted revalidation when a source is corrected.

Evaluate the harness with research-shaped failures

Completion rate rewards fluent reports. A stronger evaluation injects problems a scientific workflow must catch:

Failure	Expected behavior
Retracted or corrected source	Flags status and identifies dependent claims
Unit mismatch	Stops calculation or records explicit conversion
Data leakage between splits	Rejects evaluation and explains contamination
Non-executable notebook	Rebuilds cleanly or reports missing dependency
Contradictory studies	Preserves disagreement instead of averaging prose
Null result	Records it with the same visibility as positive results
Changed web source	Detects hash drift and requests revalidation

Measure claim-level citation entailment, source quality, artifact replay success, environment rebuild success, sensitivity to seeds and reasonable analysis choices, and time for a reviewer to audit a conclusion. Token use belongs beside these measures, not above them.

A practical adoption checklist

Begin with a narrow workflow whose ground truth can be reviewed. Require run manifests and immutable artifacts before introducing many subagents. Add branching only after the team can compare branches reliably. Export everything in open formats so the research record is not trapped in a chat UI.

Decide retention and access policy early. Research traces may include unpublished findings, licensed papers, participant data, credentials, and model-generated interpretations that should not be treated as facts. Separate metadata needed for audit from sensitive payloads, encrypt both appropriately, and log reviewer access.

Finally, plan for model and tool retirement. A historical run may depend on an unavailable hosted model. Preserve outputs and event traces, and document what can and cannot be replayed. Exact model determinism may be impossible; provenance still lets a reviewer understand the boundary.

Frequently asked questions

Does open source make a scientific agent reproducible?

It makes orchestration inspectable, which helps. Reproducibility also requires fixed inputs, identified models, environments, artifacts, and a successful replay process.

Are multiple subagents equivalent to peer review?

No. They may share models, sources, and assumptions. Human peer review and genuinely independent replication address different risks.

Should failed runs be retained forever?

Retention depends on sensitivity and policy, but failures relevant to reported conclusions should be represented in the research record. Metadata can sometimes be retained when raw data must expire.

Verdict

Scientific provenance is a specialized form of the trace discipline described in agent observability and debugging. Code and data execution should also inherit the isolation principles in our secure sandbox comparison.

Open-source scientific harnesses are useful because their orchestration can be inspected and extended. Their credibility, however, comes from immutable provenance and replay—not from the number of agents in a workflow. The best interface should make it easier to ask “which exact evidence changed this conclusion?” than to generate another summary.