Fable 5.1 Long-Running Agents: Tests, Recovery and Cost
SandBase's Fable 5.1 evaluation design: test a billing-client upgrade across two repositories, recover from tool failures, cap retries, and count accepted-task cost.
Your coding agent says a billing-client upgrade is finished after three hours. The client’s tests pass, but its caller in another repository still sends the old request format. Has Fable 5.1 completed the task? Not until both sides work together. This SandBase article uses that illustrative scenario to define what a developer should check before accepting a long run. It is an evaluation design, not a report of a completed multi-hour test.
Key takeaway
- Fable 5.1 is designed for multi-step coding and tool work; hours spent running do not establish completion.
- Test both the changed client and its callers, preserving the original compatibility assertions.
- After a tool timeout, check whether a write completed before retrying it.
- Compare the full bill for accepted tasks, including failures and retries, not cache-read prices alone.
Anthropic describes Fable 5.1 as a model for work that takes hours and spans multiple applications. Its product page names codebase-wide changes, code review, browser operations, asynchronous agents, and long-running knowledge work. For the release figures and API IDs, see the Claude Fable 5.1 overview. Here the question is narrower: what must a long-running agent deliver before you merge its patch?
The best way to evaluate the release is therefore a loop:
read context -> plan -> change -> test -> inspect -> recover -> deliverThe model may perform that loop more reliably, but the loop still belongs in your system design.
Anthropic’s release page plots Fable 5.1 performance against inference cost across several reasoning settings.
Why long tasks fail
Long tasks compound small errors. A missing constraint in the first plan can become a wrong file edit several turns later. A test failure can be “fixed” by weakening the assertion. A tool timeout can be mistaken for a successful write. A context summary can preserve the latest symptom while dropping the original acceptance criteria.
Fable 5.1’s positioning directly targets these failure modes. Anthropic says it tries to fix root causes instead of symptoms, reports progress, recovers when a step fails, and writes tests to check its work. Those are useful behaviors, but they are not guarantees. A production agent still needs observable state and explicit stop conditions.
Tools, budgets and acceptance rules for Fable 5.1
A harness is the surrounding program that manages tools, task state, and tests. One illustrative task configuration is:
{
"goal": "upgrade the billing client",
"acceptance": ["tests pass", "API remains backward compatible"],
"budget": {"minutes": 45, "tool_calls": 80},
"artifacts": ["plan.md", "diff.patch", "test-report.json"]
}Then separate five concerns:
- Planning: require a short plan before mutation.
- Mutation: restrict writes to an explicit workspace.
- Verification: run tests and inspect generated artifacts independently.
- Recovery: classify failures as code, environment, tool, or policy failures.
- Delivery: refuse to report success until acceptance criteria are machine-checked.
This structure turns “the agent worked for three hours” into an auditable run. It also makes model comparisons fairer: measure completed tasks, retries, tool calls, and human interventions rather than output quality from one prompt.
What the launch examples reveal
The reported GPU-kernel results show the kind of multi-step work that needs executable checks, not just plausible code.
Anthropic’s customer examples are revealing because they describe complete workflows. Millennium says Fable 5.1 traced a rare crash through a core dump and an external vendor library. MongoDB says the model researched a large services codebase, produced an extensible design, ran unattended for hours, and returned a visual walkthrough with evidence.
These are anecdotes, not independent benchmark results. They do illustrate the target behavior: persistence across phases, external evidence, and a handoff artifact that a human can review.
For your own evaluation, recreate that shape with repository tasks that include hidden regressions, incomplete documentation, flaky tools, and a required final report. A model that produces a plausible patch but cannot explain test coverage is not completing the job.
Cost and latency move to the system level
Anthropic’s cost section separates cache reads from other tokens and reports different savings for typical and highly agentic workloads.
Long-running agents spend money outside the final answer. Context reads, retries, tool calls, screenshots, browser sessions, and evaluator passes can dominate the bill. Fable 5.1’s cache-read price is $0.25 per million tokens, while input and output are listed at $10 and $50 per million tokens. Those numbers help, but they do not predict task cost by themselves.
Track at least:
- cost per accepted task;
- median and p95 wall-clock time;
- number of recovery loops;
- percentage of runs requiring a human;
- rollback rate after “success.”
The right comparison is not “which model is cheapest per token?” It is “which model gets a validated change into production with the least total supervision?”
Where the model still needs boundaries
Do not give a long-running agent unrestricted credentials because it can recover from errors. Use short-lived tokens, an isolated checkout, network allowlists, and approval gates for deployment or irreversible actions. Record tool inputs and outputs so a reviewer can reconstruct what happened after context compaction.
If Claude Code is running the tools, use the steps for switching Auto Mode back to manual approval and try the temporary-file exercise before assigning a cross-repository task. Check the tool’s approval settings separately from the model’s capabilities.
Also account for safety routing. Anthropic says most Claude applications route flagged cybersecurity and biology requests to other Claude models; direct API customers must configure its Fallback API. Do not assume every third-party endpoint automatically retries that way. Record the effective model or fallback information when the service exposes it; otherwise mark it unknown rather than assuming every answer came from the requested model.
Fable 5.1 matters if it makes the whole action loop more dependable. The engineering opportunity is to pair that capability with a harness that makes every plan, failure, test, and handoff inspectable.
How it compares with a normal coding agent
| Workflow | Short coding assistant | Fable 5.1-style long run | What the harness must add |
|---|---|---|---|
| Small function or explanation | Usually enough | Often unnecessary | Basic lint and unit test |
| Cross-repository change | Needs frequent human steering | Designed for longer autonomous phases | Workspace snapshot and acceptance tests |
| Rare production bug | May stop at the first plausible cause | Can inspect artifacts and keep tracing | Core dumps, symbol access, and evidence log |
| Browser or multi-app work | Tool-specific handoffs | Can coordinate several applications | Per-tool permissions and replayable trace |
The model is not automatically better for every row. A cheap, fast model is usually the right choice for a one-file edit. Fable’s premium makes sense when the cost of context loss, repeated handoffs, or missed root causes is higher than the extra inference spend.
An acceptance test for a cross-repository change
This is an evaluation design, not a completed benchmark. Suppose a billing client is used in two repositories and its old API must remain compatible. Freeze old request samples, expected responses, and protected tests before asking Fable 5.1 to upgrade the dependency.
| Checkpoint | Required evidence | What must not count as success |
|---|---|---|
| Before editing | Both revisions, dependency versions, existing failing tests | Attributing an existing failure to the new patch |
| After editing | Both patches, compatibility tests, build exit codes | Testing the client but not its caller |
| After a tool timeout | Original error and a check of whether the write completed | Repeating a write without checking its state |
| At delivery | Clean-checkout commands and results, unfinished items | Passing only with local leftovers or weakened assertions |
Give each candidate identical tools, a 45-minute window, and an 80-call tool budget. These are illustrative limits, not measured optimal settings. If the run reaches its cap without passing, preserve the patch and error log for a human decision instead of paying for indefinite retries.
For actual outputs and bills, see the same-task Astra and Fable 3D game comparison, whose results apply only to that sample. The separate article on why coding agents lose their goal covers general context failures; this page focuses on accepting or rejecting a Fable long run.
FAQ
Is Fable 5.1 just a more expensive autocomplete model?
No. Its intended unit of work is a multi-stage task involving tools and verification. For a small edit, however, the extra capability may not justify the price or latency.
Should I let it run unattended overnight?
Only inside a disposable or isolated workspace with short-lived credentials, network restrictions, checkpoints, and an approval gate for deployment. “Unattended” should describe supervision frequency, not unrestricted authority.
How do I compare it fairly with another model?
Freeze the repository revision, prompt, tool list, timeout, sampling settings, and acceptance tests. Report first-pass success, accepted-task success, total tokens, retries, elapsed time, and human interventions. Do not compare a Fable direct response with an Opus fallback without recording the route.
What should happen after a tool failure?
The agent should classify the failure, preserve the original error, make the smallest justified change, and rerun the relevant check. If it cannot distinguish an environment failure from a code failure, stop and ask for a human decision.
Check the route and bill before a trial
The Fable 5.1 model entry on SandBase lists the route anthropic/claude-fable-5.1, checked September 9. Confirm the selected endpoint’s request format there before starting. SandBase supplies access to the model; the repository checkout, browser, tests and deployment permissions in this example must be supplied by your application. A successful API response is not evidence that those tests ran.
Keep official list pricing separate from the actual bill. A $0.25-per-million cache-read price does not make every task 75% cheaper: new input, output, cache writes, tools, and failed retries still count. If the service does not expose an internal fallback model, record it as undisclosed rather than inferring identity from writing style.
Sources checked September 9, 2026: Claude Fable product page, Fable 5.1 / Mythos 5.1 release and customer accounts, and SandBase’s exact Fable route. This update clarifies the evaluation scenario and API responsibilities; no new long-running benchmark was performed.
