Why Multimodal Agents' Context Budgets Drift When Screenshots Arrive
Why image-heavy agent sessions can outgrow a compaction ledger: how screenshots shift retention boundaries and what engineers need to log.
Ask a coding agent to adjust a web interface: show it a screenshot, change the CSS, then capture another screenshot to verify the result. By the fifth or seventh image, those files are no longer incidental attachments. They record what the page looked like at each stage of the task.
A long-running agent cannot carry every message forever. It eventually summarizes older history and decides which raw messages, tool results, and images should remain beside the summary. If the ledger making that decision does not charge for images, the displayed budget stops representing the visual history that is actually retained.
That is what “context-budget drift” means here. The model’s context window has not necessarily changed; the agent’s estimate of how much visual history it kept has.
A screenshot becomes part of the task state
An error screenshot may explain one failure and become expendable after the fix. UI screenshots are different. One image records the state before a change, another confirms whether the button, spacing, or color moved as intended, and nearby text may identify the files as before.png and after.png.
Compaction is therefore not deciding whether to delete a few decorative images. It is deciding whether the next turn still has enough visual state to make a correct comparison. If a label survives but its image disappears, or a newer state is removed while older commentary remains, the retained task history becomes incomplete.
Codex’s image-input documentation lists error screenshots, UI designs, architecture diagrams, and before-and-after comparisons as normal inputs. Once those inputs enter a development loop, image retention becomes part of context management rather than a separate media feature.
“Out of context” can happen at three different layers
Figure 1: This article concerns the third layer: how much raw visual history remains after compaction.
- Model context window. The maximum input and output that one model request can accept.
- Compaction trigger. The point at which the agent decides that older history needs to be summarized or removed.
- Retained-history budget. The amount of original history that may be carried forward alongside the compaction result.
These layers influence one another, but they are not interchangeable. A model that accepts 128K, 200K, or one million tokens does not guarantee that the harness knows how much image history it retained after compaction. The window can stay constant while the bookkeeping error happens in the third layer.
Text entered the ledger; ordinary images did not
A Codex repair exposed a concrete gap in remote compaction: the retained-message budget counted text but did not count images in ordinary messages. Image-heavy history could therefore preserve more context than the budget represented.
This is what it looks like when a context system still carries assumptions from its text-only era. The runtime can send images to the model, yet the budget that decides how much history to preserve does not put those images on the same ledger.
Figure 2: Exact image pricing is not the central issue. The old path recorded ordinary images as zero; the repaired path assigns an estimated cost based on their size.
Text in this code path also uses an approximate token calculation, so the distinction is not “text is exact and images are estimated.” The meaningful difference is simpler: text reached the ledger; the images did not. Each retained image could widen the gap between the displayed balance and the multimodal input assembled for the next request.
Missing one image cost can shift the retention boundary
The compactor walks backward through recent history, retaining newer items while subtracting their cost from a fixed allowance. An image with no recorded cost lets that walk continue farther into the past. The ledger says there is room, even though the resulting multimodal history is already larger than the budget was intended to represent.
The public evidence confirms that one-way result: image-heavy history could retain more material than the budget represented. It does not prove that the agent always repeats clicks, forgets early, or compacts too soon.
The narrower failure is already enough to complicate diagnosis. Engineers cannot explain why the next request is heavier than expected, why so many screenshots survived compaction, or why a displayed remaining percentage disagrees with the request body by looking only at the model’s advertised context window.
Seven images cross a 64K retention budget
Codex 0.149.1 provides a boundary case that can be inspected in the repository. Its integration test places seven original-detail images into the conversation and assigns each one 10,000 estimated patch tokens before triggering remote compaction.
Those seven images already total an estimated 70,000 tokens, before any adjacent text is counted. The relevant retained-message budget is 64,000.
Figure 3: These are values from an official Codex integration fixture. They are not a universal cost for screenshots or an independently run SandBase benchmark.
The test covers three configurations:
| Configuration | Expected behavior across repeated compaction |
|---|---|
| Feature left at its default | Images remain in retained history |
| Feature explicitly disabled | Images remain in retained history |
| Feature explicitly enabled | Older images are trimmed against the budget |
The repair makes three related changes. Retained images now consume the existing size-based estimate; images and adjacent harness labels are treated as one unit at the truncation boundary; and a boundary image that does not fit prevents the compactor from filling the gap with older messages. Together they preserve a newer, semantically intact stretch of visual history whose estimated cost can be deducted.
The test checks which images survive into the next request, not whether the final model answer becomes better. The corresponding compaction_image_budget flag was still marked UnderDevelopment and defaulted to false in the referenced source. The evidence therefore supports “0.149.1 contains the implementation,” not “every hosted Codex session enabled it automatically.”
Multimodal-agent logs need to answer four questions
A single display such as “42% context remaining” cannot diagnose this failure. A useful compaction event needs enough detail to answer:
- How much text, tool output, and how many images existed before compaction?
- What detail setting, dimensions, and estimated cost were assigned to each image?
- Which images remained afterward, and which were removed?
- How far did the deducted budget differ from the actual size of the next request?
These are not four fixed fields that Codex currently promises to expose. They are the minimum observability questions a multimodal agent system needs to answer. For this Codex path, the exact version, remote-compaction version, image-budget feature state, and retained image count also matter.
An internal event can stay compact while remaining useful. Record the compaction timestamp, model and harness version, text-token estimate, image count, per-image estimate, identifiers for retained and removed images, and the final serialized request size. That one record lets an engineer compare the budget decision with what was actually sent. Without both sides, a graph of “context remaining” can show that a session changed but not whether text, images, or the retention policy caused it.
Once agents can see, reliability depends on visual-history management
The Codex artifact does not prove that Claude Code or every other visual agent has the same bug. It reveals a shared engineering obligation: any agent that treats screenshots as task state and compacts a long history must account for image cost when deciding where to cut.
Text, images, tool responses, and browser state all compete for the same working memory. A longer model window adds capacity, but it cannot repair a harness that still treats images as second-class entries in its retention policy. If you are comparing models with different context windows and multimodal support, the SandBase model catalog shows the currently available options; changing models still does not replace the agent’s own image budgeting and compaction logic.
For a closer look at why an advertised long window and a stable long-running task are different decisions, see our comparison of million-context models for agents.
A reliable multimodal agent needs more than a way to feed images into the model. Images must enter the same measurable retention budget as text.
Codex 0.149.1 repairs one implementation path. The larger dividing line is whether an agent can manage its visual history as carefully as it manages text.
Sources
- Codex rust-v0.149.1 Release
- Codex 0.149.0…0.149.1 Compare
- PR #40280: Budget retained images during remote compaction
- Codex Image Inputs Documentation
