Why agents quit early

We ran Qwen3.5-9B, a nine-billion-parameter open model, as the agent in the opencode harness on three research tasks from our harness benchmark: find US importers of black tea, list biotech companies that filed a Form D for new money, and list every original Form D notice in a fortnight. Four runs in all. Three ended with the work undone. One asked the user whether to carry on, one wrote a recap of what it had found and stopped, and one announced its next step and never took it.

All three quits came right after the same event. When a run’s context fills, opencode asks the model to summarise the conversation, swaps the transcript for the summary, and sends the line below. We measured why the quit comes there, what it looks like inside the model, and whether an edit to the model or a change to the line can prevent it.

In opencode a turn with no tool call is the answer: the harness takes it and the run ends. A logging proxy in front of the model kept every request the harness sent. We sent each one again 16 times with the harness’s own sampling settings and counted the replies that carried no tool call, at 12 ordinary turns picked at random, at all 5 turns right after compaction, and at the 3 runs’ last turns.

None of the 192 replies at ordinary turns stopped. After compaction the same model stops between 3 and 11 times in 16, about half on average. None of those runs had written its document yet, so each of those stops is premature.

Nor does the model stop more when the work is done. At the last turn of the one run that delivered, where stopping is right, it stops 8 times in 16, no more often than after compaction. The stop after compaction is a coin the harness’s line tosses, not a judgement about the work.

To see the choice inside the model we read its residual stream, the running sum every layer writes into, at the token where a reply either ends or opens a tool call. For each layer we took the replies after compaction, averaged those that stopped and those that went on, and called the difference the stop direction. Fitted on two runs and scored on the third, it tells the two apart with an AUROC of 0.99 from layer 7 on. Part of that is built in, since by that token the two kinds of reply have already written different text. What is not built in is where every other turn lands. We use layer 21, and each reply becomes one number: how far it sits along the direction.

The replies that went on after compaction average −24.3, and ordinary turns sit beside them at −20.7. The replies that stopped average +25.1, about 49 units away, and no ordinary turn comes near them. At the runs’ last turns the replies split the same way, those that went on in the go well and those that stopped spread from the ridge into the stop well, whether or not the work was done.

That is the shape of a bistable system: two attractors with a ridge between them. On an ordinary turn the state starts deep in the go well and never crosses. Compaction, followed by an offer to “stop and ask”, starts it at the ridge, which is what a stop rate of one half means, and sampling decides the side. The stop well is absorbing, because in opencode a reply that lands there ends the run.

The wells say how a run ends. They do not say whether the run was already lost, so we also mapped how each run explored. Bespoke Labs drew a model’s ideas as a path over a map of basins, clusters of related ideas, and asked how often it left one. We did the same for tool calls. Every data call of every run on a task, such as trade search companies or trade company shipments, becomes a TF-IDF vector of its command and arguments. MDS lays the calls on a plane and k-means groups them, with the number of basins picked by silhouette. On the black-tea task the map keeps the order of 77% of random triplets of calls, against 50.5% for a map that only knows which run each call came from.

Eleven runs worked the black-tea task: Qwen’s two and nine from DeepSeek V4.1 Flash, GLM-5.3-Flash and Particle-1 in our earlier benchmark. All of them use the same four basins: search for importers, read an importer’s shipments, and look up a supplier’s customers two ways. Ten failed, five by quitting early and five by running out of time. They did not fail by sticking in one basin, since every run visited at least three.

Qwen differs in degree. Its runs made 37 and 39 calls where the others made 46 to 149, moved to another basin on 16% and 25% of calls against 36% to 64%, and never reached the fourth basin. One run spent 77% of its calls searching for importers. Then compaction came, and it stopped.

Across all 17 tasks and 97 runs the pattern holds. Runs that failed spent less of themselves in the basin they began in (Spearman ρ −0.25, p = 0.01). How often a run moved basin shows no reliable link with failing (ρ −0.16, p = 0.12), and how long it stayed in its first basin has none with quitting early in particular (ρ −0.09, p = 0.36). Only 2 runs never left their first basin, and both delivered.

So the basins argue against the easy story that a model quits because it is stuck on one idea. Failing runs wander. Quitting is a separate event, and the harness sets the moment for it.

If the stop is a well, an edit can try to keep the state out of it. Every edit below works at layer 21 along the stop direction, through a forward hook on vLLM that touches every token. We measured each at the same turns with 8 replies a turn. Replies that loop until the 3,072-token cap are counted apart from clean stops, because a loop is the fourth kind of pause.

Subtracting the whole direction breaks the model. Every reply runs to the cap, repeating one word (“biotech biotech biotech”) or switching languages mid-sentence. A random vector of the same length changes nothing, which says the direction, not the size of the push, is what matters.

The clamp leaves a state alone unless it has moved past the bottom of the go well, and then pulls it back there. It removes the quit entirely: no clean stop in 40 replies after compaction. But 12 of those 40 now loop to the cap. The model also loses its correct stop, since at the delivered run’s last turn it no longer stops. Some replies show the conflict outright: the thinking reads “Let me present this clearly to the user”, and the action is a tool call.

A gentler clamp does nothing. Capping the projection at the ridge, or three quarters of the way to the stop well, leaves the quit at 0.55 and 0.53, with no loops. The stopped replies sit far beyond either cap, so if the direction were the switch, these clamps should have stopped them. It is a readout of the decision more than a lever on it. Only pinning every token to the bottom of the go well, a pull of up to 49 units, moves the behaviour, and a pull that large costs the model its correct stops and some of its coherence.

The cheaper change is in the harness. We sent the same 5 post-compaction requests to the unedited model with only the last line replaced, 8 replies each.

Either line halves the quit, and neither produced a loop. Half the stops remain, so the offer to stop is not the whole cause. The other candidate is the summary itself. The summary opencode writes is a report of work done, and a report of work done reads like a final answer to a model choosing whether to give one.

Qwen3.5-9B in bf16 served by vLLM on one GPU with a 64k-token window, behind a proxy that logged every request and reply. The four runs used opencode as the harness, on 13 November 2026. A turn counts as a stop when it carries no tool call. Resampling resent a logged request unchanged, with its own sampling settings, and read the stop from the reply. Residual states came from teacher-forcing each sampled reply through the same weights in Hugging Face Transformers, with thinking on, and reading every layer at the reply’s last token.

Edits ran with vLLM in eager mode. Under torch.compile the forward hook is silently skipped, and two different edits gave identical numbers until we noticed. The idea maps use the data calls of 97 runs on 17 tasks from our harness benchmark plus Qwen’s live runs, with MDS or t-SNE per task, whichever keeps more triplets in order.

For a harness author: do not end a compaction with an offer to stop. A plain instruction to continue halves premature quits at no cost, and a summary written as a handover rather than a report of work done is the next thing to try.

For a model trainer: the stop is a bistable choice that barely depends on whether the work is done. The model stops as often at a finished run’s end as at an unfinished run’s compaction. An activation edit can push the state out of the stop well only by pinning it at the bottom of the other one. That removes the right stops too, and some of what it pushes lands in a loop. Training has to teach the choice what “done” means.

For anyone running a small model as an agent: a quit right after compaction is a coin that came up tails. Resuming the run gives it another toss.