How we made a long-running AI agent for knowledge work 4× faster

This post is for engineers who run AI agents on knowledge work through an agent harness such as Claude Code. One run of such an agent can take 15 min and cost $4.70 while someone waits for the document. We show how we made those runs faster and cheaper with the same model.

Two examples of knowledge work that take minutes, not seconds:

mapping everyone developing a drug for a disease;
writing a diligence memo on one company.

We hand this work to a long-running AI agent, built from three parts:

Claude Code, a coding harness.
A shell: its Bash runs in a jailed shell over the user's connected drives, with programs for trials, drug labels, filings and the web.
A run, in which the agent uses programs, reads, files the sources it used and writes the document.

Terms

Knowledge work: A task whose output is a document in which every claim cites a source the agent opened and read.
Run: One task horizon for the agent: from the request to the delivered document.
Turn: One model call and the tool calls it makes. A run is many turns.
Study: The first phase: 21 runs of the Landscape Analysis task in 10 stages, S0 to S9. S0 is the original setup; each later stage adds changes and keeps those before it. Most stages ran two or three times.
After the study: The second phase: 8 single runs of the same task, L1 to L8, each carrying three or more changes.

We ran 13 experiments on three tasks:

Task	Requirements
Landscape Analysis Word document on one disease (Parkinson's)	Five required sections: the disease and its standard of care; the companies being tracked that have a drug for it; everyone else developing for it, at most ten rows; where the tracked company's drug sits against them; sources. Every data point carries a citation, and every claim is supported by its source.
Diligence memo Word document on one company	Every data point carries a citation, and every claim is supported by its source. Draws on the company's folder, which already holds about 38 documents filed by earlier runs.
Spreadsheet task A plan of changes to a deal pipeline	Short: about a minute. Its model timings are the basis of Experiment 2 (E2).

Task

Requirements

Landscape Analysis

Word document on one disease (Parkinson's)

Five required sections:

the disease and its standard of care;
the companies being tracked that have a drug for it;
everyone else developing for it, at most ten rows;
where the tracked company's drug sits against them;
sources.

Every data point carries a citation, and every claim is supported by its source.

Diligence memo

Word document on one company

Every data point carries a citation, and every claim is supported by its source.
Draws on the company's folder, which already holds about 38 documents filed by earlier runs.

Spreadsheet task

A plan of changes to a deal pipeline

Short: about a minute. Its model timings are the basis of Experiment 2 (E2).

Findings

Run time and cost fell about fourfold and tenfold. Landscape: 937 s→233 s at best, 321 s in the latest run. Cost price per run: $4.70→$0.49. Diligence memo: 759 s→602 s.
The model was held constant; the gains are structural. The runs used Gemini 3.8 Flash. Experiment 2 (E2) tested Qwen3 235B.
The model provider matters. The same model,Gemini 3.8 Flash, stalled on 40.0% of calls served through OpenRouter→Google AI Studio and on 0.4% through Google's API directly.
The model must be able to operate the harness. Qwen3 235B, the fastest model on a short spreadsheet task, never used the shell on the Landscape Analysis Word document task. The follow-up run sent to complete its document alone took 712 s and 120 model calls.
Repeat runs vary widely. Runs of one unchanged setup spread nearly as far as all runs together, so one run cannot rank a small change.

The work

The model is Gemini 3.8 Flash. Experiment 2 (E2) tested Qwen3 235B. The harness is Claude Code with its own file tools switched off, so it works only through the shell. A run ends when it delivers a document that passes a check on its footnotes.

The figure below shows every Word document task run. A study stage that ran more than once is shown as its mean.

Seconds per run, coloured by task. Landscape Analysis: the study, S0 to S9, then L1 to L8. Diligence memo: M1 to M4.

Experiments

Each experiment states a hypothesis, the change made to test it, what was measured, and the result.

Variance between runs

Run-to-run variance of an unchanged setup

rejected

Hypothesis: Two runs of an unchanged setup produce similar documents.
Change: Run several setups two or three times without changes.
Measured: Standard deviation between repeats against standard deviation across all runs, for each accuracy measure.
Result: For every measure, the spread between repeats was at least 80% of the spread across all 21 runs (table in Results). A single run cannot rank a small change.

Model and serving

Transfer of a model's harness use from a spreadsheet task to a document task

rejected

Hypothesis: In an earlier test, Qwen3 235B was the fastest model on a short spreadsheet task (median 27 s). If its skill with the harness transfers, it is also the fastest on the long Word document task.
Change: Run the Landscape Analysis Word document task on Qwen3 235B instead of Gemini 3.8 Flash.
Measured: Whether the document is delivered; time and calls.
Result: It never used the shell: it called shell programs as if they were separate tools, and delivered a 753-byte skeleton. The follow-up run sent to complete it took 712 s, 120 calls and 3.27M prompt tokens.

Effect of the serving route on stalled calls

supported

Hypothesis: Stalls depend on the route that serves the model, not on the model.
Change: Serve Gemini 3.8 Flash through three routes.
Measured: Share of calls that stall: over 15 s with under 300 output tokens.
Result: OpenRouter→Google AI Studio40.0%stalled, 8 of 20 callsOpenRouter→Google Vertex2.8%stalled, 8 of 284 callsGoogle's API0.4%stalled, 1 of 266 calls Runs now call Google's API directly.

Hedged LLM Requests

rejected

Hypothesis: An LLM request is one call to the model's API. If a request has sent nothing back for 8 s, sending a second copy of it and keeping whichever answers first removes stalls.
Change: The gateway sends the second copy of any LLM request silent for 8 s.
Measured: How often the copy fires and wins; run time.
Result: The copy fired 6 times and won 5. Runs took 808 s, 511 s and over 900 s. The long waits came after the first chunk, which is the model's encrypted reasoning: the stalls were most likely thinking, not queueing.

Structure around the model

Effect of a smaller context window on prompt tokens

supported

Hypothesis: Telling the harness the model has a smaller window makes it compact sooner, lowering prompt tokens per call without harming the document.
Change: Window 160k → 80k tokens.
Measured: Average and largest prompt per call; sources in the document.
Result: Average prompt per call 51.5k → 27.3k tokens; largest 101k → 47k. Total prompt tokens roughly halved, and the first 80k run had 26 sources, among the most in the study.

Effect of fast-failing tools on tool time

supported

Hypothesis: Tools that fail slowly cost measurable time.
Change: Refuse a path-less grep at once (it searched every connected drive for its full time budget); fetch several trial records per lookup.
Measured: Tool time lost per run.
Result: One run lost 142 s to nine drive-wide greps before the change; no later run lost time to them.

Effect of whole-word method matching on method loading

supported

Hypothesis: Runs load eleven unrelated methods because a stronger instruction in their context outranks the outline.
Change: Match a method's trigger words as whole words, and take a work order only from the method the request names.
Measured: Method-loading calls per run.
Result: A substring match on "diligence" had given every landscape run a company-intake order. Method-loading calls fell from 11–23 per run to 0.

Effect of pre-run inputs on the first turns

supported

Hypothesis: Fetching the inputs every run needs before the model starts removes its first turns.
Change: A request carries shell commands that run before the first model call; their output goes into the prompt.
Measured: Reads, tool time, calls, wall time.
Result: Reads 14 → 3; tool time 39 s→20 s. Best run: 233 s, 43 calls, 32 sources. In the study, a run freed from setup spent the saved turns on more research.

Effect of multi-subject tool calls on turns

supported

Hypothesis: Answering many subjects in one tool call reduces turns.
Change: Research programs answer a whole column of subjects as one table, each answer with its citation; parallel tool calls kept apart by id at the gateway.
Measured: Model calls per run.
Result: 62 calls before; 43, 43, 35, 30 in the runs after, together with Experiment 8 (E8).

E10

Effect of write-path repairs on memo run time

supported

Hypothesis: Slow memo runs are slow at writing, not research: a failed write is followed by a full rewrite.
Change: Fix heredoc quoting; stop the write guard reading document text as a redirect; send large heredocs to a file; make the footnote check name the lines to edit.
Measured: Time lost to refused writes; output tokens; wall time.
Result: One refused write cost about 250 s. After the fixes, output tokens fell 78k → 44k and the memo took 794 s→602 s.

E11

Effect of pre-converted documents on memo tool time

supported

Hypothesis: Converting and reading 38 documents one per call inflates context and time.
Change: Convert the folder's documents to text before the run starts.
Measured: Tool time; model calls.
Result: Tool time 79 s→28 s. With inputs supplied first (Experiment 8 (E8)), calls fell 121 → 85.

Instructions to the model

E12

Effect of a batching instruction on run time

rejected

Hypothesis: Telling the model to batch its work shortens runs.
Change: Add a batching paragraph; three pairs, each against a control started at the same moment.
Measured: Model calls, wall time, document length.
Result: Runs were slower (462 s→524 s): documents grew from about 11k to 19k characters. Calls fell (71 → 57 mean).

E13

Effect of command-combining instructions on turns

rejected

Hypothesis: Instructing the model to combine its commands reduces turns.
Change: Add the instruction to the request.
Measured: Shell turns per run.
Result: Mostly not followed. Behaviour changed when the inputs or a tool's answer changed (Experiments 6 to 11 (E6 to E11)), not when the instructions did.

Results

Stalled calls by serving route. A stall is a call over 15 s that returns under 300 output tokens.

ReferencesE3

Route to Gemini 3.8 Flash	Calls	Stalled	Share
OpenRouter→Google AI Studio	20	8	40.0%
OpenRouter→Google Vertex	284	8	2.8%
Google's API	266	1	0.4%

Spread between repeats of one setup against spread across all 21 runs.

ReferencesE1

Measure	Across all runs (SD)	Between repeats (SD)
Sources defined	8.46	8.02
Citation markers	22.99	20.44
Numbers cited, prose	0.125	0.167
Competitor rows	2.16	2.13
Document length	4.4	4.84

The Diligence memo Word document task, run by run.

ReferencesE10 E11

Memo run	Seconds	Model calls	Sources	Changes in the run
M1	759	85	–	inputs gathered; 38 documents read one per call
M2	794	81	–	documents as text up front; a refused write cost about 250 s
M3	602	83	21	the refusal fixed; footnotes fixed in place
M4	652	–	22	large writes spilled to a file; 65 footnotes

The Landscape Analysis Word document task after the study, run by run.

ReferencesE3 E8 E9

Landscape run	Seconds	Model calls	Sources	Changes in the run
L1	584	–	–	Gemini served by Google directly
L2	506	–	–	trial lookups batched as one table
L3	448	–	–	trial search fixed for multi-word queries
L4	321	62	36	inputs gathered before the clock
L5	233	43	32	the field read by condition; memos as extracts
L6	346	43	20	gathered results inline; one 91 s write
L7	363	35	23	parallel tool calls kept apart
L8	321	30	23	the write path fixed

Conclusions

A model's speed on one task did not predict whether it could operate the harness on another. In Experiment 2 (E2), Qwen3 235B, the fastest model on the spreadsheet task, never used the shell on the Landscape Analysis and delivered a 753-byte skeleton.
Stall rates depended on the serving route, not on the model. In Experiment 3 (E3), the same model stalled on40.0% of calls through one route and 0.4% through Google's API.
Inputs supplied in the prompt before the first turn removed the turns that fetched them. In Experiment 8 (E8), reads fell from 14 to 3 and tool time from 39 s to 20 s.
A tool that failed slowly cost its full time budget. In Experiment 6 (E6), one run lost142 s to drive-wide searches that a fast refusal now ends at once.
An unintended instruction in the run's context outranked the request. In Experiment 7 (E7), a substring match gave every landscape run a company-intake order; removing it took method-loading calls from 11–23 per run to 0.
The longest memo calls were rewrites after a refused write. In Experiment 10 (E10), repairing the write path took the memo from 794 s to 602 s.
Instructions on how to work did not shorten runs. In Experiment 12 (E12), a batching instruction cut model calls and lengthened the documents, and runs were slower. In Experiment 13 (E13), the model mostly did not follow an instruction to combine its commands.

Limitations

The effect of each change was not isolated. The study's stages are cumulative, and each run after it carries three or more changes. Tool time is attributed to a change directly from the traces; the model's share is inferred.
Factual accuracy was not measured. The checks establish that every claim cites a source the run opened. They do not establish that the claim is true.
The samples are too small to detect small effects. Each setup ran two or three times in the study, and each change ran once after it. Given the run-to-run variance in Experiment 1 (E1), a difference of a few minutes between two runs is within noise.

Run from 15 to 19 October 2026 as hedwigai workbook runs in harnessroom, the harness Claude Code, the model google/gemini-3.8-flash: through OpenRouter in the study, through Google's API afterwards.
Seconds, model calls, prompt tokens and sources are each run's own record; the traces, read through hedwigai's MCP tool get_trace, give each model call's milliseconds and tokens and each tool call. Cost price is the provider's rate for the tokens recorded.
Study stages with more than one run show their mean. Repeat spread is the pooled standard deviation within a setup; correlations are Spearman's, over the runs whose cost is known.
The organisation the work was done for is not named, nor are the companies it tracks.
The citation and claim checks are free tools: Citation Checker and Claim Checker.

More head-to-head evals

Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.

Measure	Qwen3 235B	GLM-5.2	Gemini 3.8 Flash	Claude Opus 5
Seconds to the plan, median	27	48	57	94
Plans written, of three runs	2	3	3	3

Read the comparison SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.

Measure	SubQ	jev	Decider	Nimble
Found, mean of five columns at section length	64%	65%	66%	68%
Mechanism of action, found at section length	57%	51%	61%	54%

Read the comparison Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.

Measure	ask	Apollo	treg
Usable leads	13 of 40	7 of 40	6 of 28
Cost per usable lead	$0.037	$0.27	$0.23

Read the comparison Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.

Measure	ask with jev	ask with Decider
Questions right, of 92	89	89
Wrong answers	1	2

Read the comparison Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.

Measure	jev	ask
Our 370 clinical decisions, right	94.6%	93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)	77.0%	74.4%

Read the comparison Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.

Measure	ask	Claude Opus 5.5
Questions right, of 95	59	63
Small companies right, of 20	13	9

Read the comparison Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.

Measure	realestate-us	Best of four assistants
Floor-plan rents right, of 26	26	23
Amenities claimed the listing doesn't state	12	129

Read the comparison Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.

Measure	ask	jev
Right, of the questions answered	98%	79%
Wrong answers, of 92	2	19

Read the comparison Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.

Measure	Trained localiser	Best of four frontier models
Mean overlap with the defect (IoU, 0 to 1)	0.421	0.109

Read the comparison