Blog
hedwigaihedwigaihedwigai

Model editing

Qwen knows a rule applies and still stops. Can an edit to its weights fix that?

Qwen3.5-9B, a small open language model, makes the edit a document asks for and then stops, even when a standing rule needs a second edit. Once the first edit is written, its activations show that the rule applies. We tested four ways to change the model, and Claude Opus 5.5, a larger language model, wrote one of them.

A hedwigai workbook is a document kept together with its rules. Its notes are rules in plain English that every later edit must keep. Its schema lists the tables and their formula columns, and code checks every edit against it.

Some edits trigger a rule. Figure 1 shows one. When Tesla’s operating expenses change, a sentence that lists Tesla’s expense lines no longer adds up to the table. The rule then asks for a second edit that the request did not name. Research on reasoning about actions calls such an indirect effect a ramification: it follows from a constraint on the document, not from the action itself. We call the edit that carries it out a follow-on edit.

Figure 1 · A request whose edit triggers a rule

REQUEST

Change Tesla’s total operating expenses to $13.5B.

TABLE 3 · operating expenses

| Tesla, Inc. | 31 December 2025 | $12.7B $13.5B | $4.4B | 4.6% |

NOTE BENEATH THE TABLE

… Tesla’s “Total operating expenses” — research and development $6.4B, selling, general and administrative $5.8B, and restructuring and other $0.5B …

STANDING RULE

Prose that states a figure shown in a table, or says how a table figure is built, must agree with the table after every edit.

The three lines add up to $12.7B. After the requested edit they no longer add up to the table’s total, so the rule asks for a follow-on edit: change a line so that the three add up to $13.5B, or say that the table’s total differs from their sum.

Qwen3.5-9B is a small open model. Unedited, it tried the follow-on edit in 9 of 160 drafts that needed one, and it made it correctly in none. Larger models fail in the same way. On ActionReasoningBench, by Handa, Dolin, Kumbhar, Baral and Son, GPT-4o answered none of the questions that involve ramifications, and o1-preview answered 18.4% (arXiv 2406.04046).

Varam, Najafi and Atienza recently described weight-space programming, in which a language model reads a network’s weights and activations and writes the change to the weights itself, with no gradient step (arXiv 2611.13758). They found that it repairs faults with a clear cause, and that it does not improve a network that already works. We asked whether it, or a cheaper edit to the weights, makes Qwen carry out a rule’s follow-on edit.

We compared four changes to Qwen’s weights. A steering vector adds a fixed direction to the model’s activations at one layer. A rank-one edit writes such a direction into one weight matrix. LoRA is a cheap way to fine-tune: it trains a small low-rank change to some of the weight matrices. The fourth change is the one Claude Opus 5.5 wrote. We measured each on two documents. The Big Tech workbook is the one every method was built on, with 40 requests. The held-out workbook is a second document that no method saw while it was built.

We asked five questions

  1. Question 1How often does Qwen act on a rule?
  2. Question 2Does Qwen register that a rule applies?
  3. Question 3Can a steering vector or a rank-one edit make it act?
  4. Question 4Does fine-tuning with LoRA teach the rules?
  5. Question 5Can Claude Opus 5.5 write the edit?

Setup

We measured two workbooks

We served Qwen3.5-9B in bf16 with vLLM, with thinking off, at temperature 0.7 and top-p 0.95. Each prompt was the one hedwigai uses in production for a single edit. It holds the request, the workbook’s notes and the document. The model answers with a list of exact-text replacements. Code then applies them and rewrites every formula cell, as production does.

The Big Tech workbook holds the revenue, net income and operating expenses of ten companies in three tables, with prose, footnotes and 16 standing notes. For this study we added the text rule of Figure 1 as a 17th. We wrote 40 one-figure requests for it:

  • 20 trigger no rule, so the requested change is the whole answer.
  • 5 trigger the sort rule: Table 1 is kept in order of revenue, and the new figure moves a company past a neighbour.
  • 5 trigger the growth-sentence rule: a sentence names the companies with the highest and lowest net income growth, and the new figure changes one of them.
  • 10 trigger the text rule of Figure 1. In seven, a table figure changes and the prose must follow. In three, the request changes the prose and a table cell must follow.

The held-out workbook is a different document, built by an earlier run. It holds quarterly and annual revenue and net income for eight companies, from their 10-K and 10-Q filings. We wrote 16 requests for it: 8 trigger no rule and 8 trigger the text rule. No method saw these cases while it was built, and each method ran on them once.

A draft is fully right when it applies, does the request, passes the schema and keeps every rule. Code checks the sort and growth-sentence rules exactly. For the text rule, a rubric for each case was written before any run. Code checks that the right lines changed, and we read every draft that passed against its rubric. A draft tries the follow-on edit when it changes more than the requested cell in the place the rule points to.

An edit can also damage the model elsewhere. KL divergence measures how far one probability distribution is from another, and it is zero for identical ones. We measured this drift as the mean KL divergence between the edited and unedited model’s next-token distributions over 30 passages of Wikipedia text (wikitext-2), 768 tokens each. Before the runs we set the bar for success: on the Big Tech workbook, at least half of the triggered drafts try the follow-on and at least 90% of the untriggered drafts stay fully right; drift stays under 0.1 nats.

Question 1

Qwen keeps a rule only when the edit leaves it alone

With the production prompt, Qwen3.5-9B made 155 of 160 untriggered drafts fully right (97%). On the 160 triggered drafts it tried the follow-on 9 times and got it right in none. It changes what the request names and nothing else.

Adding the notes and the schema to the prompt makes the model try more often, in 74 of 160 triggered drafts. It keeps the rule in at most 8, and its untriggered drafts fall to 91 of 160 fully right (57%). The extra text makes Qwen edit more everywhere.

Figure 2 · Qwen tries a follow-on edit only when it is told to

  • Tries the follow-on edit
  • Fully right
CasesProduction promptNotes and schema addedFollow-on named
Needs nothing more
155/160
91/160
not asked
Sort rule
3/40
0/40
20/40
0/40
33/40
8/40
Growth-sentence rule
0/40
0/40
20/40
0/40
27/40
0/40
Text rule
6/80
0/80*
34/80
8/80*
45/80
20/80*
Unedited Qwen3.5-9B on the Big Tech workbook, 8 drafts per case. A draft is fully right when it applies, does the request, passes the schema and keeps every rule. *Text-rule drafts here passed the code check, which bounds the right answers from above.

Question 2

Qwen does not register the rule, and it cannot always carry it out

A failure to keep a rule has two possible causes. The model may not notice that the rule applies, or it may notice and be unable to make the edit. An edit to the model can supply noticing. It cannot supply a skill the model lacks.

To test noticing, we fitted a linear probe, a classifier that reads the model’s residual stream at one layer. It had to tell triggered from untriggered prompts at the prompt’s last token. Under leave-one-out, it was right 52% to 65% of the time at every layer, close to the 50% of a coin. Before it writes anything, Qwen does not represent that the request triggers a rule.

To test the skill, we added one line to each triggered prompt that names the follow-on edit. Qwen then tried it in 105 of 160 drafts (66%). It sorted the table correctly in 8 of 40 drafts and wrote the growth sentence correctly in none of 40. At most 20 of 80 text-rule drafts kept the rule. Even when the model notices, it often cannot do the work.

Question 3

A steering vector makes Qwen edit more, not edit correctly

The added line of Question 2 gives a direction. At each layer we took the mean residual stream of the prompts with the line, minus the mean of the prompts without it, at the last token. The 20 prompt pairs agree on it: a direction fitted without one pair has a cosine of 0.80 to 0.86 with the full one.

A steering vector adds a multiple of that direction to the residual stream at one layer, on every token. A rank-one edit writes the same direction into the weights instead. It adds α·d·kᵀ/(kᵀk) to the layer’s down_proj matrix, where d is the direction and k is the mean input to down_proj over the model’s own drafts. The edit then fires wherever the input resembles k.

Figure 3 · No steering vector or rank-one edit reaches the target

  • Unedited
  • Steering vector
  • Random direction
  • Rank-one edit
  • LoRA
  • Claude Opus 5.5's edit
target0%10%20%30%40%50%60%50%60%70%80%90%100%Triggered drafts that try the follow-on editUntriggered drafts fully rightUnedited: tries 5.6%, untriggered fully right 96.9%UneditedSteering, layer 13, strength 0.5: tries 3.8%, untriggered fully right 98.8%Steering, layer 13, strength 1: tries 5%, untriggered fully right 98.8%Steering, layer 13, strength 2: tries 6.2%, untriggered fully right 93.8%Steering, layer 14, strength 1: tries 2.5%, untriggered fully right 97.5%Steering, layer 14, strength 2: tries 0%, untriggered fully right 96.2%Steering, layer 15, strength 0.5: tries 6.2%, untriggered fully right 95%Steering, layer 15, strength 1: tries 16.2%, untriggered fully right 93.8%layer 15, strength 1Steering, layer 15, strength 1.5: tries 21.2%, untriggered fully right 87.5%Steering, layer 15, strength 2: tries 26.2%, untriggered fully right 80%Steering, layer 16, strength 1: tries 18.8%, untriggered fully right 82.5%Steering, layer 16, strength 2: tries 31.2%, untriggered fully right 77.5%layer 16, strength 2Steering, layer 17, strength 0.5: tries 7.5%, untriggered fully right 95%Steering, layer 17, strength 1: tries 8.8%, untriggered fully right 92.5%Steering, layer 17, strength 2: tries 15%, untriggered fully right 91.2%Steering, layer 19, strength 0.5: tries 2.5%, untriggered fully right 98.8%Steering, layer 19, strength 1: tries 2.5%, untriggered fully right 98.8%Steering, layer 19, strength 2: tries 3.8%, untriggered fully right 98.8%Steering, layer 21, strength 0.5: tries 3.8%, untriggered fully right 98.8%Steering, layer 21, strength 1: tries 5%, untriggered fully right 98.8%Steering, layer 21, strength 2: tries 3.8%, untriggered fully right 100%Random direction 1, strength 2: tries 3.8%, untriggered fully right 98.8%Random direction 2, strength 2: tries 5%, untriggered fully right 92.5%Random direction 3, strength 2: tries 10%, untriggered fully right 98.8%Rank-one edit, layer 15, α 1: tries 8.8%, untriggered fully right 93.8%Rank-one edit, layer 15, α 2: tries 20%, untriggered fully right 87.5%LoRA, rank 16: tries 50%, untriggered fully right 81.2%LoRAClaude Opus 5.5's edit: tries 35%, untriggered fully right 97.5%Claude Opus 5.5
Each dot is one way of changing Qwen3.5-9B, measured on the Big Tech workbook: 4 drafts per case for each edit, 8 unedited, 2 for Claude Opus 5.5’s edit. Steering is shown at layers 13 to 21 and strengths 0.5 to 2; the random directions have the same size as the steering vector at strength 2. The shaded corner is the bar we set before the runs.

Only layers 15 and 16 moved the model. At layer 15 with strength 1, 16% of triggered drafts tried the follow-on and 94% of untriggered drafts stayed fully right. At strength 2, the tries rose to 26% and the untriggered drafts fell to 80%. The rank-one edit traced the same curve. Three random directions of the same size gave 4% to 10% tries, so the effect belongs to the direction. Steering at strength 1 kept the rule in 2 of 80 triggered drafts, and the rank-one edit at α = 2 in 5 of 80.

The held-out workbook shows what the direction does. Untriggered drafts with more than one edit rose from 9 of 64 to 32 at strength 1 and 46 at strength 2. Requests done fell from 64% to 49% and 35%. The rule was kept in 10 of 64 drafts unedited and in 9 to 11 with an edit. The direction means “make more edits”, not “notice the rule”.

Figure 4 · On the held-out workbook, only LoRA improves on the unedited model

ModelRequests doneUntriggered fully rightUntriggered with extra editsText rule kept
Unedited
82/128 · 64%
34/64 · 53%
9/64 · 14%
10/64 · 16%
Steering, strength 1
63/128 · 49%
24/64 · 38%
32/64 · 50%
11/64 · 17%
Steering, strength 2
45/128 · 35%
12/64 · 19%
46/64 · 72%
10/64 · 16%
Rank-one edit, α 2
54/128 · 42%
20/64 · 31%
34/64 · 53%
9/64 · 14%
LoRA
91/128 · 71%
45/64 · 70%
15/64 · 23%
11/64 · 17%
Claude Opus 5.5's edit
60/128 · 47%
31/64 · 48%
10/64 · 16%
8/64 · 13%
The held-out workbook: 16 cases, 8 drafts each, 128 drafts per row. Steering and the rank-one edit are at layer 15. Every text-rule draft that kept the rule, in every row, came from the two cases h10 and h12.

Question 4

LoRA learns the cases it was shown

LoRA fine-tunes a model by training a small low-rank change to chosen weight matrices. It is the usual cheap baseline for teaching a model a behaviour. We wrote 248 further Big Tech requests, none of them among the 40 cases, each with a correct answer: 100 untriggered, 60 sort, 60 growth-sentence and 28 text. We trained rank-16 adapters on the MLP layers for two epochs, with the loss on the answer tokens only.

On the Big Tech workbook, LoRA kept the text rule in 17 of 40 drafts. Every one of the 17 is an operating-expense case, answered with a sentence learned from the training answers. The sentence says that the table’s new figure “differs from the sum of the lines listed above”. It failed all 12 drafts where a table cell had to follow the prose, and all 8 Netflix drafts, where it left a second paragraph unchanged. It wrote the growth sentence correctly in 3 of 20 drafts and never re-sorted the table, despite 60 sort examples. Untriggered drafts fell from 97% to 81% fully right.

On the held-out workbook, LoRA raised requests done from 64% to 71%, and untriggered drafts fully right from 34 to 45 of 64. It kept the text rule in 11 of 64 drafts, against 10 unedited, all from the same two cases. Its drift was 0.013 nats. LoRA taught the request better, and the rules only where a training answer could be copied.

Question 5

Claude Opus 5.5's edit works on the workbook it studied and fails on a new one

We gave the task to a new Claude Opus 5.5 agent with none of our earlier findings. It could run Qwen3.5-9B forward, read any activation or weight, and score a candidate change on the Big Tech workbook (40 cases, 2 drafts each). Its change had to be low-rank, at most rank 64 in total, on the decoder layers. Gradients were forbidden, and it had 30 scored passes. The held-out workbook stayed with us. Figure 5 shows the process.

Figure 5 · The writer reads the model, writes a change and tries it

Writer: Claude Opus 5.5reads the Big Tech workbook's 40 cases and 248 further editsScorerholds the held-out cases1 · Readforward passesover 288 cases2 · Diagnosestops after 1 editlayer 20 knows3 · WriteΔW = α·d·kᵀclosed form4 · Try40 cases × 2production scorer5 · Decidebest pass so far2 of 30 used6 · Held-out16 cases × 8run once, at the endwrite again(pass 1 → pass 2)
Steps 1 to 5 belong to the writer, which never saw the held-out cases. Step 6 belongs to the scorer and ran once, on the change the writer handed back.

At the end of its first edit, the model chooses between "},, which opens another edit, and "}, which closes the list. The writer measured the logit gap between the two. On text-rule cases the gap was −5.3, against −3.5 on cases that need nothing. The model was most certain that it was finished exactly where it was not.

Yet at this token the information is present. At layer 20, a probe separated text-rule cases from untriggered ones with an AUC of 0.98. Once Qwen has written the changed figure, it represents that prose depends on it. It does not act on that. Luo, Pan, Aghzal and Yao call this value–policy dissociation: an agent’s hidden states encode which action is better while it takes the worse one. In their grid world it happened on 34% to 58% of states (arXiv 2611.10457). Okcu, Wang, Xie and colleagues measured the same gap in another grid world, and there a rank-one steering edit changed the action no more often than a random direction (arXiv 2610.17444). In our case the signal is near chance at the prompt and appears only after the first edit. The writer also found that the sort and growth-sentence rules fail for a different reason: forced to continue, Qwen never re-sorted the table and wrote +859% for a growth that is −60%. It left those two rules alone.

Its change has two rank-one entries on layer 20’s down_proj, shown in Figure 6. In the first, the key k is a ridge regression on down_proj inputs, fitted to score 1 at the end of a first edit in a text-rule case and near zero elsewhere. The value d is the mean residual where the model continues minus the mean where it stops. The second entry fires where a second edit begins and points it at the prose, not at the operating-income cell. Every value came from forward passes, means and least squares.

Figure 6 · Each entry adds a direction where its key matches

input x12,288→W4,096 × 12,288down_proj, unchanged+α ·d4,096kᵀ 12,288→residual4,096k reads x: is this the position?d writes: add another editchange to the output = α · d · (k · x)
At most positions k · x is close to zero, and the model runs as before. Where the key matches, the change adds a multiple of d to the residual stream.

Figure 7 · The first entry moves the decision the right way in both groups

−8−6−4−20above 0:a 2nd editText-rule cases13 of 25 above 0 after−5.3−0.71Cases that need nothing0 of 34 above 0 after−3.5−7.9Open dot: unedited. Filled dot: with the first entry of the edit. Measured by the writer on the Big Tech cases.

The writer used 2 of its 30 passes. Its change kept 39 of 40 untriggered drafts fully right and tried the follow-on in 14 of 40 triggered drafts. The text rule was kept in 3 of 20 drafts by its rubric. The code check passed 7, but in four of those the new expense lines do not add up to the new total. The sort and growth-sentence rules stayed at zero.

On the held-out workbook, requests done fell from 64% to 47%. Untriggered drafts fully right went from 34 to 31 of 64, and the text rule was kept in 8 of 64, against 10 unedited. The drift was 0.049 nats, the largest of the edits and under the bar. The change repaired a fault with a clear cause on the cases it was fitted to. On a new document it did not help.

The process obeyed seven rules: the writer never saw the held-out cases, stayed within its budget and rank, handed back a change it had scored, and used no gradient. We wrote the process in TLA+, a language for describing how a system moves from state to state. TLC, its model checker, explored all 21 reachable states and found that every rule holds. The real run followed the module because the harness enforced each step: it refused a 31st pass and any change above rank 64, and the held-out files were never in the writer’s folder.

The TLA+ module and TLC’s result
---------------------------- MODULE WeightEdit ----------------------------
EXTENDS Naturals, FiniteSets
CONSTANTS Budget, MaxRank              \* 30 scored passes, total rank 64

Entries == {"continue", "target"}
Method  == [continue |-> "ridge key, mean-difference value",
            target   |-> "ridge key, mean-difference value"]
Changes == {c \in SUBSET Entries :
              c # {} /\ ("target" \in c => "continue" \in c)}
Rank(c)  == Cardinality(c)
Prose(c) == IF c = {"continue"} THEN 3 ELSE 7   \* code-check passes, of 20

VARIABLES phase, seen, passes, last, scored, final, heldout
vars == <<phase, seen, passes, last, scored, final, heldout>>

Init == /\ phase = "read"   /\ seen = {}   /\ passes = 0
        /\ last = {}        /\ scored = {} /\ final = {}  /\ heldout = 0

Read     == /\ phase = "read" /\ seen' = {"training", "extra"} /\ phase' = "diagnose"
            /\ UNCHANGED <<passes, last, scored, final, heldout>>
Diagnose == /\ phase = "diagnose" /\ phase' = "write"
            /\ UNCHANGED <<seen, passes, last, scored, final, heldout>>
Write(c) == /\ phase = "write" /\ c \notin scored /\ Rank(c) <= MaxRank
            /\ last' = c /\ phase' = "try"
            /\ UNCHANGED <<seen, passes, scored, final, heldout>>
Try      == /\ phase = "try" /\ passes < Budget
            /\ passes' = passes + 1 /\ scored' = scored \cup {last} /\ phase' = "decide"
            /\ UNCHANGED <<seen, last, final, heldout>>
Again    == /\ phase = "decide" /\ passes < Budget
            /\ \E c \in Changes : c \notin scored
            /\ phase' = "write"
            /\ UNCHANGED <<seen, passes, last, scored, final, heldout>>
Stop     == /\ phase = "decide"
            /\ final' = CHOOSE c \in scored : \A o \in scored : Prose(c) >= Prose(o)
            /\ phase' = "heldout"
            /\ UNCHANGED <<seen, passes, last, scored, heldout>>
HeldOut  == /\ phase = "heldout" /\ heldout' = heldout + 1 /\ phase' = "done"
            /\ UNCHANGED <<seen, passes, last, scored, final>>
Done     == phase = "done" /\ UNCHANGED vars

Next == Read \/ Diagnose \/ (\E c \in Changes : Write(c)) \/ Try
        \/ Again \/ Stop \/ HeldOut \/ Done
Spec == Init /\ [][Next]_vars /\ WF_vars(Next)

WriterNeverSeesHeldOut == "heldout" \notin seen
WithinBudget           == passes <= Budget
WithinRank             == \A c \in scored : Rank(c) <= MaxRank
FinalWasTried          == final # {} => final \in scored
FinalIsBestTried       == final # {} => \A o \in scored : Prose(final) >= Prose(o)
HeldOutOnceAtTheEnd    == heldout <= 1 /\ (heldout = 1 => final # {})
NoGradients            == \A e \in Entries : Method[e] = "ridge key, mean-difference value"
=============================================================================

TLC 1.7.4 · 25 states generated, 21 distinct, depth 10

holds WriterNeverSeesHeldOut

holds WithinBudget

holds WithinRank

holds FinalWasTried

holds FinalIsBestTried

holds HeldOutOnceAtTheEnd

holds NoGradients

What it means

Edits changed how much Qwen edits, not whether it keeps the rules

Qwen3.5-9B fails a workbook’s rules in two ways. Before it writes, it does not register that a rule applies. When told, it often lacks the skill, such as computing growth or re-sorting a table. An edit to the model can act only on the first failure.

Every edit read from activations, whether ours or Claude Opus 5.5’s, changed how many edits Qwen makes. On the Big Tech workbook that looked like progress. On the held-out workbook it lowered the share of requests done. The paper’s finding held here: a fault with a clear cause can be repaired on the cases studied, and a missing skill cannot be written in. LoRA did best on the held-out workbook, and its gain was in doing the request.

The rules that held throughout are the ones code enforces. Code rewrites every formula cell after the draft. In an earlier pass of this study, on 10 cases, no draft passed the schema until code wrote those cells, and 59% were fully right once it did. The sort rule and the growth sentence follow from the tables, so code can write them too. The text rule cannot be computed, and it still needs a model to read the prose and a check of what it wrote.

For people who keep documents with an assistant: write a rule as a formula or a check wherever you can. A rule left in prose is kept only as often as the model happens to notice it.

For people who edit models: test an edit on a document that the writer never saw. Our edits reached the Big Tech workbook’s numbers and lost them on the next workbook.

Six things limit these results

  • The held-out workbook has 16 cases. Its text-rule passes all come from two cases, h10 and h12, whose prose repeats the request’s own words.
  • Text-rule results were read by hand against rubrics. The Netflix cases are ambiguous, because the request makes the table disagree with the filing that the prose quotes.
  • Claude Opus 5.5 also screened variants by sampling the same 40 cases with its own checker, outside the 30 scored passes. Its Big Tech numbers are on cases it studied.
  • A probe that reads the rule shows that the information is present, not that the model uses it. Okcu and colleagues make the same point.
  • Each edit was measured with 4 drafts per case on the Big Tech workbook, and Claude Opus 5.5’s passes with 2.
  • LoRA’s training requests come from the same Big Tech document as the 40 cases, so its Big Tech numbers measure transfer within one document.

Method notes

We ran every model the same way

Sampling. Qwen3.5-9B in bf16 with vLLM, thinking off, temperature 0.7, top-p 0.95, seed 7. Unedited runs used 8 drafts per case; edits used 4 on the Big Tech workbook and 8 on the held-out workbook.

Steering. A vLLM plugin added the direction to the output of one decoder layer on every token. Rank-one edits and Claude Opus 5.5’s change were added to the weights and saved as a new checkpoint.

LoRA. Hugging Face PEFT, rank 16, alpha 32, dropout 0.05 on the gate, up and down projections; learning rate 1e-4 with a cosine schedule, batch 8, 62 steps. The adapter was merged for sampling.

Scoring. The production scorer applied each draft, rewrote formula cells and checked the schema. The rule checks and the rubric reading were added for this study.

More head-to-head evals

Does Physical Agent aim its tool calls at the object?Physical Agent writes a robot tool call on 400 of 400 frames, but moving the object 16 cm leaves its targets where they were.
MeasurePhysical AgentQwen3.5-4B
LIBERO frames answered with a tool call400 / 40018 / 400
Target movement per cm of object movement (median)0.02 cm—
Read the comparison
Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.
Measureask with jevask with Decider
Questions right, of 928989
Wrong answers12
Read the comparison
Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.
Measurejevask
Our 370 clinical decisions, right94.6%93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)77.0%74.4%
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.
Measureaskjev
Right, of the questions answered98%79%
Wrong answers, of 92219
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison