Blog
hedwigaihedwigaihedwigai

Weight-space programming

How do you keep a cartoon’s cast on model across a whole episode?

Gitarja is a claymation series about a Javanese princess in 1328. The first episode has 16 scenes, eight characters, a trading ship and four places that come back. An open image model, MiniMax-H3, draws every still. A video model then animates each still.

The image model has no memory. Each shot is drawn from that shot’s text alone, so the princess’s face, skin and diadem change from shot to shot. Animators call a drawing that has drifted from the design “off model”. Our earlier cut had many such shots.

We fixed it with two parts. The first is a canon: one picture of each character, ship and place, which every shot is drawn from. The second is a checker that remembers the canon. It is a frozen Qwen3.5-9B with one small weight edit for each entity. The edits are written by a program, not trained, and they need no labelled frames. A new story needs only its own canon.

Canonone sheet per characterone plate per placenever changesGoldfishthe image modelsees only this shot'stext and canon pictures4 takessame text and pictures,four seedsElephantQwen3.5-9B, frozen,plus one weight editper character and place;keeps the take whoseweakest entity is strongestthe next shot starts from the canon again, never from the last shot

Step 1

One canon picture for each character, ship and place

A character’s canon is a full-body sheet on a plain background. We made each one from an approved still: the image model redraws the character alone, and we pick one of three by eye. A place’s canon is an empty plate: the place with nobody in it, in clear light.

Every scene’s text then points at the canon. “Gitarja” becomes “Gitarja, drawn exactly as the woman in Picture 2”, and the sheet goes in as Picture 2. The image model sees only the pictures of what is in that shot. We call it the goldfish.

Canon sheets: Gitarja, Rajadewi, the king, the guard commander, the physician, Brahma, Indra and the ship
The canon: seven characters and the ship. Each sheet is the only picture of that entity that any shot is drawn from.
Empty walled garden plate
Walled garden
Empty courtyard plate
Courtyard
Empty river port plate
River port
Empty bedchamber plate
Bedchamber

Step 2

Sheet twins: the same shot twice, once with one thing wrong

A checker has to learn what “off model” looks like for each character. Labelled frames are slow to make and a new story has none. So we let the image model make the examples.

For each entity it draws pairs from the sheet. Both pictures of a pair have the same scene, the same seed and the same words, except one change in the second: a lighter skin, a cartoon face, a long narrow face, the head piece in another colour, other hair, a younger face, a garment in another colour. For a ship the changes are the hull, the sail, the masts, the prow, the painted eyes and the shape. The changes are the same list for every character, so a new story needs nothing written by hand.

Faithful twin
Faithful
Lighter skin
Lighter skin
Cartoon face
Cartoon eyes
Narrow face
Long narrow face
Ivory diadem
Ivory diadem
Silver diadem
Silver diadem
Brown hair
Brown hair
Round smiling face
Round young face
Faithful ship
Ship, faithful
Red sail
Red sail
Plain prow
Plain prow
Blue rail
Blue rail

Because the two pictures share everything else, their difference inside the checker is the drift alone. The scene, the light and the pose cancel out.

faithful twindrifted twin, same scene and seedscenes spread the points; the drift moves each pair the same way
A sketch of layer 10's picture tokens, averaged over the shot. Scenes move the points a long way. The drift moves every pair by a similar small step. The edit reads along that step.

Step 3

One weight edit per entity, solved in closed form

The checker is asked: “Picture 1 is the reference. Is Picture 2 exactly the same woman?” The base model answers from the answer token. It sees the costume well and the face poorly. On Gitarja’s real frames its Yes-minus-No score ranked good frames above off-model ones only 63% of the time.

The drift is visible earlier. At layer 10, the picture’s own tokens separate the twins. The edit adds new neurons in two places and changes no original weight.

  • Layer 10, every picture token. Eight neurons read the drift direction and write a flag into directions the model does not use. Their gates shut on text, on ordinary prose and on other characters’ pictures.
  • Layer 31, the answer token. 256 neurons, gated to open only on a question about this entity, read what the model’s own attention has carried there and move the answer toward Yes or No.

The direction is a regularised difference of the twins’ mean picture tokens. The gates and readout are least-squares solves on forward passes. The strength of the flag is chosen by cross-validation that holds out one kind of drift at a time, so the edit has to catch a drift it never saw.

sheetsheetsheetshotshotshotIsitsame?answer……layer 108 neurons on picture tokensread the drift direction, write a flaglayer 31gated readout at the answerflag → Yes or No; shut on all other textthe frozen model's own attention carries the flagYes − NoEach neuron adds D · silu(G z + b) · (U z) to the layer's output. z is the layer's input; G, U, D and b are solved in closed form. No original weight changes.
Real frames from the earlier cut, labelled by eye, scored once. AUROC is the chance that a good frame scores above an off-model one.

Gitarja

Qwen3.5-9B0.63
With its edit0.79

Bars start at 0.5, which is chance.

The edit was fitted from twins alone. The real frames were scored once, after the edit was fixed. With Gitarja’s edit on, the checker ranked the ship and Brahma as well as before.

Step 4

The elephant picks the take

A score that ranks frames well is enough to pick between takes. The goldfish draws 4 takes of each shot from the same text and canon. A small vision model finds each entity in each take. The checker, with every entity’s edit on, scores each crop against its sheet. We keep the take whose weakest entity is strongest.

On Gitarja’s real frames, picking at random from 4 gives an on-model frame 80% of the time. The base model’s pick is on model 86% of the time. The edited checker’s pick is on model 96% of the time.

Scene 1: the ship at night with Brahma and Indra
Night at sea: the ship, Brahma and Indra, each from its sheet.
Scene 7: Indra, Brahma and Gitarja in the courtyard
The courtyard plate, with three characters.
Scene 6: the king, Gitarja and the guard commander
Three sheets in one shot.
Scene 11: the physician and the king in the bedchamber
The bedchamber plate.

Step 5

Rules the whole episode must keep

A character can be on model and the shot can still be wrong. In our earlier cut, the crowd behind the guard commander was the guard commander, again and again. The image model had copied his sheet onto every face.

An earlier cut: every kneeling figure behind the guard commander has his face
An earlier cut. Each kneeling figure has the guard commander’s face.

So the episode now has written rules. Each rule names where it is checked and the question the checker answers. A new story writes its own list. The code does not change.

RuleWhere it is checkedThe checker’s question
On modelEvery still and every sampled clip frameThe entity's edit: same as its sheet?
One face, one personEvery pair of people in a still, and the castSame individual, same face and body?
Keeps the propEvery sampled frame of the clips that name itIs he holding the tablet up?

“One face, one person” is checked on every pair of people that the small vision model finds. Its pass mark is set on the canon: the same character drawn twice must pass, and two different characters must not. The rules for a still are written in TLA+, apart from the code that measures them. A take is valid only if it keeps all of them.

A clip must keep the rules in every frame. The checker reads every eighth frame. In one early clip, Brahma held his tablet up to the pole star and then let it go. The clip’s own text had told him to lower it. We fixed the text, and a rule now checks for it.

Three frames of a clip: Brahma holds the tablet up, then it is gone
Frames 0, 40 and 80 of an early clip. The tablet is gone by frame 40. The rule fails on those frames.

A failing clip is drawn again with new seeds, at most three times. If no attempt keeps every rule, the best one is kept and the shot is flagged for a person. The pipeline is written in TLA+ too. It states that a clip is never made from a still that broke a rule, and that no shot is drawn again forever. We checked both on a small model of the loop.

CastRules.tla — a valid take
Distinct(s, t) ==
  \A p, q \in People[s][t] :
    p # q => Same[s][t][p][q] < SameBar

Valid(s, t) == OnModel(s, t) /\ Distinct(s, t)
Pipeline.tla — what the loop guarantees
ClipOnlyFromValidStill ==
  \A s \in Shots : phase[s] \in
    {"clipcheck", "done", "flagged"} => stillOK[s]

CanonOnlyIfBothPassed ==
  \A s \in canon : stillOK[s] /\ clipOK[s]

EveryShotEnds == \A s \in Shots :
  <>(phase[s] \in {"done", "flagged", "escalated"})

The test

Do viewers stay longer?

A checker’s score is our own measure. The real test is whether people watch. We wrote the test down before posting anything.

  • Two cuts of the same episode: the new one, and the cut made before this method. They share the script, the voices, the music and the timing. Only the pictures differ.
  • Each cut becomes the same six Shorts. That makes six pairs, posted one a day. The cut posted first alternates.
  • We read “stayed to watch” for each Short once, seven days after it went up.
  • The method wins only if the new cut wins all six pairs. Five of six is a lead, not proof.

We will post the result here, whichever way it goes.

For a new story

What a new story needs

  1. One approved still per character, ship and recurring place. The image model turns each into a sheet or a plate.
  2. Scene texts that name the entities. The program points each name at its canon picture.
  3. About 80 twins per entity, drawn by the same image model in about 25 minutes on one GPU.
  4. One weight edit per entity, about 12 minutes each on one GPU. No labelled frames.
  5. Four takes per shot, and the edited checker keeps one.
  6. A list of rules: who must look like their sheet, who must look unlike each other, and which props stay in hand.

Limits

What it does not do yet

The edit ranks frames well, but its Yes/No cut lands in the wrong place on real frames. That is why we use it to pick between takes and not to accept or reject a single take. The twins show only the drifts on our list. A drift outside the list, such as a wrong number of fingers, is not covered.

Two earlier attempts failed, and we report them. Matching a frame’s picture tokens against the sheet directly was at chance on characters it was not fitted on. Off-model pictures drawn from words alone, with no sheet, were easy to tell apart for reasons that had nothing to do with drift.

The episode

The final cut

Every still in the final cut was picked by the edited checker. The video model then animated each one.

  • Image model: MiniMax-H3 (Q4 GGUF), six steps. Video: Wan 2.2 image-to-video. Checker: Qwen/Qwen3.5-9B in bf16. Crops: Qwen3-VL-4B grounding.
  • Twins: 8 scenes × (2 faithful + 8 drifts) per entity for the first four entities, 4 scenes for the others; each pair also read under a matched random crop and exposure.
  • Edit: layer 10, 8 neurons on picture tokens; layer 31, 256 gated neurons at the answer. Direction by shrinkage LDA on paired differences. Flag strength by leave-one-drift-out cross-validation.
  • Rules: on-model bar at each entity’s 10th percentile over the episode’s takes; same-person bar by Youden’s J on canon pairs; prop bar by Youden’s J on prop twins. Clips: every 8th frame, at most three re-draws.

More head-to-head evals

Does Physical Agent aim its tool calls at the object?Physical Agent writes a robot tool call on 400 of 400 frames, but moving the object 16 cm leaves its targets where they were.
MeasurePhysical AgentQwen3.5-4B
LIBERO frames answered with a tool call400 / 40018 / 400
Target movement per cm of object movement (median)0.02 cm—
Read the comparison
Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.
Measureask with jevask with Decider
Questions right, of 928989
Wrong answers12
Read the comparison
Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.
Measurejevask
Our 370 clinical decisions, right94.6%93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)77.0%74.4%
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.
Measureaskjev
Right, of the questions answered98%79%
Wrong answers, of 92219
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison