hedwigaihedwigaihedwigai

Replay of recorded runs

A solo agent and two teams tune the same kernel, call by call

Each setup gets 40 calls to Qwen3.5-4B. The easy design tops out at 0.50×. The fast design reaches up to 1.06×, but only when tile size, memory layout and sync are all set right; with any one wrong it fails. Every move below is one call from our recorded runs on a Mac, replayed, so it uses no credit. The task and its scorer are synthetic. The post explains the experiment.

Run 18 of 20. Across all 20, mean best: solo agent 0.69×, fixed team 0.93×, evolving team 0.89×.

Best speed so far, run 18

  • Solo agent
  • Fixed team
  • Evolving team
  • first call past 0.50×

Solo agent

0.47×

Never past 0.50×

One agent, one history.

0.50× easy-design ceiling1.0× experteasy design 30failed 2fast design 0Agent 0.39×

The 40 calls, in order

  • easy design 30
  • failed 2
  • malformed 8
  1. #40 Agent: simple · 128 · 4 · 4 · swizzled · async → 0.39×
  2. #39 Agent: simple · 64 · 4 · 4 · swizzled · async → 0.44×
  3. #38 Agent: simple · 64 · 4 · 4 · swizzled · async → 0.44×
  4. #37 Agent: simple · 128 · 4 · 4 · swizzled · async → 0.39×
  5. #36 Agent: simple · 64 · 4 · 4 · swizzled · async → 0.44×
  6. #35 Agent: path=? · 32 · 4 · 4 · layout=? · sync=? → path must be one of simple/fast
  7. #34 Agent: path=? · 64 · 8 · 4 · layout=? · sync=? → path must be one of simple/fast
  8. #33 Agent: simple · 64 · 4 · 4 · swizzled · async → 0.44×
  9. #32 Agent: simple · 64 · 4 · 4 · swizzled · async → 0.44×
  10. #31 Agent: path=? · 32 · 8 · 4 · swizzled · async. → path must be one of simple/fast

Supervisor

None. One agent keeps its whole history, and nothing changes how it works.

Fixed team

1.06×

Past 0.50× at call 6

Three set roles share one log.

0.50× easy-design ceiling1.0× experteasy design 1failed 6fast design 25Implementer 1.02×Researcher 0.99×Planner 1.02×

The 40 calls, in order

  • fast design 25
  • easy design 1
  • failed 6
  • malformed 8
  1. #40 Implementer: path=? · 128 · 4 · 3 · layout=? · sync=? → path must be one of simple/fast
  2. #39 Planner: fast · 128 · 4 · 4 · swizzled · async → 1.02×
  3. #38 Researcher: fast · 128 · 8 · 4 · swizzled · async → 0.99×
  4. #37 Implementer: fast · 128 · 4 · 4 · swizzled · async → 1.02×
  5. #36 Planner: fast · 128 · 4 · 4 · swizzled · async → 1.02×
  6. #35 Researcher: path=? · tile=? · 16 · stages=? · layout=? · sync=? → path must be one of simple/fast
  7. #34 Implementer: path=? · tile=? · unroll=? · 4 · layout=? · sync=? → path must be one of simple/fast
  8. #33 Planner: path=? · 128 · 4 · 3 · layout=? · sync=? → path must be one of simple/fast
  9. #32 Researcher: path=? · tile=? · unroll=? · 4 · layout=? · sync=? → path must be one of simple/fast
  10. #31 Implementer: path=? · 128 · 4 · 4 · layout=? · sync=? → path must be one of simple/fast

Supervisor

None. The three roles and their shared log never change.

Evolving team

1.06×

Past 0.50× at call 2

A supervisor redesigns the team every 10 calls.

0.50× easy-design ceiling1.0× experteasy design 0failed 13fast design 17Tuner 0.99×Aggregator failedStaller 1.02×

The 40 calls, in order

SSSS
  • fast design 17
  • failed 13
  • malformed 6
  • supervisor 4
  1. #40 Staller: fast · 128 · 4 · 4 · swizzled · async → 1.02×
  2. #39 Aggregator: fast · 32 · 4 · 3 · row · async → compile error, instruction shape not supported for this tile
  3. #38 Tuner: fast · 128 · 8 · 4 · swizzled · async → 0.99×
  4. #37 Staller: fast · 64 · 4 · 3 · swizzled · async → 0.98×
  5. #36 Aggregator: fast · 32 · 1 · 1 · row · async → compile error, instruction shape not supported for this tile
  6. #35 Tuner: fast · 64 · 4 · 3 · row · async → tests failed, output mismatch (412 of 4096 values wrong)
  7. #34 Staller: fast · 128 · 4 · 4 · row · async → tests failed, output mismatch (412 of 4096 values wrong)
  8. #33 Aggregator: fast · 32 · 1 · 1 · row · async → compile error, instruction shape not supported for this tile
  9. #32 Tuner: fast · 64 · 8 · 3 · row · async → tests failed, output mismatch (412 of 4096 values wrong)
  10. #30 Staller: fast · 128 · 4 · 2 · swizzled · async → 0.94×

Supervisor’s diagnosis, call 31

“The team is stuck because the Tuner ignored the allowed range of stages and unroll values, generating invalid configurations, while the Aggregator and Staller are either hitting compile errors or missing the precise 1.0x boundary on valid, high-performance configurations.”

All 20 runs

Best speed in each run. Filled cells reached the fast design. Pick a column to replay it.

Run1234567891011121314151617181920MeanFast designTokens per run
Solo agent0.69×8 of 2042k
Fixed team0.93×16 of 2044k
Evolving team0.89×15 of 2030k

The supervisor’s calls come out of the same 40, and its words are shown as it wrote them, including the wrong diagnoses and, with the 0.8B model, the malformed role names. The design of the experiment follows Evolving Agent Teams (Cao, Mao, Li, Gonzalez and Stoica). We had no part in that paper, and this toy does not reproduce it. The harness, prompts and scorer are in a gist.