Three models, one harness, twelve tasks: an interim report on a benchmark still collecting

A round is one task and one repeat: three fresh workbooks seeded with the task's tabs, one per model, from which the run can read nothing but its own seed and the web, and started together with the task's instruction. Each run has 900 seconds, except T10, which asks for 30 minutes of breadth and gets 1800. The rounds go round-robin, every task's first repeat before any task's second, so a collection stopped early still covers every task.

The revisions of every service are pinned before a round and read again after it. A run whose revisions changed underneath it is kept, marked, and run once more at the end of its round; the second run is the one counted. The 12 tasks:

Delivered means the run wrote a new version of its document and the harness kept it; it says nothing yet about whether the document is right. A turn is one call to the model. Seconds are the run's own clock, from start to its last word.

Both refusals, T05 r1 and T09 r2, are DeepSeek V4.1 Flash's, and both were the harness's fault, not the model's: they are two of the bugs below. Those runs, and the others the bugs touched, go round again under the fixed harness once the fifth repeat is in.

A benchmark of models is only that if each run used the model it names and nothing it should not have. Every run's trace is read for 4 things:

• Workbook only: no successful read of Google Drive, attachments, the catalog or other workbooks.
• Model: every model call went to the configured model through the Claude Code harness.
• Tabs: every seed tab reached the run.
• Notes: no calls to the notes tools.

117 of 117 runs pass all 4, counting the runs later replaced by a re-run. A failure of any one would stop the collection: a failed seal at once, the others if all three models fail together or one task fails twice running.

Re-runs: 7 rounds, 21 runs, were run again because a deploy landed mid-round. Of those 21 re-runs, 12 saw another deploy too; they are kept, since a run is repeated once.

Running 180 runs through the harness that serves every hedwigai workbook is also the hardest test that harness gets. Each of these was found in a run's trace and fixed the same day.

A shell that went silent

In T10's first round, DeepSeek V4.1 Flash and GLM-5.3-Flash stopped getting answers from their shell partway through: every command, even echo, ran into its timeout, and so did the harness's own file tools. Both ran to the end of their 1800 seconds (1964 s and 1930 s). DeepSeek V4.1 Flash's last message blamed the obvious suspect: "A `jq` pass over the large 10-K JSON dumps locked up the shell." It was a shell loop piped into cut, which deadlocked once 64 KB of output was waiting, and took the session with it. The fix was pushed at 08:35 UTC; T10 has not hung since.

Citing the workbook refused as written from memory

hedwigai refuses to keep a version whose sources cannot be followed: a footnote has to name a URL, a file or a command. In T05, which asks for a contract's terms as they stand today, the contract is a tab of the workbook. DeepSeek V4.1 Flash read it and cited each term to it in words, "Agreement tab, §2.1", and all 12 sources were refused as references written from memory; only the literal path tabs/Agreement.md passed. Later, in T09, a formula edit whose only footnote named the workbook itself was refused the same way. A citation to the run's own seed is the most checkable source there is; both forms are now accepted, by fixes pushed at 08:35 and 10:01 UTC.

Sources asked of formulas

After a run delivers, hedwigai looks for rows that state facts with no source and offers to find them. T09 asks for spreadsheet formulas, and all three models' first runs ended with the offer to find sources for 9 rows, every one of them a formula. Answering yes would have started a paid research run over arithmetic. Since 08:35 UTC, rows that are formulas are no longer asked about.

Three runs, one slot

The account was meant to run three at once, so a round could start all three models together. In the very first round, GLM-5.3-Flash waited 301 seconds for DeepSeek V4.1 Flash to finish before it could begin, and its record read "running" the whole time. Since 09:16 UTC the account has 6 run slots, two rounds run side by side, and no run has waited.

Collection goes on to the fifth repeat of T12, then the runs the bugs touched go round again. Only then is anything graded: each delivered document against a key written before the first run, blind to the model that wrote it. Which model got what right is for the final, graded results, which come later.