How does Nanbeige4.3-33B score at 4 bits per weight?

Nanbeige4.3-33B is an open reasoning model with 33 billion weights. Its makers report these scores on its Hugging Face model card:

We tested a copy of the model that stores most of its weights in 4 bits instead of 16. We used the model card’s sampling settings and allowed each answer up to 60,000 tokens. All our scripts are in a public gist.

Biology has only 19 questions, so each answer moves its score by about 5 points. Most of the lost GPQA points are in chemistry. The model got 14 of the 72 organic chemistry questions wrong. In IFEval, 9 answers were empty. In each of them, the model spent the whole 16,000-token limit thinking and never wrote an answer.

The similar models are those that the model card compares with Nanbeige4.3-33B. Their scores come from the same table, where the makers ran all four models with the same settings. We did not test them ourselves.

The margin shows how far a score could move by chance. It is one standard error. For AIME, it is worked out across the 30 problems. For GPQA and IFEval, it is worked out from the number of questions.

Three settings differ between our run and the makers’ run. Each could explain part of the gap.

We did not test the 16-bit model, so we cannot say how much of the gap comes from the 4 bits.

The model thinks for a long time before it answers. Everything it writes is counted in tokens, which are words or parts of words. An answer that reaches the token limit is cut off before its final answer, so it counts as wrong. We scored the same answers twice: once cut at 32,000 tokens and once at 60,000. The difference shows the effect of the limit alone.

On AIME, 6 correct answers needed more than 32,000 tokens. Even at 60,000, 4 of the 60 AIME answers and 2 of the 198 GPQA answers were cut off. Problem 15 was cut off in both tries while its reasoning still made sense. A higher limit would probably raise the AIME score.

IFEval tasks are short pieces of writing, so we gave each answer 16,000 tokens. Most answers were short, as in the first example below. 9 of the 541 answers were cut off, as in the second. A higher limit could add at most 1.7 points.

6 of the 9 cut-off answers were to tasks that asked for a reply in a named language, such as Kannada, Punjabi or Hindi. Those cut-off answers account for 6 of the 7 missed language rules. So the weak score on this rule comes mainly from long thinking, not from replies in the wrong language.

The original model keeps each weight in 16 bits. The copy we tested keeps most of them in 4 bits, so its weights need about a third of the memory. While the model writes, it also keeps a cache of every token so far, called the KV cache. For long answers this cache needs more memory than the weights.

The gist has every script, every software version and every step of the setup.