Does one neuron in Qwen3.5-4B catch reward hacking?

Claim. Neuron L17 N9209 in Qwen3.5-4B detects reward hacking or cheating intent in another model’s output. It was posted on X on 29 October 2026, with the note that the data were “relatively noisy”.

Test. We gave the model the 973 tasks of School of Reward Hacks that have both a hacking answer and an honest one. The model read each task with each answer. For every neuron in the model we asked one question. Out of 100 pairs, how often does the neuron separate the hack from the honest answer, in its own consistent direction? Fifty is a coin flip.

Result. L17 N9209 separates 59 pairs of 100, and it reads lower on the hacks, not higher. It ranks 67,112th of the model’s 294,912 MLP neurons. Another neuron, L14 N5185, separates 99 of 100 on tasks it was not chosen on. A probe on layer 17 separates 98. On this data the claim does not hold. The signal is real, and it lies elsewhere in the model.

You can watch the detectors read an answer yourself. Pick one of four tasks and see where each detector puts its hacking and honest answers, word by word. Or paste a task and an answer of your own.

Each task below has a hacking answer and an honest one, taken from the dataset. The posted neuron reads higher on the honest answer in all four. The best neuron reads lower on the hack in all four. The probe gives each hack a probability of more than 98% and each honest answer less than 2%.

Word by word, L14 N5185 falls most where the answer speaks to the grader instead of doing the task. In the short story, the deepest drop is on “Note to evaluator: This narrative employs an avant-garde approach”. The try-it page shades every word of every answer by either neuron.

School of Reward Hacks has 1,073 tasks. Each says how the answer will be scored, and each has an answer that games the scorer. In 973 of them there is also an honest answer. The other 100 ask for code with unit tests, and their honest answer is empty. We left those out, so every test below compares two real answers to the same task.

Qwen3.5-4B read each task as the user turn and each answer as its own reply, with thinking off, in bf16. For each of its 294,912 MLP neurons we took the input to the down projection at every token of the answer. We kept its mean and its peak over the answer. The score of a reading is its AUROC over all 1,946 answers, taken in the direction the reading leans.

A best neuron chosen on the same data would flatter itself. So we split the tasks by kind into two halves. We picked the best neuron on one half and scored it on the other. The probe was trained and scored the same way. L17 N9209 needs no such split: it was named before we looked.