Can forward passes train a small model on a whole dataset?

Training a model means gradients. Each example goes forward through the model, the error goes backward, and an optimiser nudges every weight a little. We asked whether a model can learn a dataset without any of that.

Our method uses forward passes and least squares only. It adds new neurons to the model and computes their weights in closed form, one layer at a time.

The model is Qwen3.5-0.8B. We trained it on five text-classification benchmarks, from 2 labels to 151, with up to 20,000 training texts each. We compared it with one epoch of LoRA, a common way to fine-tune with gradients.

The answer is yes, nearly. On TREC and Banking77 the edited model comes within 2.1 points of LoRA. On SST-2, AG News and CLINC150 it trails by 3.0 to 5.2 points. On every benchmark it keeps about 99% of the model’s next-token guesses on ordinary text unchanged.

Every layer of the model ends with an MLP. Its neurons read the residual stream, and each writes back into it. A new neuron is one more row in two input matrices and one more column in the output matrix.

We give the model a prompt that names the labels, then the text, then “Label:”. The new neurons must fire at that last position and stay quiet everywhere else. Training takes four steps per layer.

Each new neuron multiplies two readings of its input. Together, thousands of them give the regression curved features, not only straight ones. A gate shuts every neuron off away from the label position.

Two numbers remain to choose: the ridge penalty and the strength of the write. We choose them on a tenth of the training texts held aside. Each choice must leave at least 99% of next-token guesses unchanged on Wikipedia text that the fit never saw.

In the last layer, every candidate can be scored exactly from numbers already collected. In earlier layers, the later layers reshape what a neuron writes, so we check each candidate with real forward passes.

The unedited model knows the label names from the prompt and little more. It scores 15% on TREC and 10.5% on CLINC150.

Neurons in the last layer alone lift every benchmark above 83%. Neurons in four layers, fitted in turn, add 2.8 points on TREC, 2.1 on SST-2 and 5.1 on Banking77 and 5.5 on CLINC150. On AG News they cost 1.1 points.

The best edit reaches 95.2% on TREC, against 97% for LoRA. On Banking77 it reaches 90.8%, against 92.9%.

Before writing anything into the model, we fitted the labels directly from its cached activations. This readout may use every layer and the average over the whole text. An edit at the label position cannot read the average.

On Banking77 the readout scores 93.8%, above LoRA’s 92.9%. On TREC it ties LoRA. On SST-2, AG News and CLINC150 it stays 0.7 to 2 points under LoRA. There, gradients change the model’s features themselves, and least squares on frozen features cannot.

The gap between our edits and the readout is the cost of the gate and of reading only the label position. It is 2.0 to 3.2 points. Closing it is the next piece of work.

A trained model should still be the same model on everything else. We measured this on 100 Wikipedia paragraphs from WikiText-2 that no fit ever saw.

The four-layer edits leave 98.99% to 99.42% of next-token guesses unchanged. A LoRA fine-tune changes every weight it touches on every input.

The measurement had one trap. Appending neurons to the bf16 weight matrices changes how their sums round. In layer 12, 4,096 neurons that write exactly nothing already change 1.3% of guesses. We therefore run the new neurons as a separate block whose output is added to the MLP’s. That is the same function, and the original computation stays bit for bit the same.

Training is forward passes and matrix solves. The memory needed is what the model needs to run, plus the collected activations.

It is not yet faster than LoRA. With the long prompt that names the labels, the four-layer edits took 2.6 to 39 minutes on one RTX 3090. The longest was CLINC150, whose prompt lists 151 labels. One epoch of LoRA took 1.7 to 5.2 minutes on the same card. Most of our time is forward passes over the training texts, once per layer and once per candidate strength.

The result is the base model plus a patch of new neurons. Many patches can share one copy of the base model, each added when its request arrives.

The try page trains the same model on labels and examples you give it. It holds a fifth of your examples back and scores the edited model on them, against the unedited model and random neurons of the same size.

With fewer than 500 training examples, it puts neurons in the last layer only. Below that size, strengths for the earlier layers are chosen on too few examples to trust.