How to train a private AI model on your own data and run it on-premise

A private AI model is a model trained on your own past work that runs where your data already is: a server in your building, or the device that makes the decision. Nobody outside sees the data, nobody bills you per call, and the model does one job your team does every day.

Most guides to this start with GPUs and frameworks. The hard part is earlier: picking the one check worth automating, and proving the model does it as well as your team. This guide goes in that order, in seven steps, and starts with when you should not do it at all.

Often you do not. If your data may go to a cloud provider, the work varies from day to day, and the volume is modest, a hosted model such as Claude, GPT or Gemini called through its API is cheaper to start and better at open-ended work than anything you will train. Your own model earns its place in three situations:

• The data can’t leave the building. Drawings, inspection images, patient or customer records that a contract or a regulator keeps off public AI services, so the check is still done by hand.
• The decision can’t wait for a server. A robot hand, a drone or a sensor in a field has to decide on its own, where the signal is weak or a round trip is too slow.
• The per-call bill grows with every customer. The same check runs thousands of times a day, and you pay for each one, easy or hard.

Not “AI for operations”: one decision with a clear right answer that people make many times a day. Is this solder joint good. Which of four teams should take this request. Does this carton print match the approved artwork. One check is something you can measure, so you will know whether the model works.

You likely have the training data already: past images, documents or logs, each with the decision your team made. A few hundred examples is enough to start for a narrow check. Write down how the check is done today: who does it, how long it takes, and what counts as a miss. That is the bar the model has to clear.

Set a fifth of the examples aside before training and never train on them. They are how you will test the model honestly in step 5.

A small open-weight model, a few billion parameters for text or a compact vision model for images, is enough for one narrow check and small enough to run on one server or on the device. A large general model is the wrong starting point: it costs more to run and its extra breadth is never used.

Two models often beat one. A tiny model clears the easy cases; the model trained on your data takes the rest; anything it is unsure of goes to a person. Your staff then see only the cases that need them.

Small really is enough for a narrow job. In our router experiment, a 0.8-billion-parameter model, Qwen3.5-0.8B, sent 83.2% of 600 requests to the right one of 18 scenarios after one new neuron per label was added. Left unedited it managed 8.7%.

Fine-tune with low-rank adapters (LoRA, via Hugging Face PEFT or Unsloth): only a small set of extra weights is trained, so one GPU is enough and the run takes hours, not weeks. Run it on your own machine, or on a rented one that is wiped afterwards, so the data never sits on a shared service.

Gradients are not the only way to write into a model. We trained Qwen3.5-0.8B on five classification benchmarks with forward passes alone: new neurons fitted by least squares, one layer at a time. It scored 95.2% on TREC and 90.8% on Banking77, within 2.1 points of one epoch of LoRA, and left about 99% of its answers on ordinary text unchanged. That last figure matters for a private model: you want the new skill without losing the old ones. It is not yet faster than LoRA, so LoRA is still where to start.

Run the model and your current process on the held-out examples and compare: how often each is right, how long each takes, and what the model missed, case by case. Report the misses, not just the accuracy. A model that is right 97 times in 100 is only useful once you know what the other three look like.

Hold the report to a published standard. Ours is the one we use for our interpretability experiments: every figure comes with its method and its controls. One neuron posted online as a reward-hacking detector told hacks from honest answers in only 59 pairs of 100 when we tested it; another, found on held-out data, did it in 99.

A language model serves on your own server with vLLM or llama.cpp, behind an API your systems already know how to call. A vision model exported to ONNX runs on the camera, the robot or a small box beside the line. Either way the answer is produced where the data was captured, and it keeps working when the internet does not.

The device can learn there too. Phones, cameras and other edge chips usually run only the forward pass, so they cannot train a model the usual way. The least-squares neurons in our forward-pass training experiment need nothing else: four answers became a 78 KB patch, against about 21 MB for a rank-16 LoRA, so one device can keep one base model and a small patch per person or per site.

Parts, defects, documents and requests change, and a model trained on last year’s does not know. Keep logging the cases people overrule, and retrain on them on a schedule. A model nobody retrains quietly gets worse.

A hosted model bills per million tokens read and written. These were the listed prices on 9 November 2026:

Take an illustrative check that reads a 2,000-token document and writes a 300-token answer. Run 1,000 or 10,000 times a day for 30 days, the bill is:

The bill grows in a straight line with volume; your own model’s cost does not. It is the server it runs on and the work of training and retraining it, the same at 1,000 checks a day as at 10,000. So the small cheap models settle low-volume checks, and your own model pays off when the volume is high, when you need a large model’s accuracy on a narrow job, or when the data could not be sent at any price.

The steps above rest on experiments we ran and published, each with its method, its controls and what did not work.

We run the seven steps for one check, as a pilot. You hand over a sample of past work and how you decide today; we sign your NDA, and the data is used only for your model.

Read more about a pilot, or tell us about the check.