A general vision model cannot tell you where
If you are choosing how to read industrial images — a weld, a casting, a panel, a part on a line — the question is not whether a frontier model can describe what it sees. It can. The question is whether it can put a box around the thing it just described, because that box is what a downstream system crops, measures, gates or moves an actuator to.
We measured it. Four frontier vision-language models, the same images, one verdict and one box each. The verdict is scored against a rule that never opens the image; the box against a small model trained for that one job. Both references beat all four models on their own axis — and the trained model is not slightly ahead on the box, it is a different order of thing.
What follows is the whole method and every number behind that, including the ones that are unflattering to the case. The conclusion a buyer should take from it is at the foot: on industrial images, the pointing is a job for a model trained to point, and it is not a prompt away.
The short version
Out of a hundred requests, the best of these four returns 14 boxes loose enough to crop around — and no model returns more than 7 tight enough to put a tool on. They are not the same model. The rest are silences and misses, and the misses do not announce themselves.
The verdict column is not the good news it looks like. The best model answers 86.0% correctly, which is below the 96.5% you get by answering the same way every time without looking. Reading the image is currently costing accuracy rather than adding it.
A model trained for the one job is a different order of thing. 64% of its boxes land, against a best of 24%, at a mean overlap of 0.421 against 0.109. The general models are the wrong tool for pointing rather than weak models in general — every one of them publishes a serious score somewhere.
What comes back from a hundred requests
The first table anyone should read, and the one that is hardest to quote selectively. A model that declines the difficult images looks excellent on any measure computed over the boxes it chose to draw; counted per request, declining costs what declining costs.
| model | answer parsed | box returned | usable box ≥0.3 | tight box ≥0.5 | nothing returned |
|---|---|---|---|---|---|
| Qwen3-VL-30B-A3B-Instruct | 100.0 | 96 | 8.7 | 6.8 | 4 |
| MiMo-V2.5 | 95.6 | 82 | 4.1 | 0.0 | 18 |
| Ling-3.0-flash-VL | 100.0 | 68 | 8.9 | 5.5 | 32 |
| DeepSeek-V4-Flash-Vision-Exp | 89.5 | 60 | 14.3 | 1.8 | 40 |
Every figure is per 100 requests. 0.3 overlap means the box and the true box share about a third of their area — enough to crop around, to hand to a person, or to cue a second model. 0.5 is the loosest thing most teams will act on directly. Neither threshold is generous.
The measurement as recorded
The run as it printed, with the two references that are not models. Everything in the table above is arithmetic on these columns.
| model | n | verdict | PF | boxes | meanIoU | medIoU | ≥0.3 | ≥0.5 |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen3-VL-30B-A3B-Instruct | 57 | 86.0% | 0 | 55 | 0.085 | 0.000 | 9% | 7% |
| XiaomiMiMo/MiMo-V2.5 | 45 | 84.4% | 2 | 37 | 0.032 | 0.000 | 5% | 0% |
| inclusionAI/Ling-3.0-flash-VL | 57 | 82.5% | 0 | 39 | 0.083 | 0.000 | 13% | 8% |
| deepseek-ai/DeepSeek-V4-Flash-Vision-Exp | 57 | 61.4% | 6 | 34 | 0.109 | 0.000 | 24% | 3% |
| constant baseline the same answer every time, with no image read at all | 57 | 96.5% | — | — | — | — | — | — |
| trained localiser a small model trained for this one job | — | — | — | — | 0.421 | — | 64% | — |
PF is answers that could not be read as an answer at all. boxes is how many of the n items the model returned a box for; the IoU columns are over those boxes, not over every item, which flatters a model that declined to guess.
Where each model wins
Two of the four take all five dimensions between them, and the way they split is the finding. Qwen3-VL-30B-A3B-Instruct answers on nearly every image, is never unreadable and has the best verdict — and returns 8.7 usable boxes per hundred requests. DeepSeek-V4-Flash-Vision-Exp has the boxes most likely to land, and answers least often, refuses most often and gets the verdict wrong most often. Which of those you can live with is an engineering question about your own system, not a ranking.
| Dimension | Best | Worst | What it decides |
|---|---|---|---|
| Answer can be parsed | Qwen3-VL-30B-A3B-Instruct 100.0% | DeepSeek-V4-Flash-Vision-Exp 89.5% | An unreadable answer is not a wrong answer — it is a request that has to be retried or dropped. Two of the four never produced one; one produced six. |
| Returns a box at all | Qwen3-VL-30B-A3B-Instruct 96% | DeepSeek-V4-Flash-Vision-Exp 60% | Silence is the cheapest failure and the easiest to miss in a demo: the item is simply never handled, and nothing in the log says it was hard. |
| Verdict correct | Qwen3-VL-30B-A3B-Instruct 86.0% | DeepSeek-V4-Flash-Vision-Exp 61.4% | Read against 96.5% for answering the same way every time without opening the image. Every model is below it. |
| Of the boxes drawn, share at ≥0.3 | DeepSeek-V4-Flash-Vision-Exp 24% | MiMo-V2.5 5% | The flattering measure, and the one most often quoted: it ignores every request the model walked away from. |
| Usable boxes per 100 requests | DeepSeek-V4-Flash-Vision-Exp 14.3 | MiMo-V2.5 4.1 | The same measure with the silences counted. This is the number to put into a business case. |
What the tables say
Answering the same way every time beats every model. The constant baseline scores 96.5% on the verdict without reading the image. The best of the four reaches 86.0%, and the worst 61.4%. A model that looks and then answers is doing worse than a rule that does not look, which means the verdict on this data carries almost no information and the models are finding some of it and losing more.
The median box misses entirely. Every one of the four has a median IoU of 0.000. Not low overlap — for more than half the boxes, no overlap at all. The means survive only because a small number of boxes land: the best mean is 0.109, against 0.421 for the trained localiser.
A model trained for the job is a different order of thing. 64% of the localiser's boxes clear 0.3 overlap. The best VLM manages 24%, and that same model has the worst verdict of the four and 6 unreadable answers. Nothing here is close.
Declining to answer is not scored as a miss. Only Qwen3-VL-30B-A3B-Instruct returned a box for nearly every item (55 of 57); the others answered on 34 to 39. The IoU columns are computed over boxes drawn, so a model that stays quiet on the hard ones reads better there than it would on the whole set — which is the whole reason the first table exists.
What each failure costs
These are four different failures with four different owners, and they are not degrees of one another. A system that handles the first three can still be ruined by the fourth: a confident box in the wrong place is the only one that makes a machine act.
| Failure | What it looks like | Measured here | Can it be absorbed? |
|---|---|---|---|
| Answer cannot be parsed | The request errors. The item waits for a retry or falls to a person. | 6 of 57 on DeepSeek-V4-Flash-Vision-Exp; none at all on two of the four | Easily — it is loud, it is logged, and a retry costs a second call. |
| No box returned | Nothing happens. The item is skipped and the throughput quietly drops. | 40% of requests on DeepSeek-V4-Flash-Vision-Exp, 4% on Qwen3-VL-30B-A3B-Instruct | Yes, if somebody is counting. It shows up as work not done rather than work done wrong. |
| Box returned, box misses | The arm moves to the wrong place, or the crop sent on for inspection is of nothing. | Median overlap 0.000 on all four — more than half of every model's boxes miss entirely | This is the expensive one. Nothing downstream can tell it from a hit. |
| Verdict right, box wrong | The system says there is something and cannot say where. A person re-does the looking. | 86.0% verdicts against 8.7 usable boxes per 100 on Qwen3-VL-30B-A3B-Instruct | Only where a human was going to look anyway. Then the model saved nothing. |
What this means for a physical system
Physical intelligence is the case where the box is not a picture for a person — it is a coordinate something moves to. That raises the bar in a specific way: silence becomes throughput lost, a miss becomes motion in the wrong place, and neither can be cleaned up by a reader who notices the answer looks odd.
| The job | What it needs | Ready? | Why |
|---|---|---|---|
| Describing a scene to a person | Language about an image | yes | This is what the published benchmarks measure and what these models are genuinely good at. Nothing here contradicts it. |
| Routing an item to the right human queue | A verdict, no coordinates | no | Not on this data. Every model scores below the 96.5% you get by answering the same way every time, so the routing would be worse than a rule that costs nothing to run. |
| Cueing a second-stage detector | A rough box, roughly right | with care | A general model returns something usable on a small minority of requests, and the rest have to be caught by whatever sits downstream rather than trusted. |
| Driving an actuator from the box | A tight box, nearly always | no | Tight boxes are in the low single digits per hundred requests, and the median box does not touch the object at all. |
| Measuring, counting or gating on position | A tight box, every time | no | The same numbers, with no human in the loop to catch the miss. A model trained for the one job is the only thing in this measurement that is close. |
What follows from that table is not “wait for better models”. It is to stop asking one model to do both halves. A general model is the best thing available for reading a scene, deciding what kind of thing is in front of it and writing down what happened; it is, on this evidence, the worst available way to say where something is. The pointing is a job for a model trained to point — 64% of its boxes land against a best of 24%, and that gap is not a prompt away.
That has a cost consequence worth stating plainly. A trained localiser has to be built per job, and it does that one job — which reads as the expensive option next to a single general model serving everything. The tables above are what the cheap option actually delivers: at best 14 usable boxes per hundred requests, with the rest distributed across silence and confident error. The comparison is not general-versus-trained on price. It is trained-versus-nothing on whether the system works at all.
These are not weak models
Each publishes a serious result. One line and a source each — the four report different benchmarks, and lining them up would invent a comparability nobody has established.
| Model | Published result | Source |
|---|---|---|
| Qwen3-VL-30B-A3B-Instruct | MMMU 74.2 an independent reproduction attempt landed at 70.19 | study |
| Ling-3.0-flash-VL | Artificial Analysis Intelligence Index v4.1.1 — 42 four points above the non-VL Ling-3.0-flash | study |
| DeepSeek-V4-Flash-Vision-Exp | ZeroBench 35.0 Pass@5 reported above Opus 4.8 at 34.0 on the same measure | study |
| MiMo-V2.5 | Claw-Eval 62.3 on the general subset the widely quoted MiMo figures are for MiMo-V2.5-Pro, a different variant | study |
What is missing from that column is the point. None of those benchmarks measures putting a box on an object. A model can answer questions about an image at 74 on MMMU and still have a median overlap of zero when asked where the thing is, because answering about an image and pointing at part of one are different abilities — and only the first is routinely published. A procurement decision made on the published numbers is a decision made on the half of the problem that was measured.
What this means if you are buying
Use a general model for the language and a trained one for the coordinates. They are two different jobs and this measurement separates them cleanly. Reading a scene, deciding what kind of thing is in front of it, writing down what happened — a frontier model is the best thing available. Saying where — on this evidence, the worst.
The specialised model is not the expensive option. It is the one that works. A trained localiser has to be built per job and does only that job, which reads as costly next to one general model serving everything. Then you price what the general model actually returns: 14 usable boxes per hundred requests at best, 7 tight enough to act on, and the rest split between saying nothing and being confidently wrong. A pilot that cannot clear the second number does not become a line that runs.
Price the misses, not the requests. The silences are cheap — something downstream notices nothing arrived. The wrong boxes are not: they arrive looking exactly like the right ones, and whatever they cost is paid by the process, not by the model. That is the number to put against the cost of training for the job.
Then check it on your own images. 57 items is a small set and it is ours — it says what these models did on this data, not what they would do on yours. The direction of the gap is large enough to act on; the exact figures are not precise enough to rank the four against each other. The method above is written down so you can run it on your parts, with your defects, and get a number that is about your line. If it is useful, we will run it with you.
Two things this measurement deliberately does not do, so they are not read into it: no prompt engineering was attempted beyond asking plainly, and the trained localiser is offered on this task alone — it cannot do the rest of what any of these four can. These are also four models at one moment, and published scores move every few months, which is the reason for publishing the method rather than only the verdict.