Writing

Defect localization in industrial images: four frontier models and a trained localizer

Factories photograph what they make (a weld, a casting, a panel, a part on a line) to catch defects. Frontier models, the largest general AI models from the big labs, can describe these pictures well. The harder job is localization: marking where in the image the defect is, by drawing a box around it. That box is what a machine further down the line crops, measures, or moves a tool to. When no model can do it, the looking falls to a person.

We gave four frontier models that can read images the same 57 pictures. Each was asked for a verdict about the defect and for a box. We compared the verdicts with a rule that never looks: one that gives the same answer every time without opening the image. We compared the boxes with a trained localizer, a small model trained only to draw these boxes.

Both beat all four models. The rule that never looks got 96.5% of verdicts right; the best model got 86.0%. 64% of the trained localizer's boxes landed close enough to use, against at most 24% for any of the four. On industrial images, pointing at a defect is a job for a model trained to point, and it is not a prompt away.

The short version

What comes back from a hundred requests

The first table anyone should read. Some models skip the hard images and draw no box. A model like that looks excellent if you only score the boxes it chose to draw. Counting per hundred requests, every skipped image counts against it, which is what it costs you.

Every figure is per 100 requests. 0.3 overlap means the box and the true box share about a third of their area — enough to crop around, to hand to a person, or to cue a second model. 0.5 is the loosest thing most teams will act on directly. Neither threshold is generous.

The full results

The run exactly as it printed, with the two comparisons that are not frontier models: the rule that never looks (“constant baseline”) and the trained localizer. Everything in the table above is arithmetic on these columns.

n is how many images the model was scored on. PF (parse failures) is answers that could not be read as an answer at all. boxes is how many of the n images the model returned a box for. meanIoU and medIoU are the average and the middle overlap; ≥0.3 and ≥0.5 are the share of boxes at or above that overlap. The overlap columns count only boxes drawn, not every image, which flatters a model that declined to guess.

Where each model wins

Two of the four take all five dimensions between them, and the way they split is the finding. Qwen3-VL-30B-A3B-Instruct answers on nearly every image, is never unreadable and has the best verdict — and returns 8.7 usable boxes per hundred requests. DeepSeek-V4-Flash-Vision-Exp has the boxes most likely to land, and answers least often, refuses most often and gets the verdict wrong most often. Which of those you can live with is an engineering question about your own system, not a ranking.

What the numbers mean

What each failure costs

These are four different failures with four different owners, and they are not degrees of one another. A system that handles the first three can still be ruined by the fourth: a confident box in the wrong place is the only one that makes a machine act.

What this means for a machine that acts on the box

Sometimes the box is not a picture for a person to look at. It is a position that a robot arm or other machine moves to. That raises the bar. A missing box means work not done. A wrong box means the machine moves to the wrong place. And there is no reader in between to notice that the answer looks odd.

These are not weak models

Each publishes a serious result. One line and a source each — the four report different benchmarks, and lining them up would invent a comparability nobody has established.

What is missing from that column is the point. None of those benchmarks measures putting a box on an object. A model can answer questions about an image at 74 on MMMU and still have a median overlap of zero when asked where the thing is, because answering about an image and pointing at part of one are different abilities — and only the first is routinely published. A procurement decision made on the published numbers is a decision made on the half of the problem that was measured.

What this means if you are buying

What can still go wrong

• 57 images is a small set, and it is ours. It says what these models did on this data, not what they would do on yours. The gap is large enough to act on; the exact figures are not precise enough to rank the four against each other.
• Not every model was scored on every image. An answer that was refused or could not be read adds no row, so one model was scored on 45 images and the rest on 57.
• No prompt engineering was tried beyond asking plainly.
• The trained localizer does this one job. It cannot do the rest of what any of the four frontier models can.
• These are four models at one moment. Published scores move every few months, which is why the method is published and not only the verdict.