Interpretability
Does Physical Agent aim its tool calls at the object?
Junxiao Xu, Rui Yan and Ye Zhao at Georgia Tech released Physical Agent. It is Qwen3.5-4B fine-tuned to move a robot arm. It looks at a camera view and a wrist view, reasons in words, and then makes an ordinary tool call. The call’s arguments are up to eight positions for the robot’s hand.
A vision-language-action model, or VLA, usually swaps the language head for an action head. Physical Agent keeps the language model whole and writes its actions as text. The authors trained it on about 1,000 robot demonstrations, 21,267 decisions in all, in under 50 GPU-hours. They report 79.2% on the LIBERO benchmark, against 80% for the same base model trained as a VLA. And it can still answer exam questions: 56.8% on MMLU-Pro, against 61.8% for the base model and 0% for the VLA.
The weights are open, so we ran them next to the base model. We asked one question. When Physical Agent writes a position, does that position come from where the object is?
The short answer is that the format is learned well and the aim is not. The first call is mostly the same opening move for a task, wherever the object sits.



LIBERO-Spatial
“pick up the black bowl between the plate and the ramekin and place it on the plate”

LIBERO-Object
“pick up the alphabet soup and place it in the basket”

LIBERO-Goal
“open the middle drawer of the cabinet”

LIBERO-10
“put both the alphabet soup and the tomato sauce in the basket”
frames answered with a tool call
It writes the call every time
Given a tool to move the hand, Physical Agent called it on every frame. Qwen3.5-4B called it on 18.
Read how we measured itcm of target per cm of object
The target does not follow the object
We slid the named object across a 16 cm range. The first call's targets stayed where they were.
Read how we measured itboxes found on RefCOCO, base 90%
It points worse than the base model
Asked to box an object in a photo, it finds 58 of 100. On robot frames it fills the box with arm coordinates in metres.
Read how we measured itnats from the base model on MMLU-Pro
General text barely moved
On exam questions its next-token distribution is close to the base model's. On robot prompts the gap is 4.41 nats.
Read how we measured itThe call
Physical Agent writes a well-formed call on every frame
We rendered 400 start frames from LIBERO: four suites, ten tasks each, ten layouts per task. Each model got the task’s instruction, both camera views, and one declared tool, move_end_effector, which takes a list of up to eight targets with x, y, z and a gripper value.
Physical Agent called the tool on all 400 frames, and every call parsed. 362 of them carry exactly eight targets, the chunk size it was trained on. The first target lies a median 11.9 cm from the hand, so each call is a short, smooth step.
Qwen3.5-4B called the tool on 18 of 400 frames within 900 tokens. Given Physical Agent’s own reasoning and asked to finish the call, it writes zeros or round guesses. The ability to write a position in the robot’s coordinates is new, and the fine-tune put it there.
We also found where the change sits. All 723 weight tensors changed. Each layer's matrices moved a median of 1.8 to 3.7%, and the language model holds 90.8% of the change. The vision encoder holds 4.5%. In the embeddings, the rows that moved most are robot words, in Chinese and in capitalised English:
- 瓶bottle
- 接近approach
- 命令command
- 低low
- 碗bowl
- 握grip
- 手腕wrist
- 手指finger
- 高度height
- 位置position
- 而不是rather than
- ␣Bowl
- ␣Can
- ␣Hand
- ␣Approach
- ␣Close
- ␣Basket
The aim
Moving the object does not move the target
A position the model writes should depend on where the object is. So before writing its first target, does the model work out where that target should be? The number it writes cannot tell us. A model can write a plausible number without having worked anything out. So we looked inside the model while it writes.
A language model writes one token at a time: a word, or a piece of a number. Before each token, what it has read passes through 32 layers, one after another. Each layer hands the next a list of 2,560 numbers. The last layer’s list decides the token. So whatever the model has worked out by some layer must be in that layer’s list.
We paused Physical Agent at the moment its next token would be the first target’s x. There we copied the list from every layer, on each of the 400 frames. The base model read exactly the same text, and we copied its lists at the same moment. Copying changes nothing; the model goes on to write as before.
Then we trained a probe: a small reader that takes one layer’s list and guesses where the first target is. Each frame has a known answer, the target Physical Agent actually wrote. We measure it as the distance from the gripper, in x, y and z. If the probe guesses well on frames it was not trained on, that layer already holds the target.
We kept the probe as simple as we could: multiply each of the 2,560 numbers by a weight, and add them up. Training only picks the weights. A reader this simple can only find what the layer holds in plain view. A more powerful reader could work the answer out by itself, and then its score would describe the reader, not the model.
A probe could also cheat by memorising tasks. The 400 frames come from 40 tasks, ten layouts each. So we trained each probe on 32 tasks and tested it on the 8 it had never seen. We did that five times, so every task was tested once. R² scores the guesses. At 1 the probe finds every target exactly. At 0 it does no better than always guessing the average target.
400 start frames from 40 tasks, five-fold, whole tasks held out. Values below −0.25 are drawn at −0.25.
Across tasks, Physical Agent’s probe peaks at R² = 0.53 after layer 22. The base model’s peaks at 0.23. So the fine-tuned model knows, deep in its layers, roughly where a task’s first move goes.
Within a task the probe reads almost nothing, at most 0.08. That could be because LIBERO barely moves the objects: across a task’s layouts, the named object shifts a median of 1.2 cm. So we moved the objects ourselves.
On ten libero_object tasks we slid the named object in 4 cm steps, from 8 cm one way to 8 cm the other, along x and along y. Then we asked Physical Agent for its first call on each frame.
Each call lists eight positions for the hand, one after another, so the eighth is where that move ends. We tracked the eighth. If the model aims at the object, it should shift as far as the object did.

Soup moved 8 cm one way

Where LIBERO puts it

Soup moved 8 cm the other way
The alphabet soup moved 16 cm. The end of the move shifted 5 cm the other way. The reasoning was the same at every position: “I adjust the lateral approach so the fingers straddle the can rather than the neighboring objects, preparing to close.”
Every task looks like this. For each task we measured how far the end of the move shifts for each centimetre the object shifts. Following the object would give 1; ignoring it gives 0. The median is −0.01 along x and 0.02 along y, and no task reaches 0.5.
10 libero_object tasks × 2 layouts, the named object moved −8 to +8 cm in 4 cm steps.
The camera views do matter, but for something else. With both views black, the move ends a median 86.7 cm away, outside the robot’s reach. The views tell the model which frame its numbers are in. With the views from another task, it lands 3.6 cm from where it would with its own. A different layout of the same task moves it 3 cm, and the gap between two tasks’ average moves is 2.9 cm. The first call is a near-generic opening move.
Seeing the object
The fine-tune made pointing worse
If the model cannot aim, can it still see the object? We asked both models to box an object, in the format Qwen3.5 was trained to use: “Locate the ketchup. Answer only with JSON: {"bbox_2d": [x1, y1, x2, y2]}.”
We did this on 100 libero_object frames, where the box counts as right when its centre is nearest the named object. And we did it on 100 everyday photos from RefCOCO, where it counts as right when it overlaps the true box by more than half.
LIBERO frames
RefCOCO photos
A box is four numbers from 0 to 1000 that place two corners on the image. For the alphabet soup the base model writes {"bbox_2d": [706, 460, 800, 640], "label": "alphabet soup"}.
On robot frames Physical Agent never writes a box like that. On the 84 frames where its answer could be read, it wrote four small decimals, at least one of them negative. No corner of an image sits at a negative position. These numbers are on the scale of the arm targets in its tool calls, which are in metres. On 49 frames it wrote a run of them, each a little further on, like the steps of an arm move: {"bbox_2d": [0.0541, -0.0620, 0.2575, -0.0065], "bbox_2d": [0.0552, -0.0657, 0.2573, -0.0066], …
On photos with no robot in them it still answers in the right format, but finds 58 of 100 against the base model’s 90. The vision encoder itself changed. Image by image, its output has a cosine of 0.75 with the base model’s on LIBERO frames and 0.88 on photos.
The base model is not perfect on LIBERO either. It finds the salad dressing 0 times in 10, taking it for the ketchup, and the tomato sauce 1 time and the bbq sauce 2. The small bottles in a 256-pixel render are hard to tell apart.
In the loop
In the loop, it often picks up the wrong object
The first call is one step. A robot in a loop sees new frames after every call and can correct itself. So we ran Physical Agent in a closed loop on the ten libero_object tasks. Each turn it saw both views and wrote one call; each target became one controller step; it had 30 turns.
In two runs of the ten tasks it finished 3 and 2. In 10 of the 20 episodes it first closed its gripper on the wrong object, and the mix-ups repeated from run to run:
- Asked for the ketchup, took the bbq sauce
- Asked for the alphabet soup, took the milk
- Asked for the butter, took the tomato sauce
- Asked for the milk, took the butter
- Asked for the chocolate pudding, took the orange juice
In an earlier test episode it was told to pick up the alphabet soup. It went for the salad dressing, and then wrote that “the can is now near the basket opening.” The reasoning describes the task it was given, not the scene in front of it.
This loop is ours, not the authors’. Their tool schema, their tool replies and their controller are not published yet, and the authors report far higher success on this suite. So 3 of 10 says how it runs in our harness, not in theirs. The wrong grasps are the part that does not depend on the harness: the hand went, accurately, to the wrong thing.
Try it yourself
Does the hand go to the object you name?
- One of the ten LIBERO grocery scenes, and your own instruction
- Both camera views, the reasoning and the eight positions at every turn
- Go on from where the arm stopped with a new instruction
Three runs free, then 12 cents a run.
Run the armWhat it kept
On general text, the fine-tune changed little
The claim that sets Physical Agent apart is that it stays a language model. We checked it from the inside. On 150 MMLU-Pro questions, the KL divergence between the two models’ next-token distributions averages 0.12 nats, with a median of 0.05. On 100 robot prompts it is 4.41 nats.
That fits the authors’ 56.8% on MMLU-Pro. The model still talks like Qwen3.5-4B about anything that is not a robot, and talks very differently about robots.
What it means
The action format works; the aim is the next problem
We like this design. Writing actions as tool calls keeps a model’s language, and the numbers show it: general text barely moves, and the calls are well-formed every time. A thousand demonstrations and a few dozen GPU-hours bought that.
What they did not buy is aim. The first call does not follow the object, and the fine-tune made the model worse at finding objects in pictures. On LIBERO, where each task’s objects barely move between layouts, a good opening move for the task goes a long way. That may be why the authors’ LIBERO-PRO scores, where objects move further, drop to 23.3% when the task changes and 5.8% when positions change.
Base Qwen3.5-4B can already box most objects. A model that pointed first and moved second would have the object’s position in its own words before it wrote a target.
Try it yourself
Does the hand go to the object you name?
- One of the ten LIBERO grocery scenes, and your own instruction
- Both camera views, the reasoning and the eight positions at every turn
- Go on from where the arm stopped with a new instruction
Three runs free, then 12 cents a run.
Run the arm- Models: GatechLIDAR/Physical-Agent-Ninelong-4B and Qwen/Qwen3.5-4B in bf16, served with vLLM. Calls used thinking on and greedy decoding. Pointing used thinking off.
- Frames: LIBERO through robosuite 1.4.1 and MuJoCo 3.2.3, 256 × 256 agent and wrist views, rotated 180°, the way LIBERO-trained policies usually see them. Targets are in the simulator’s coordinates, in metres.
- Probes: ridge regression, the penalty picked by an inner cross-validation, five outer folds split by task. “Within a task” scores the target minus its task mean; a shuffled-label control scored below zero at every layer.
- Weight change: ‖ΔW‖ / ‖W‖ for each matrix. The embedding change is low-rank, with 42% of it in a single direction.
- Closed loop: each target was one operational-space step toward it of at most 5 cm (three steps in the second run), with the gripper value applied. The tool reply carried the two new views. The figures on LIBERO, LIBERO-PRO and MMLU-Pro quoted above are the authors’.
More head-to-head evals
| Measure | Particle-1 | DeepSeek V4.1 Flash | GLM-5.3-Flash |
|---|---|---|---|
| Rows right, blind | 91.2% | 89.4% | 90.8% |
| Median seconds per run | 375 | 315 | 144 |
| Measure | GLM-5.3 | DeepSeek V4.1 Flash |
|---|---|---|
| Right, of 75 | 75 | 71 |
| Seconds a question, median | 3.9 | 1.8 |
| Measure | First run | Best since |
|---|---|---|
| Landscape, seconds | 937 | 233 |
| Memo, seconds | 759 | 602 |
| Measure | Qwen3 235B | GLM-5.2 | Gemini 3.8 Flash | Claude Opus 5 |
|---|---|---|---|---|
| Seconds to the plan, median | 27 | 48 | 57 | 94 |
| Plans written, of three runs | 2 | 3 | 3 | 3 |
| Measure | SubQ | jev | Decider | Nimble |
|---|---|---|---|---|
| Found, mean of five columns at section length | 64% | 65% | 66% | 68% |
| Mechanism of action, found at section length | 57% | 51% | 61% | 54% |
| Measure | ask | Apollo | treg |
|---|---|---|---|
| Usable leads | 13 of 40 | 7 of 40 | 6 of 28 |
| Cost per usable lead | $0.037 | $0.27 | $0.23 |
| Measure | ask with jev | ask with Decider |
|---|---|---|
| Questions right, of 92 | 89 | 89 |
| Wrong answers | 1 | 2 |
| Measure | jev | ask |
|---|---|---|
| Our 370 clinical decisions, right | 94.6% | 93.0% |
| PubMedQA, right (best EdgeEvals scored: 62.2%) | 77.0% | 74.4% |
| Measure | ask | Claude Opus 5.5 |
|---|---|---|
| Questions right, of 95 | 59 | 63 |
| Small companies right, of 20 | 13 | 9 |
| Measure | realestate-us | Best of four assistants |
|---|---|---|
| Floor-plan rents right, of 26 | 26 | 23 |
| Amenities claimed the listing doesn't state | 12 | 129 |
| Measure | ask | jev |
|---|---|---|
| Right, of the questions answered | 98% | 79% |
| Wrong answers, of 92 | 2 | 19 |
| Measure | Trained localiser | Best of four frontier models |
|---|---|---|
| Mean overlap with the defect (IoU, 0 to 1) | 0.421 | 0.109 |