Blog
hedwigaihedwigaihedwigai

Interpretability

Does Physical Agent aim its tool calls at the object?

Junxiao Xu, Rui Yan and Ye Zhao at Georgia Tech released Physical Agent. It is Qwen3.5-4B fine-tuned to move a robot arm. It looks at a camera view and a wrist view, reasons in words, and then makes an ordinary tool call. The call’s arguments are up to eight positions for the robot’s hand.

A vision-language-action model, or VLA, usually swaps the language head for an action head. Physical Agent keeps the language model whole and writes its actions as text. The authors trained it on about 1,000 robot demonstrations, 21,267 decisions in all, in under 50 GPU-hours. They report 79.2% on the LIBERO benchmark, against 80% for the same base model trained as a VLA. And it can still answer exam questions: 56.8% on MMLU-Pro, against 61.8% for the base model and 0% for the VLA.

The weights are open, so we ran them next to the base model. We asked one question. When Physical Agent writes a position, does that position come from where the object is?

The short answer is that the format is learned well and the aim is not. The first call is mostly the same opening move for a task, wherever the object sits.

A Franka Panda arm above a table with a basket, a salad dressing bottle, a milk carton and several cansThe view from the camera on the robot's wrist, looking down at the table between its two fingers
What Physical Agent sees with each instruction: the camera view of the table (left) and the camera on the robot’s wrist (right). The robot is a Franka Panda arm. Here the instruction is “pick up the alphabet soup and place it in the basket”; the soup is the can at the back right.
A LIBERO scene for the instruction: pick up the black bowl between the plate and the ramekin and place it on the plate

LIBERO-Spatial

“pick up the black bowl between the plate and the ramekin and place it on the plate”

A LIBERO scene for the instruction: pick up the alphabet soup and place it in the basket

LIBERO-Object

“pick up the alphabet soup and place it in the basket”

A LIBERO scene for the instruction: open the middle drawer of the cabinet

LIBERO-Goal

“open the middle drawer of the cabinet”

A LIBERO scene for the instruction: put both the alphabet soup and the tomato sauce in the basket

LIBERO-10

“put both the alphabet soup and the tomato sauce in the basket”

One task from each of LIBERO’s four suites. Scenes from LIBERO by Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu and Peter Stone, rendered by us at the start of each task.

The call

Physical Agent writes a well-formed call on every frame

We rendered 400 start frames from LIBERO: four suites, ten tasks each, ten layouts per task. Each model got the task’s instruction, both camera views, and one declared tool, move_end_effector, which takes a list of up to eight targets with x, y, z and a gripper value.

Physical Agent called the tool on all 400 frames, and every call parsed. 362 of them carry exactly eight targets, the chunk size it was trained on. The first target lies a median 11.9 cm from the hand, so each call is a short, smooth step.

Qwen3.5-4B called the tool on 18 of 400 frames within 900 tokens. Given Physical Agent’s own reasoning and asked to finish the call, it writes zeros or round guesses. The ability to write a position in the robot’s coordinates is new, and the fine-tune put it there.

We also found where the change sits. All 723 weight tensors changed. Each layer's matrices moved a median of 1.8 to 3.7%, and the language model holds 90.8% of the change. The vision encoder holds 4.5%. In the embeddings, the rows that moved most are robot words, in Chinese and in capitalised English:

  • 瓶bottle
  • 接近approach
  • 命令command
  • 低low
  • 碗bowl
  • 握grip
  • 手腕wrist
  • 手指finger
  • 高度height
  • 位置position
  • 而不是rather than
  • ␣Bowl
  • ␣Can
  • ␣Hand
  • ␣Approach
  • ␣Close
  • ␣Basket

The aim

Moving the object does not move the target

A position the model writes should depend on where the object is. So before writing its first target, does the model work out where that target should be? The number it writes cannot tell us. A model can write a plausible number without having worked anything out. So we looked inside the model while it writes.

A language model writes one token at a time: a word, or a piece of a number. Before each token, what it has read passes through 32 layers, one after another. Each layer hands the next a list of 2,560 numbers. The last layer’s list decides the token. So whatever the model has worked out by some layer must be in that layer’s list.

We paused Physical Agent at the moment its next token would be the first target’s x. There we copied the list from every layer, on each of the 400 frames. The base model read exactly the same text, and we copied its lists at the same moment. Copying changes nothing; the model goes on to write as before.

Then we trained a probe: a small reader that takes one layer’s list and guesses where the first target is. Each frame has a known answer, the target Physical Agent actually wrote. We measure it as the distance from the gripper, in x, y and z. If the probe guesses well on frames it was not trained on, that layer already holds the target.

What a probe reads, and how it is tested. Each layer gets its own probe; layer 22 is shown.
what it has readthe tasktwo camera viewsits reasoning… "x": 32 layers, one after another12232the last layerpicks the token:-0.0248a copy of layer 22’s list:2,560 numbers0.41-1.200.072.30⋮0.90the probeeach × a weight,all added upits guess at thetarget, from thegripper (x, y, z)vs. what themodel wrote40 tasks:train the probe on 32test on 8 unseenDone five times with a different 8, so every task is tested once.

We kept the probe as simple as we could: multiply each of the 2,560 numbers by a weight, and add them up. Training only picks the weights. A reader this simple can only find what the layer holds in plain view. A more powerful reader could work the answer out by itself, and then its score would describe the reader, not the model.

A probe could also cheat by memorising tasks. The 400 frames come from 40 tasks, ten layouts each. So we trained each probe on 32 tasks and tested it on the 8 it had never seen. We did that five times, so every task was tested once. R² scores the guesses. At 1 the probe finds every target exactly. At 0 it does no better than always guessing the average target.

How well a probe on each layer predicts where the first target sits (R², on tasks it never saw)
-0.20.00.20.40.6emb8162432layer
Physical Agent, across tasksQwen3.5-4B, across tasksPhysical Agent, within a task

400 start frames from 40 tasks, five-fold, whole tasks held out. Values below −0.25 are drawn at −0.25.

Across tasks, Physical Agent’s probe peaks at R² = 0.53 after layer 22. The base model’s peaks at 0.23. So the fine-tuned model knows, deep in its layers, roughly where a task’s first move goes.

Within a task the probe reads almost nothing, at most 0.08. That could be because LIBERO barely moves the objects: across a task’s layouts, the named object shifts a median of 1.2 cm. So we moved the objects ourselves.

On ten libero_object tasks we slid the named object in 4 cm steps, from 8 cm one way to 8 cm the other, along x and along y. Then we asked Physical Agent for its first call on each frame.

Each call lists eight positions for the hand, one after another, so the eighth is where that move ends. We tracked the eighth. If the model aims at the object, it should shift as far as the object did.

The alphabet soup task, soup moved 8 cm one way

Soup moved 8 cm one way

The alphabet soup task, where libero puts it

Where LIBERO puts it

The alphabet soup task, soup moved 8 cm the other way

Soup moved 8 cm the other way

The alphabet soup, the can at the back right, slid along the table. Everything else stays where it was. Scenes from LIBERO by Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu and Peter Stone, rendered by us at the start of each task.
A map of the table, laid out as in the camera pictures above, with the robot at the top. Each shade is one run: where we put the alphabet soup, and the eight positions Physical Agent asked the hand to go through.
↑ the robotthe soup: 16 to 32 cm from the handthe hand starts herethe soup, put in five places 4 cm apartwhere each move ends (big dots): 11 to 26 cm from its soup10 cm

The alphabet soup moved 16 cm. The end of the move shifted 5 cm the other way. The reasoning was the same at every position: “I adjust the lateral approach so the fingers straddle the can rather than the neighboring objects, preparing to close.”

Every task looks like this. For each task we measured how far the end of the move shifts for each centimetre the object shifts. Following the object would give 1; ignoring it gives 0. The median is −0.01 along x and 0.02 along y, and no task reaches 0.5.

How far the end of the move shifts for each centimetre the object shifts, one dot per task
-0.500.51follows the objectmoved in xmoved in y

10 libero_object tasks × 2 layouts, the named object moved −8 to +8 cm in 4 cm steps.

The camera views do matter, but for something else. With both views black, the move ends a median 86.7 cm away, outside the robot’s reach. The views tell the model which frame its numbers are in. With the views from another task, it lands 3.6 cm from where it would with its own. A different layout of the same task moves it 3 cm, and the gap between two tasks’ average moves is 2.9 cm. The first call is a near-generic opening move.

Seeing the object

The fine-tune made pointing worse

If the model cannot aim, can it still see the object? We asked both models to box an object, in the format Qwen3.5 was trained to use: “Locate the ketchup. Answer only with JSON: {"bbox_2d": [x1, y1, x2, y2]}.”

We did this on 100 libero_object frames, where the box counts as right when its centre is nearest the named object. And we did it on 100 everyday photos from RefCOCO, where it counts as right when it overlaps the true box by more than half.

Boxes that land on the named object

LIBERO frames

Qwen3.5-4B64 / 100
Physical Agent0 / 100

RefCOCO photos

Qwen3.5-4B90 / 100
Physical Agent58 / 100

A box is four numbers from 0 to 1000 that place two corners on the image. For the alphabet soup the base model writes {"bbox_2d": [706, 460, 800, 640], "label": "alphabet soup"}.

On robot frames Physical Agent never writes a box like that. On the 84 frames where its answer could be read, it wrote four small decimals, at least one of them negative. No corner of an image sits at a negative position. These numbers are on the scale of the arm targets in its tool calls, which are in metres. On 49 frames it wrote a run of them, each a little further on, like the steps of an arm move: {"bbox_2d": [0.0541, -0.0620, 0.2575, -0.0065], "bbox_2d": [0.0552, -0.0657, 0.2573, -0.0066], …

On photos with no robot in them it still answers in the right format, but finds 58 of 100 against the base model’s 90. The vision encoder itself changed. Image by image, its output has a cosine of 0.75 with the base model’s on LIBERO frames and 0.88 on photos.

The base model is not perfect on LIBERO either. It finds the salad dressing 0 times in 10, taking it for the ketchup, and the tomato sauce 1 time and the bbq sauce 2. The small bottles in a 256-pixel render are hard to tell apart.

In the loop

In the loop, it often picks up the wrong object

The first call is one step. A robot in a loop sees new frames after every call and can correct itself. So we ran Physical Agent in a closed loop on the ten libero_object tasks. Each turn it saw both views and wrote one call; each target became one controller step; it had 30 turns.

In two runs of the ten tasks it finished 3 and 2. In 10 of the 20 episodes it first closed its gripper on the wrong object, and the mix-ups repeated from run to run:

  • Asked for the ketchup, took the bbq sauce
  • Asked for the alphabet soup, took the milk
  • Asked for the butter, took the tomato sauce
  • Asked for the milk, took the butter
  • Asked for the chocolate pudding, took the orange juice

In an earlier test episode it was told to pick up the alphabet soup. It went for the salad dressing, and then wrote that “the can is now near the basket opening.” The reasoning describes the task it was given, not the scene in front of it.

This loop is ours, not the authors’. Their tool schema, their tool replies and their controller are not published yet, and the authors report far higher success on this suite. So 3 of 10 says how it runs in our harness, not in theirs. The wrong grasps are the part that does not depend on the harness: the hand went, accurately, to the wrong thing.

Try it yourself

Does the hand go to the object you name?

  • One of the ten LIBERO grocery scenes, and your own instruction
  • Both camera views, the reasoning and the eight positions at every turn
  • Go on from where the arm stopped with a new instruction

Three runs free, then 12 cents a run.

Run the arm
Continue in

What it kept

On general text, the fine-tune changed little

The claim that sets Physical Agent apart is that it stays a language model. We checked it from the inside. On 150 MMLU-Pro questions, the KL divergence between the two models’ next-token distributions averages 0.12 nats, with a median of 0.05. On 100 robot prompts it is 4.41 nats.

That fits the authors’ 56.8% on MMLU-Pro. The model still talks like Qwen3.5-4B about anything that is not a robot, and talks very differently about robots.

What it means

The action format works; the aim is the next problem

We like this design. Writing actions as tool calls keeps a model’s language, and the numbers show it: general text barely moves, and the calls are well-formed every time. A thousand demonstrations and a few dozen GPU-hours bought that.

What they did not buy is aim. The first call does not follow the object, and the fine-tune made the model worse at finding objects in pictures. On LIBERO, where each task’s objects barely move between layouts, a good opening move for the task goes a long way. That may be why the authors’ LIBERO-PRO scores, where objects move further, drop to 23.3% when the task changes and 5.8% when positions change.

Base Qwen3.5-4B can already box most objects. A model that pointed first and moved second would have the object’s position in its own words before it wrote a target.

Try it yourself

Does the hand go to the object you name?

  • One of the ten LIBERO grocery scenes, and your own instruction
  • Both camera views, the reasoning and the eight positions at every turn
  • Go on from where the arm stopped with a new instruction

Three runs free, then 12 cents a run.

Run the arm
Continue in
  • Models: GatechLIDAR/Physical-Agent-Ninelong-4B and Qwen/Qwen3.5-4B in bf16, served with vLLM. Calls used thinking on and greedy decoding. Pointing used thinking off.
  • Frames: LIBERO through robosuite 1.4.1 and MuJoCo 3.2.3, 256 × 256 agent and wrist views, rotated 180°, the way LIBERO-trained policies usually see them. Targets are in the simulator’s coordinates, in metres.
  • Probes: ridge regression, the penalty picked by an inner cross-validation, five outer folds split by task. “Within a task” scores the target minus its task mean; a shuffled-label control scored below zero at every layer.
  • Weight change: ‖ΔW‖ / ‖W‖ for each matrix. The embedding change is low-rank, with 42% of it in a single direction.
  • Closed loop: each target was one operational-space step toward it of at most 5 cm (three steps in the second run), with the gripper value applied. The tool reply carried the two new views. The figures on LIBERO, LIBERO-PRO and MMLU-Pro quoted above are the authors’.

More head-to-head evals

Particle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash in Claude Code with hedwigai workbooksParticle-1, DeepSeek V4.1 Flash and GLM-5.3-Flash, 180 runs graded blind, rows right: 91.2%, 89.4% and 90.8%, intervals overlapping.
MeasureParticle-1DeepSeek V4.1 FlashGLM-5.3-Flash
Rows right, blind91.2%89.4%90.8%
Median seconds per run375315144
Read the comparison
GLM-5.3 against DeepSeek V4.1 Flash on 75 questionsTwo models answer the same 75 questions; one is always right, the other 14× cheaper.
MeasureGLM-5.3DeepSeek V4.1 Flash
Right, of 757571
Seconds a question, median3.91.8
Read the comparison
How we made a long-running AI agent for knowledge work 4× fasterOne cited knowledge work task through 13 experiments: 937 s to 233 s, with the model held constant.
MeasureFirst runBest since
Landscape, seconds937233
Memo, seconds759602
Read the report
Ten models plan a change to a spreadsheet: which do it in under a minute, traced call by callTen models drive one agent harness through the same spreadsheet request; six write the plan, timed and traced call by call.
MeasureQwen3 235BGLM-5.2Gemini 3.8 FlashClaude Opus 5
Seconds to the plan, median27485794
Plans written, of three runs2333
Read the comparison
SubQ, jev, Perplexity's Decider and Bespoke's Nimble on biotech diligence memos: finding the sentences a column is aboutFour decision models judge every sentence of ten biotech diligence memos for five columns, scored against sentences a person marked.
MeasureSubQjevDeciderNimble
Found, mean of five columns at section length64%65%66%68%
Mechanism of action, found at section length57%51%61%54%
Read the comparison
Leads with a reason to write: hedwigai ask, Apollo and treg on one brief40 leads each, every one checked: the right person, an inbox that accepts, and a dated reason to write.
MeasureaskApollotreg
Usable leads13 of 407 of 406 of 28
Cost per usable lead$0.037$0.27$0.23
Read the comparison
Self-hosted decisions for ask: Perplexity's open Decider in place of jev on 92 company questionsThe decisions inside hedwigai ask made by Perplexity's open Decider on two NVIDIA RTX 3090s, against jev.
Measureask with jevask with Decider
Questions right, of 928989
Wrong answers12
Read the comparison
Clinical decisions: jev and ask on System One, PubMedQA, SciFact and a 370-decision setEdgeEvals' System One board, PubMedQA, SciFact, and a clinical set built to System One's rules.
Measurejevask
Our 370 clinical decisions, right94.6%93.0%
PubMedQA, right (best EdgeEvals scored: 62.2%)77.0%74.4%
Read the comparison
Executive lookup for US-listed companies: Claude Opus 5.5 and ask on 95 questionsWho runs a public company, and how to email them: a frontier model from memory against hedwigai ask.
MeasureaskClaude Opus 5.5
Questions right, of 955963
Small companies right, of 20139
Read the comparison
Rental comps: floor-plan rents and amenities on 14 listings, five assistants comparedRent per floor plan and amenities for US rentals: hedwigai realestate-us against Claude, GPT, Gemini and Perplexity with web search.
Measurerealestate-usBest of four assistants
Floor-plan rents right, of 262623
Amenities claimed the listing doesn't state12129
Read the comparison
Company screening from SEC filings: jev, Vela 2.0 and ask on 92 yes/no questionsYes-or-no questions about public companies, answered by two decision models alone and by hedwigai ask.
Measureaskjev
Right, of the questions answered98%79%
Wrong answers, of 92219
Read the comparison
Defect localization in industrial images: four frontier models and a trained localizerFour frontier models asked to put a box around what they just described.
MeasureTrained localiserBest of four frontier models
Mean overlap with the defect (IoU, 0 to 1)0.4210.109
Read the comparison