October 13, 2026
Kanishka Rao, Principal Software Engineer

Gemini 4 Argon marks a leap forward in physical understanding, code generation, and visual-semantic reasoning—unlocking entirely new workflows across the robotics stack and accelerating the path toward Physical AGI.
This capability jump is evident on the Embodied Reasoning (ERQA) benchmark, designed to evaluate a model’s ability to understand and solve physical spatial problems in 3D environments. Through pure reasoning alone, Argon achieved an 87% score, outperforming all other frontier models.

As a general-purpose physical reasoning model, Argon can be applied across the entire robotics lifecycle. In this post, we share representative prompts for Argon across four key areas:
- Robot Control → Mobile manipulation with MuJoCo simulation practice, zero-shot robot control, and closed-loop steering of video world models.
- Real2Sim → Reconstructing physically calibrated simulation twins from robot camera feeds, egocentric human video, or hand-drawn paper sketches.
- Physical Understanding → Agentic multimodal hindsight labeling, fine-grained understanding of dexterous manipulation, dense value functions, and progress estimation.
- Research & Self-Improvement → Action refinement via simulation and iterative self-improvement on physical hardware.
Unlocking Robot Control with Argon
1. Spot Mobile Manipulation: Practicing in MuJoCo to Trash the Can
Argon controls a Boston Dynamics Spot® quadruped (12-DoF legs + 6-DoF arm) to learn a new skill: stepping on a foot-pedal trash bin to discard a soda can. Argon creates an articulated MuJoCo digital twin of the foot-pedal bin to practice the motion before stepping on the physical pedal and dropping the can into the open bin.
Spot mobile manipulation — Real-to-Sim-to-Real foot-pedal bin deposit: Spot picks up the soda can, inspects the wrist camera, practices three-legged tripod balance in MuJoCo, steps on the foot pedal, confirms the gripper is centered over the open bin, and drops the can inside.
Prompt:
There's a Diet Coke can behind you and another on the black table on your left.Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.2. Zero-Shot Control
Argon controls a simple robot arm to reorient an overturned mug upright and place a crumpled napkin inside it. Argon reasons using visual coordinate wireframes and action primitives, completing the task over multiple interaction turns.
Left: Third-person view of the physical robot; Right: Calibrated wrist camera view with projected 2 cm world coordinates and gripper-closing reticle crosshairs.
Prompt:
[Attached: wrist_cam.png, obs.json]
Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.
## Execution Discipline- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.
## Visual Coordinate Reference (Annotated on wrist_cam.png)- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base- `+Y` (LEFT in image): World left | `-Y` (RIGHT): World right- `+Z`: Height above table | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)- `Cyan + crosshair`: Current 3D pinch center- `Cyan [ + ] brackets`: Current jaw opening and closing line- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis
## Closed-Loop Turn Workflow1. Inspect current state & view `wrist_cam.png`: `curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).2. Select VLM call granularity upfront on Turn 1: - Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done` - Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done` - Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.
## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)robot.place(x=0.24, y=0.06, z=0.075)robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()3. Omni Control
Argon steers a robot inside the Gemini Omni world model to arrange five scattered Scrabble tiles (G, R, A, N, O) neatly in a row to spell out A - R - G - O - N. Rather than specifying a single static prompt, Argon steers the robot in a closed visual loop. It generates timestamped action prompts and inspects intermediate rollout keyframes