Skip to content
Main Site News Console

Gemini 4 Argon for Robotics

· DeepMind Translated
DeepMind

October 13, 2026

Kanishka Rao, Principal Software Engineer

Image 3: argon__robotics__hero

Gemini 4 Argon marks a leap forward in physical understanding, code generation, and visual-semantic reasoning—unlocking entirely new workflows across the robotics stack and accelerating the path toward Physical AGI.

This capability jump is evident on the Embodied Reasoning (ERQA) benchmark, designed to evaluate a model’s ability to understand and solve physical spatial problems in 3D environments. Through pure reasoning alone, Argon achieved an 87% score, outperforming all other frontier models.

Image 4: gemini-4-argon__benchmark__robotics

As a general-purpose physical reasoning model, Argon can be applied across the entire robotics lifecycle. In this post, we share representative prompts for Argon across four key areas:

  • Robot Control → Mobile manipulation with MuJoCo simulation practice, zero-shot robot control, and closed-loop steering of video world models.
  • Real2Sim → Reconstructing physically calibrated simulation twins from robot camera feeds, egocentric human video, or hand-drawn paper sketches.
  • Physical Understanding → Agentic multimodal hindsight labeling, fine-grained understanding of dexterous manipulation, dense value functions, and progress estimation.
  • Research & Self-Improvement → Action refinement via simulation and iterative self-improvement on physical hardware.

Unlocking Robot Control with Argon

1. Spot Mobile Manipulation: Practicing in MuJoCo to Trash the Can

Argon controls a Boston Dynamics Spot® quadruped (12-DoF legs + 6-DoF arm) to learn a new skill: stepping on a foot-pedal trash bin to discard a soda can. Argon creates an articulated MuJoCo digital twin of the foot-pedal bin to practice the motion before stepping on the physical pedal and dropping the can into the open bin.

Video 1

Spot mobile manipulation — Real-to-Sim-to-Real foot-pedal bin deposit: Spot picks up the soda can, inspects the wrist camera, practices three-legged tripod balance in MuJoCo, steps on the foot pedal, confirms the gripper is centered over the open bin, and drops the can inside.

Prompt:

There's a Diet Coke can behind you and another on the black table on your left.
Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.
Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.

2. Zero-Shot Control

Argon controls a simple robot arm to reorient an overturned mug upright and place a crumpled napkin inside it. Argon reasons using visual coordinate wireframes and action primitives, completing the task over multiple interaction turns.

Video 2

Left: Third-person view of the physical robot; Right: Calibrated wrist camera view with projected 2 cm world coordinates and gripper-closing reticle crosshairs.

Prompt:

[Attached: wrist_cam.png, obs.json]
Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.
## Execution Discipline
- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.
## Visual Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base
- `+Y` (LEFT in image): World left | `-Y` (RIGHT): World right
- `+Z`: Height above table | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Current 3D pinch center
- `Cyan [ + ] brackets`: Current jaw opening and closing line
- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis
## Closed-Loop Turn Workflow
1. Inspect current state & view `wrist_cam.png`:
`curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).
2. Select VLM call granularity upfront on Turn 1:
- Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done`
- Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done`
- Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`
3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.
## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)
robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)
robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robot.place(x=0.24, y=0.06, z=0.075)
robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)
robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)
robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)
robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)
robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()

3. Omni Control

Argon steers a robot inside the Gemini Omni world model to arrange five scattered Scrabble tiles (G, R, A, N, O) neatly in a row to spell out A - R - G - O - N. Rather than specifying a single static prompt, Argon steers the robot in a closed visual loop. It generates timestamped action prompts and inspects intermediate rollout keyframes