VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World VLA Evaluation

TL;DR

Problem & Motivation

Vision-Language-Action (VLA) models like OpenVLA, π₀, and π₀.₅ are touted as general-purpose robot brains, but how do you actually evaluate them in the real world? Existing options all hurt:

• Simulation benchmarks (Meta-World, LIBERO, RoboCasa) are cheap but suffer the sim-to-real gap; numbers don't transfer.
• Grasp-only real-world benchmarks (REPLAB, SceneReplica) lack task diversity.
• High-end real-world benchmarks (FurnitureBench, ManipulationNet) need expensive hardware and careful engineering to replicate.
• Centralized eval services (RoboArena) force you to ship policies to someone else's robot — opaque and slow.

VLA-REPLICA's design principle: real-world evaluation should be accessible, standardized, and locally executable. Every lab should be able to build the same setup, run the same scenes, and get comparable numbers — without phoning home.

Key Concepts

SO-101 arm

LeRobot's 6-DoF, mostly 3D-printed, self-assembled arm driven by Dynamixel STS3215 servos. Cheap and hackable, but every unit calibrates to its own raw encoder offsets — hence the need for a "universal action space".

Universal action space

A normalization (per Eqs. 1–3) that converts raw 12-bit servo encoder values to joint angles in degrees so a policy trained on one SO-101 can drive a different SO-101 unchanged.

Light box

A 32×32×32 inch photography softbox at ~5600 K that encloses the workspace, eliminating ambient-lighting drift between labs and times of day.

Test scene

A fully specified initial configuration of target + distractor objects on the table. The benchmark ships 50 ID + 40 OOD scenes; users align objects to a reference photo via live top-camera overlay.

ID vs. OOD evaluation

In-distribution: same objects, varied placement within the support of training demos. Out-of-distribution: deliberate shifts in object color/shape or task semantics (e.g., new repetition counts).

AprilTag calibration

A fiducial at a fixed location next to the arm whose 6-DoF pose is logged, so a new rig's top/wrist cameras can be aligned to the original via tag pose + visual overlay matching.

The Method

VLA-REPLICA is a recipe, not a model. It has three pillars: a cheap standardized rig, an action-space normalization that makes SO-101 arms interchangeable, and a scene reconstruction protocol that fixes initial conditions across labs.

Pillar 1 — Hardware bill of materials

The paper reports a total cost under $1,100 USD (~$1,050 in the intro), composed of:

An initial reproducibility check: a user with no prior knowledge of the benchmark built the rig in under one hour following the appendix instructions.

Pillar 2 — SO-101 action normalization

Each SO-101 unit's servos have different raw offsets, so raw encoder commands are not portable. The paper defines a universal action as the joint-angle vector in degrees, recovered from per-arm calibration constants (offsets, min/max sweeps). A trained policy outputs degrees; each lab's launcher converts back to that arm's raw encoder values via Eq. (3). The math is in §Math Walkthrough.

Pillar 3 — Scene reconstruction protocol

Object placements are matched to reference photos via a live top-camera overlay, the same trick SceneReplica used for grasp scenes. Combined with AprilTag-anchored camera alignment, this lets a fresh rig recreate the exact same initial conditions for all 90 test scenes.

Task suite (10 tasks, 500 demos)

50 demos per task, collected by SO-101 leader→follower teleoperation, with a defined "home position" at start/end of each episode.

Math Walkthrough

The only equations in the paper formalize the SO-101 universal action space. Let \(\hat{\mathbf{a}} = \{a_1,\dots,a_6\}\) be the 6-D vector of raw servo encoder values sent to the arm, with each \(a_i\in[0,4095]\). Calibration captures three vectors per arm: the offsets at the canonical "L-shape" zero-pose, and the min/max sweeps over joint limits, \(\{\mathbf{a}^{\text{offset}},\mathbf{a}^{\text{min}},\mathbf{a}^{\text{max}}\}\).

Per-arm scaling matrix

Define a diagonal matrix that converts encoder ticks into degrees:

Forward: encoder → universal action

Demonstrations are stored in \(\hat{\mathbf{u}}\), so policies learn to output degrees, not raw ticks.

Inverse: universal action → encoder

At deployment, each lab runs Eq. (3) with its own calibration constants. As long as every SO-101 is calibrated, the same trained policy drives every arm — this is what makes cross-lab reproducibility (Table 6) possible.

Results

Seven baselines were trained / fine-tuned for 40k steps on the same 500 demos (LeRobot codebase). For X-VLA the vision encoder, language encoder and policy transformer are all fine-tuned; for π₀ only the VLM is fine-tuned and the vision encoder is frozen.

In-distribution success rate (5 runs per task, 10 ID tasks)

π₀.₅ is the clear ID winner. Pretrained VLAs (π₀, π₀.₅, SmolVLA) generally beat from-scratch imitation learning (ACT, DiT-D, DiT-F), supporting the claim that VLA pretraining provides a useful initialization for low-data target-domain adaptation.

Out-of-distribution success rate (5 runs, 8 OOD tasks)

Color/shape generalization survives — π₀, π₀.₅, SmolVLA keep OOD performance close to ID on pick-and-place and folding. Memory-counting collapses across the board: when asked to "shake the pepper 2 times" (unseen count), policies just keep shaking. None of the seven methods solves task 8 (press button n×) OOD.

Cross-lab reproducibility (independently built rig)

A second VLA-REPLICA setup was built at a different location by a user with no prior experience. Four models were re-evaluated on the highest-success subsets of ID (tasks {1,2,5,6,7}) and OOD (tasks {1,2,5}):

Differences are within a few task-cells (≤ 0.05 on the average), supporting the headline claim: a policy fine-tuned on the released demos can be evaluated on an independently built copy of the rig and get comparable numbers.

Intuition & Analogies

Limitations & Open Questions

• Single embodiment. Only the SO-101 6-DoF arm; tabletop manipulation only. No bimanual, no mobile bases, no different grippers.
• Small task scale. 10 tasks / 500 demos is dwarfed by simulation benchmarks (Meta-World 50, LIBERO/RoboCasa 100) and by web-scale VLA pretraining corpora.
• Residual variation across rigs. Even after AprilTag + overlay calibration, small differences in lighting, hardware tolerances and object placement can introduce inconsistencies — the reproducibility table is "close", not "identical".
• Memory / temporal counting unsolved. Every evaluated method, including π₀.₅, collapses on the counting tasks OOD. The benchmark exposes the gap but does not propose an architectural fix.
• Rigid-object grasping is fragile. Stack-blocks and put-blocks-in-box are hard for all baselines even ID, suggesting gripper-pose precision is a bottleneck the platform itself imposes.
• 40k-step fine-tuning budget is fixed. The relative ranking of methods could shift with longer training or different recipes; the paper does not sweep this.

Takeaways

• Reproducible real-world VLA evaluation is now ~$1,050. A light box, an SO-101, an D455, a webcam, and an AprilTag — assembled in under an hour.
• Action-space normalization (Eqs. 1–3) is the key trick that makes policies portable between physically different SO-101 units.
• π₀.₅ leads the field on both ID (0.54) and OOD (0.35) splits, with π₀ and SmolVLA tied for second OOD at 0.30. From-scratch imitation learning (ACT, DiT-D, DiT-F) trails everywhere.
• Pretrained VLAs generalize to colors and shapes but fail at counting — a clean, low-cost demonstration of what current VLA pretraining does and doesn't give you.
• Cross-lab numbers reproduce within ~0.05 on a rig built blind by a new user, validating the "anywhere in the world" claim for the subset evaluated.

Links & Citation

📄 Original PDF arXiv:2605.20774 ↗ Project page ↗