3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding
TL;DR
3DVLA is a plug-and-play 3D add-on for pretrained 2D Vision-Language-Action models. It bolts on (1) multi-view spatial fusion with 3D positional encoding, (2) object-centric 3D instance probes, (3) a masked self-supervised predictor that hallucinates occluded geometry, and (4) an uncertainty-gated geometry aggregator. With only ~0.086B extra parameters, it reaches 86.0% average on LIBERO-Plus (vs. 84.2% for ฯโ.โ ) and lifts ฯโ from 46.4% โ 54.5% (Easy) and 16.3% โ 23.2% (Hard) on RoboTwin 2.0.
Problem & Motivation
Vision-Language-Action (VLA) models like OpenVLA, ฯโ, and ฯโ.โ inherit huge 2D vision-language pre-training, but their understanding of the 3D scene is thin. The paper identifies three coupled failures:
- Weak 3D spatial extraction. Existing VLAs encode single views or loosely fuse multiple views without ever reconstructing a metric 3D frame, so they cannot reason about precise 6-DoF poses.
- Insufficient 3D instance understanding. Objects are treated as flat image regions; the model has no volumetric extent, no 3D boundary, and cannot reliably tell which pixels belong to which object in 3D.
- Fragile reasoning under occlusion. On LIBERO-Plus, simply nudging the camera so the target is partially occluded crashes SOTA VLAs' success rates โ a purely-2D model cannot infer the missing geometry.
Off-the-shelf 3D perception backbones (ImVoxelNet, R4Det, etc.) could fix this, but plugging them into a VLA forces a redesign that discards 2D VLM pre-training and demands dense 3D instance annotations that don't exist at robotic-data scale. 3DVLA aims to retain VLM priors, require no extra manual labels, and still inject robust 3D reasoning.
Key Concepts
- Multi-View Spatial Fusion
- Flattens \(V\) view feature maps into one unordered token sequence, adds a learned absolute-3D positional embedding from unprojected coordinate grids, and runs a Transformer encoder. Output: a globally consistent 3D memory \(\mathcal{M}\).
- 3D State Probes
- Learnable queries \((\mathbf{c}_j,\mathbf{p}_j)\): a semantic vector plus an explicit 3D anchor point. They are projected into each view to do multi-scale deformable sampling, then refined with occlusion-aware gating and a
tanh-bounded coordinate update. - Global 3D Hungarian Matching
- Pseudo-GT 2D boxes/masks come from a frozen open-world detector (VL-SAM v2); their 3D centroids are obtained by unprojecting with depth. A single Hungarian assignment in 3D eliminates the per-view ID conflicts that plague 2D matching.
- Coordinate-Driven 3D Predictor
- An I-JEPA-style masked self-supervised branch โ but instead of discarding the predictor after pre-training, 3DVLA keeps it and reuses it at inference to synthesize visual tokens for occluded 3D locations.
- Continuous 3D RoPE
- Rotary positional encoding applied to queries/keys whose phase angle \(\theta_d=p\cdot 10000^{-2d/D_{axis}}\) is computed continuously per axis \(p\in\{x,y,z\}\), giving the attention a relative-distance prior in 3D.
- Uncertainty-Guided Routing
- A gate that absorbs synthesized 3D context into the instance representation only when the 2D classifier is unsure (\(u_i=1-\sigma(l_i)\) is high), zero-initialized with bias \(-3.0\) so training starts as an identity mapping.
The Method
3DVLA sits between the frozen 2D vision encoder and the action expert of an existing VLA. The four modules form a pipeline that lifts 2D tokens into 3D, extracts entities, repairs occluded geometry, and feeds spatially-grounded instance tokens to the policy.
The pieces
1. Multi-View Spatial Fusion
Flatten \(\mathcal{X}_{fmap}\in\mathbb{R}^{V\times C\times H\times W}\) into \(N=V\!\cdot\!H\!\cdot\!W\) tokens; add a 2-layer MLP+LayerNorm 3D position embedding derived from unprojected world coordinates; pass through a Transformer encoder to enforce cross-view consistency.
2. Object-Centric 3D Instance Module
Instantiate \(N_q\) state probes in continuous 3D; project each \(\mathbf{p}_j\) into every view with depth-clamped projection; multi-scale deformable sampling; tanh-bounded coord refinement; global Hungarian matching in 3D against pseudo-GT from a frozen VL-SAM detector.
3. Coordinate-Driven SSL Predictor
Asymmetric multi-view masking forces cross-view reasoning. A Hybrid Attention block uses RoPE-3D on Q/K plus Fourier 3D pos-enc + view embedding added to the values. The predictor is kept after pre-training to complete tokens at occluded coordinates.
4. Spatially-Conditioned Geometry Aggregation
End-effector-relative positional encoding gives the action expert offsets in gripper-centric coordinates. An uncertainty-guided gate (\(u_i=1-\sigma(l_i)\)) only mixes 3D geometry into instance tokens when the 2D classifier is unsure, preserving confident features.
Math Walkthrough
Lifting 2D tokens to 3D
Per-view tokens are flattened to \(\mathcal{M}_{flat}\in\mathbb{R}^{N\times C}\) and given an explicit 3D position embedding:
where \(\mathbf{P}_{flat}\) holds the unprojected world coordinates of every token. Continuous 3D RoPE is then applied on attention queries and keys with per-axis phases \(\theta_d=p\cdot 10000^{-2d/D_{axis}}\) for \(p\in\{x,y,z\}\).
Instance probes & view projection
For each probe \((\mathbf{c}_j,\mathbf{p}_j)\) and view \(v\) with intrinsics \(\mathbf{K}_v\) and extrinsics \([\mathbf{R}_v\,|\,\mathbf{t}_v]\):
The depth clamp prevents numerical blow-up near the camera plane. Multi-scale deformable sampling around \(\mathbf{u}_{j,v}\) gives the view-specific feature:
Bounded coordinate refinement
An occlusion-aware gate \(g_{j,v}=\sigma\!\big(\operatorname{MLP}(\mathbf{h}_{j,v})/\tau_{view}\big)\) suppresses bad views. The 3D anchor is updated with a hard physical bound:
The \(\tanh\) keeps per-layer displacement bounded by \(\alpha\), guaranteeing stable convergence.
Global 3D matching & loss
Hybrid Attention for occlusion completion
Context \(\tilde{\mathcal{X}}_{vis}=\mathcal{X}_{vis}+\Phi_{pos}(\mathbf{P}_{vis})+\mathbf{E}_{view}(\mathbf{V}_{vis})\) and target \(\tilde{\mathcal{Z}}_{tgt}=\mathcal{Z}_{tgt}+\Phi_{pos}(\mathbf{P}_{tgt})+\mathbf{E}_{view}(\mathbf{V}_{tgt})\). RoPE-3D is applied only to Q and K:
Uncertainty-guided routing
The gate is zero-initialized with bias \(-3.0\) so \(\mathbf{g}_i\approx\mathbf{0}\) at the start of training, leaving the pretrained 2D features untouched. The final instance representation is \(\hat{\mathbf{c}}_i=\mathbf{c}_i+\mathbf{g}_i\odot\operatorname{LayerNorm}(\mathbf{h}_i)\).
Results
LIBERO-Plus (zero-shot transfer from LIBERO)
Success rate (%) across the seven perturbation dimensions plus average. Bigger is better.
| Method | Camera | Robot | Lang. | Light | Bg. | Noise | Layout | Avg. |
|---|---|---|---|---|---|---|---|---|
| OpenVLA | 0.8 | 3.5 | 23.0 | 8.1 | 34.8 | 15.2 | 28.5 | 15.6 |
| NORA | 2.2 | 37.0 | 65.1 | 45.7 | 58.6 | 12.8 | 62.1 | 39.0 |
| UniVLA | 1.8 | 46.2 | 69.6 | 69.0 | 81.0 | 21.2 | 31.9 | 42.9 |
| ฯโ-Fast | 65.1 | 21.6 | 61.0 | 73.2 | 73.2 | 74.4 | 68.8 | 61.6 |
| RIPT-VLA | 55.2 | 31.2 | 77.6 | 88.4 | 91.6 | 73.5 | 74.2 | 68.4 |
| OpenVLA-OFT | 56.4 | 31.9 | 79.5 | 88.7 | 93.3 | 75.8 | 74.2 | 69.6 |
| ฯโ | 61.0 | 40.8 | 63.5 | 89.3 | 84.1 | 80.1 | 76.4 | 69.4 |
| ฯโ.โ | 71.2 | 77.1 | 89.9 | 94.7 | 94.0 | 84.2 | 84.3 | 84.2 |
| ฯโ.โ + 3DVLA | 75.6 | 77.4 | 88.6 | 97.4 | 97.7 | 85.3 | 86.6 | 86.0 |
RoboTwin 2.0 (physics-rich, contact-heavy)
| Method | Easy | Hard |
|---|---|---|
| DP | 28.0% | 0.6% |
| ACT | 29.7% | 1.7% |
| RDT | 34.5% | 13.7% |
| DP3 | 55.2% | 5.0% |
| ฯโ | 46.4% | 16.3% |
| ฯโ + 3DVLA | 54.5% | 23.2% |
| X-VLA | 70.0% | 39.0% |
| X-VLA + 3DVLA | 72.6% | 42.1% |
Ablation on RoboTwin 2.0 (built on ฯโ)
| Model | Inst. | Pred. | Rout. | Easy | Hard |
|---|---|---|---|---|---|
| ฯโ (baseline) | โ | โ | โ | 46.4% | 16.3% |
| A | โ | โ | โ | 50.1% | 18.9% |
| B | โ | โ | โ | 53.4% | 21.8% |
| C (ours) | โ | โ | โ | 54.5% | 23.2% |
Micro-ablations on LIBERO-Plus: removing the voxel-free spatial fusion drops average from 85.0% โ 84.6%; swapping global 3D Hungarian for 2D matching โ 84.5%; removing end-effector relative PE โ 84.8%. Masking ratio: 25% gives 85.5% (the network short-circuits by interpolating from neighbors); 50% gives 85.7% โ proving asymmetric heavy masking is what forces real cross-view 3D reasoning. 3DVLA adds only ~0.086B parameters.
Intuition & Analogies
Why a separate 3D module beats "just train on more views". A 2D VLA sees several pictures of the same scene but has no enforced rule that says "this red pixel in view A is the same physical point as that red pixel in view B". The Spatial Fusion module is the rule: every token is tagged with an explicit world-coordinate embedding before it ever talks to the others, so cross-view consistency falls out of the attention math instead of having to be discovered from data.
The predictor as an "imagination organ". Masked self-supervised methods like I-JEPA throw away the predictor head once pre-training is done โ it was just a training-time scaffold. 3DVLA's insight is that, once the predictor is conditioned on 3D coordinates, it doubles as a runtime tool for hallucinating the parts of objects you can't see right now. The Camera-perturbation gains (71.2 โ 75.6, and 71.4 โ 73.6 in the ablation) are exactly the slice this is built for.
Uncertainty gating as a humility check. If the 2D detector is confident, the 3D context is a distraction and could corrupt good features. The zero-initialized routing gate (\(\mathbf{g}_i\approx\mathbf{0}\) at start, bias \(-3.0\)) means the 3D branch has to earn the right to write into the instance token โ and it does so only when the 2D evidence is ambiguous.
Limitations & Open Questions
- Depends on calibrated depth and camera matrices. The instance probe projection uses \(\mathbf{K}_v\) and \([\mathbf{R}_v\,|\,\mathbf{t}_v]\) explicitly; deploying on a robot without good calibration or depth would weaken the geometry signal.
- Pseudo-GT from a frozen detector. 3D centroids are derived from VL-SAM v2 masks unprojected with depth. Errors in the open-world detector propagate into the matching cost.
- Simulation benchmarks only. Results are reported on LIBERO-Plus and RoboTwin 2.0 (simulation); the paper does not report real-robot evaluations.
- Modest absolute gains in Language and Robot splits. On LIBERO-Plus the Language split actually dips slightly (89.9 โ 88.6), suggesting the 3D context is not always helpful when the bottleneck is linguistic.
- RoboTwin Hard remains low in absolute terms. Even with 3DVLA, the best ฯโ-based number is 23.2%; the X-VLA-based 42.1% leaves a wide gap to robust manipulation.
Takeaways
- 3D reasoning can be bolted onto a 2D VLA without retraining the VLM. Plug-and-play gains on ฯโ, ฯโ.โ , and X-VLA.
- Enforce multi-view consistency at the token level โ adding a learned 3D position embedding before fusion outperforms decoder-side tricks.
- State probes carrying explicit 3D anchors + global 3D Hungarian matching avoid the cross-view ID conflicts that 2D matching reintroduces.
- Don't throw the masked-SSL predictor away. Keep it and reuse it at inference to fill in occluded geometry โ this is the single biggest win on the Camera-perturbation split.
- Gate 3D context by 2D uncertainty. A zero-initialized routing gate protects the pretrained features and only adds 3D information when the policy actually needs it.
Links & Citation
๐ Original PDF arXiv:2605.29416 โ
Xia, Z., Tang, Y., Wei, B., & Wang, Y. (2026). 3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding. arXiv:2605.29416.