AffordVLA: Injecting Affordance Representations into VLAs via Implicit Feature Alignment

TL;DR

Problem & Motivation

Vision-Language-Action (VLA) models such as RT-2, OpenVLA, π0, and π0.5 map an RGB image plus a language instruction directly to actions. Their VLM backbones, however, are pre-trained on global semantics and object appearance. The paper analyses VLA attention maps and finds that visual attention spreads across the whole object and even the background, so even when the robot identifies the right object it can grab the wrong part — the body of a skillet instead of the handle.

Prior fixes inject affordance information explicitly, concatenating masks or keypoints produced by an external detector to the policy input. This works but inherits three structural problems:

• Annotation hunger. Open-vocabulary affordance datasets only have tens of thousands of samples.
• Cascading errors. The policy is bottlenecked by the detector at inference time.
• Latency. Running an extra perception module per step blows the real-time budget — the paper measures π0.5 + explicit affordance input at 4.8 Hz / 206.9 ms, ~2.6× slower than bare π0.5.

The driving question: can the affordance prior be internalised into the VLA's own features during training, so inference looks exactly like a vanilla VLA but the features quietly point at the right region?

Key Concepts

Affordance

The action possibilities an object offers an actor (Gibson). In robotics this is operationalised as task-conditioned functional regions on the object: the same knife affords grasping at the handle and cutting at the blade.

Vision-Language-Action (VLA) model

A policy that consumes an RGB observation + language instruction (via a pretrained VLM) and emits robot actions end-to-end. AffordVLA builds on π0.5 (PaliGemma-3B backbone + flow-matching action expert).

Zero-shot affordance teacher

A 0.8B-parameter pipeline that, given an image and instruction, parses the instruction into part-level concept prompts ("hammer handle") with Qwen3-VL, then runs an open-vocabulary affordance head built on SAM 3's perception encoder to produce a task-conditioned visual representation \(Z_t^{\mathrm{aff}}\) and a pixel-level mask \(M_t^{\mathrm{aff}}\). No affordance annotations are used.

Implicit feature alignment

Instead of feeding the affordance mask into the policy, train the VLA's intermediate visual tokens to match the teacher's affordance tokens via a cosine-similarity loss. The teacher is frozen during training and removed at inference.

Explicit vs implicit injection

Explicit = concatenate the affordance output to the VLM input (incurs runtime overhead, depends on detector at deployment). Implicit = use it only as a supervisory signal on intermediate features.

Conditional flow matching action expert

The π0-style action head: an 18-layer Transformer trained to predict the vector field of a noisy action chunk \(A_t=[a_t,\dots,a_{t+H-1}]\) conditioned on the VLM context (H=30, 10 denoising steps at inference).

The Method

AffordVLA has three components — affordance teacher, understanding expert, action expert — but only the latter two are present at inference.

The three components

Why align the intermediate representation, not the mask?

The paper deliberately distils \(Z_t^{\mathrm{aff}}\) (the teacher's last fusion-decoder features) rather than the final binary mask \(M_t^{\mathrm{aff}}\). The intermediate feature still carries task semantics and local spatial structure jointly, which is a far smoother target than a thresholded mask. The mask is only used to validate the teacher and regularise it.

Why layer 12 (of 18)?

Shallow layers preserve texture but their alignment signal is washed out by later cross-modal layers. Very deep layers converge to a modality-agnostic semantic space and lose the spatial structure affordances need. The middle-deep sweet spot (layer 12) balances both.

Math Walkthrough

1. Understanding expert context

Vision, language, and state are tokenized and jointly modelled by the VLM:

2. Action loss (flow matching)

The action expert predicts the velocity field \(v_\theta(A_t^\tau, C_t)\) of a noisy chunk \(A_t^\tau\); the target is \(A_t-\epsilon\) with \(\epsilon\sim\mathcal{N}(0,I)\):

3. Affordance teacher output

The task parser maps observation + instruction into a concept prompt \(p_t=R(I_t,l_t)\), and the open-vocabulary affordance module returns both an intermediate representation and a pixel mask:

A learned sinusoidal positional embedding is added so the VLA can match spatial order: \(\tilde Z_t^{\mathrm{aff}} = Z_t^{\mathrm{aff}} + P\).

4. Bringing the VLA features into the teacher space

Let \(X_t^{V,(m)}\in\mathbb{R}^{N_v\times d_v}\) be the visual tokens at layer \(m=12\), \(d_v=2048\). A small MLP with bilinear resizing maps them into the teacher's space:

5. The cosine alignment loss

Token-wise cosine similarity, averaged over the \(N\) spatial positions:

6. Total objective

The teacher gradients are blocked — only the alignment MLP, the visual encoder, and the rest of the VLA are updated. At inference, the alignment MLP and teacher are discarded entirely.

Results

Zero-shot affordance teacher on AGD20K (Unseen split)

Despite never training on AGD20K, the teacher beats every weakly- and zero-shot baseline and the best fully supervised method on two of three metrics:

KLD drops from UAD's 1.878 to 0.905 and NSS rises from 1.092 to 1.906, confirming the teacher is high quality enough to serve as a supervisory signal.

RoboTwin2.0 simulation (50 Easy demos per task, evaluated under Easy and Hard)

+14.8 pp over the strongest Easy baseline (DP3) and +18.6 pp over the strongest Hard baseline (RDT). Per-task highlights under Hard: Place Bread Skillet 28.0% (next-best 4.0%), Place Cans Plasticbox 26.0% (next-best 5.0%).

Real-world inference efficiency

Explicit affordance injection costs 2.6× latency; implicit alignment costs essentially nothing. AffordVLA also wins the eight real-world tasks (pour water, hang mug, cut banana, hammer block, sweep, wipe, place marker, sort cluttered scene) over π₀, π₀.₅, and π₀.₅+explicit affordance.

Ablations

• Alignment on/off (RoboTwin2.0): removing the alignment loss drops success rate on every task, with larger drops under Hard — confirming the loss is the source of robustness, not just larger compute.
• Attention visualisation: with alignment, the 12th-layer attention concentrates on tool handles, container openings, and contact regions instead of background and full objects.
• Training efficiency: to reach an average 45% success rate, AffordVLA needs about 5.2k fewer iterations than the no-alignment baseline.
• t-SNE: aligned VLA tokens move toward the teacher's distribution but retain their own cluster centres, evidence that affordance priors are added without erasing the VLA's semantic representation.

Intuition & Analogies

Limitations & Open Questions

• Teacher noise leaks in. In cluttered or unusual scenes, an inaccurate teacher representation becomes a noisy supervisory signal — the policy can only be as well-targeted as its teacher.
• Visual-only intervention. Alignment touches the VLM features; it does not directly shape action generation. Tasks dominated by fine contact dynamics or continuous force modulation may see smaller gains.
• Generalisation breadth untested. Real-world experiments use a single UR5 platform and 50 demos per task; long-horizon, multi-subgoal, and cross-embodiment evaluation are explicitly flagged as future work.
• Mid-depth alignment is a heuristic. Layer 12 of 18 works for this PaliGemma stack, but the optimal alignment depth almost certainly depends on architecture.
• Teacher cost during training. The teacher is 0.8B parameters running on 1008×1008 inputs — training is non-trivial even though inference is free.

Takeaways

• Affordance can be a training-time prior, not an inference-time input. A cosine-similarity loss between the VLA's 12th-layer visual tokens and a frozen teacher's task-conditioned features is enough to reshape attention onto functional regions.
• Teacher quality matters; mask quality does not. Distilling the intermediate representation \(Z_t^{\mathrm{aff}}\) beats distilling the final mask \(M_t^{\mathrm{aff}}\) because it preserves both semantics and spatial structure.
• Implicit beats explicit on both axes. Higher success rate and ~2.6× lower latency than explicit affordance concatenation.
• +30 pp Easy, +22 pp Hard average success over π₀ on RoboTwin2.0, with zero runtime overhead vs π₀.₅.
• Training is also faster: ~5.2k fewer steps to a 45% success rate — affordance supervision is a cheap, dense learning signal.

Links & Citation

📄 Original PDF arXiv:2605.17517 ↗