Behavior Cloning of MPC for 3-DOF Robotic Manipulators
TL;DR
The authors train neural networks to imitate an Inverse-Kinematics + MPC expert driving a 3-DOF arm in MuJoCo. A 4-layer MLP [256,128,64,32] matches the expert in closed loop — 84.98% success within 5 cm and a 2.9 cm mean final error — while running at 1.10 ms / step, roughly a 3× inference-latency reduction over MPC, with 25% lower CPU. A surprise: static MLPs beat sliding-window and GRU variants, confirming the (q, q̇) state is effectively Markovian for this task.
Problem & Motivation
Model Predictive Control (MPC) is the workhorse of robotic manipulation because it gives strong stability and constraint handling. The price: it solves a fresh constrained optimization at every control tick. On a small CPU — or just at higher control rates — that quadratic program (QP) starts dominating the loop and its solve time fluctuates with the optimization landscape, which is exactly what you don't want under hard real-time constraints.
Neural networks can amortize that optimization: train once offline so that, at deployment, a single forward pass returns a near-optimal action. The paper studies this for a 3-degree-of-freedom robotic manipulator in MuJoCo, asking two practical questions:
- Which architecture family — classical regressors, deep MLPs, sliding-window MLPs, GRU RNNs — actually clones the MPC well enough to close the loop?
- How much speed and resource budget do you really save, and at what cost in precision?
Key Concepts
- Model Predictive Control (MPC)
- A controller that, at each step, predicts the system's future over a horizon \(N\) and solves a constrained optimization for the action sequence that minimizes a quadratic cost. Only the first action is executed, then the problem is re-solved.
- Behavior Cloning (BC)
- The simplest form of imitation learning: collect (state, expert-action) pairs and fit a supervised regressor. No environment interaction during training — and no reward function.
- Inverse Kinematics (IK)
- The map from a Cartesian target \(p_{\text{des}}\) to joint angles \(q_{\text{des}}\) that put the end-effector there. Solved here by damped least-squares Jacobian iteration.
- Damped Least Squares (DLS)
- A regularized pseudo-inverse, \(J^\top(JJ^\top+\lambda^2 I)^{-1}\), that stays numerically stable near singularities by paying a small bias.
- Markov property (in this paper)
- The hypothesis — borne out empirically — that the optimal torque depends only on the current state \((q,\dot q)\) and target, not on history. If true, recurrence is unnecessary.
- Direction Accuracy (DA)
- A custom offline metric: the fraction of joint-torque components whose sign the network gets right. A coarse "is it pushing the right way?" check.
The Method
The pipeline has three stages: an expert that combines IK and MPC, a data generation loop in MuJoCo that records its (state, target, action) triples, and a family of neural surrogates trained to match the expert's torques.
The expert: IK + MPC
The expert is hierarchical. IK turns the sampled Cartesian target \(p_{\text{des}}\in\mathbb{R}^3\) into joint references \(q_{\text{des}}\) via damped least-squares with a step clamp (max norm \(\alpha\in[0,1]\)) and joint-angle wrapping \(q_i\leftarrow\operatorname{atan2}(\sin q_i,\cos q_i)\). MPC then computes torques \(\tau_{\text{MPC}}\) over a finite horizon \(N\) using a simplified double-integrator model (MuJoCo absorbs Coriolis/gravity via qfrc_bias). The QP is solved with CasADi + OSQP.
Data generation
For each episode the pipeline (1) samples a reachable target in cylindrical coordinates, (2) runs IK then MPC to get torques, (3) steps MuJoCo with \(\tau = \tau_{\text{MPC}} + \tau_{\text{env}}\), (4) logs \((q,\dot q,p_{\text{des}},\tau_{\text{MPC}})\) per step. Failed IK/MPC steps are dropped to keep the dataset clean. Stored as HDF5 grouped by episode; the MPCDataset wrapper exposes a 9-D feature vector \([q;\dot q;p_{\text{des}}]\) and a 3-D torque label.
Three input formats
Flat
Treat every timestep as i.i.d. — \(X\in\mathbb{R}^{N\times 9},\ Y\in\mathbb{R}^{N\times 3}\). Used by sklearn baselines and the custom MLP.
Sequential
Whole episodes as sequences — \(X\in\mathbb{R}^{E\times T\times 9}\). Used by the GRU.
Sliding window (W=5)
Concatenate current state with the past 5 states (target appended once) — short-term memory injected into an MLP.
Noise injection
Small Gaussian noise on both inputs (joint state) and outputs (torques) to mimic sensor/actuator disturbances.
Student architectures
Two families compared. The MLP family scales depth and width; the GRU family scales hidden dimension and number of recurrent layers:
| Name | Type | Hyperparameters |
|---|---|---|
| MLP_Small | MLP | hidden = [64, 32] |
| MLP_Medium | MLP | hidden = [128, 64] |
| MLP_Deep | MLP | hidden = [256, 128, 64, 32] |
| MLP_Deep_Scaled | MLP | hidden = [512, 256, 128, 64] |
| GRU_Shallow | GRU | hidden = 64, layers = 1 |
| GRU_Medium | GRU | hidden = 128, layers = 2 |
| GRU_Deep | GRU | hidden = 128, layers = 4 |
| GRU_Wide | GRU | hidden = 256, layers = 2 |
Sliding-window MLPs reuse the same four MLP shapes with \(W=5\).
Math Walkthrough
1. IK with damped least squares
Let \(p(q):\mathbb{R}^3\to\mathbb{R}^3\) be forward kinematics and \(J(q)=\partial p/\partial q\in\mathbb{R}^{3\times 3}\) the Jacobian. With Cartesian error \(e=p_{\text{des}}-p(q)\):
The \(\lambda^{2} I\) term is what makes it "damped" — without it, near a singularity \((JJ^\top)^{-1}\) explodes. Step size is clamped: \(\Delta q\leftarrow \Delta q\cdot \alpha/\lVert\Delta q\rVert\) if \(\lVert\Delta q\rVert>\alpha\).
2. MPC as a finite-horizon QP
With state \(x=[q,\dot q]^\top\in\mathbb{R}^6\) and the simplified double-integrator \(\ddot q=\tau_{\text{MPC}}\), discrete-time dynamics are \(x_{k+1}=x_k+\Delta t\,[\dot q_k;\tau_k]^\top = f(x_k,\tau_k)\). MPC solves:
Here \(x_{\text{ref}}=[q_{\text{des}}^\top,\,0^\top]^\top\) (target with zero velocity), and \(Q,R,Q_N\succ 0\). Only \(\tau_0\) is executed; the QP is re-solved next tick.
3. Behavior-cloning objective
With expert torques \(\tau_{\text{MPC}}\) and student \(\pi_\theta\), the supervised loss is just regression to the expert:
The paper compares MSE vs MAE losses head-to-head and adopts MSE because models trained with it had lower test MSE and lower run-to-run variance (Table II below).
4. Offline metrics
Beyond MAE and RMSE, two metrics matter:
5. Online success
An episode is a success if the final end-effector is within tolerance \(\varepsilon\) of the target: \(\lVert p_{\text{final}}-p_{\text{des}}\rVert_2<\varepsilon\). Three tiers are reported: strict \(\varepsilon=0.02\)m, moderate \(\varepsilon=0.03\)m, relaxed \(\varepsilon=0.05\)m.
Results
Regression baselines (offline, 35k samples, 80/20 split, 5 runs)
| Model | MSE | MAE | Expl. Var. | Dir. Acc. |
|---|---|---|---|---|
| Ridge | 8.729 ± 0.098 | 1.418 ± 0.009 | 0.221 | 0.681 |
| Random Forest | 0.097 ± 0.003 | 0.111 ± 0.002 | 0.991 | 0.999 |
| MLP Regressor | 0.053 ± 0.013 | 0.094 ± 0.009 | 0.994 | 0.938 |
| Gradient Boosting | 1.017 ± 0.059 | 0.243 ± 0.005 | 0.843 | 0.996 |
| KNN Regressor | 0.237 ± 0.038 | 0.091 ± 0.002 | 0.976 | 0.9987 |
Linear regression (Ridge) collapses — the inverse-dynamics map is highly non-linear. KNN wins MAE and direction accuracy but is hurt by outliers (higher MSE); the shallow MLP Regressor wins overall MSE.
Loss-function comparison (PyTorch MLP & GRU)
| Model · Loss | Test MSE | Test MAE |
|---|---|---|
| MLP · MSE | 0.1460 ± 0.0522 | 0.1474 ± 0.0126 |
| MLP · MAE | 0.2017 ± 0.0455 | 0.0892 ± 0.0078 |
| GRU · MSE | 1.1442 ± 0.1441 | 0.2581 ± 0.0138 |
| GRU · MAE | 2.1821 ± 0.1001 | 0.3570 ± 0.0101 |
MSE-trained models had lower test error and tighter variance. MSE adopted as the default for everything after this.
Architecture sweep (offline test MSE, 5 runs)
The paper's Figure 6 reports the following trends from their architecture sweep:
- MLP_Deep achieved the lowest mean test MSE overall (≈ 0.05), outperforming MLP_Small, MLP_Medium, and even MLP_Deep_Scaled.
- Depth helped, width didn't. Going wider (512→256→…) gave no significant gain over MLP_Deep.
- Sliding window (W=5) hurt slightly on medium and deep MLPs — the redundant history introduced noise rather than signal.
- GRUs lagged. GRU_Shallow was worst overall (MSE ≈ 2.7); GRU_Deep and GRU_Wide improved but never matched the MLPs.
Closed-loop success (1000 episodes, 150 steps max)
| Tolerance ε | Sklearn baselines | MLP_Deep | MPC Expert |
|---|---|---|---|
| 0.05 m (relaxed) | < 35% | 84.98% | ≈ MPC ref. |
| 0.02 m (strict) | — | 64.16% | — |
Mean final tracking error for MLP_Deep was 2.9 cm — most failures are near-misses just outside ε.
Computational efficiency
| Quantity | MPC Expert | MLP_Deep |
|---|---|---|
| Mean inference latency / step | ~3.3 ms (≈ 3× of student) | 1.102 ± 0.614 ms |
| Latency variance | high (QP-dependent) | tight, deterministic |
| CPU utilization | baseline | −25% |
The headline number — "3× reduction in inference latency" — comes from this latency comparison. The determinism is arguably as important: hard real-time loops care about worst-case, and the QP's tail latency is what bites you.
Intuition & Analogies
The interpreter vs. the phrasebook. MPC is a simultaneous interpreter — given any input, it computes the right response from first principles, every time, but at a high per-utterance cost. Behavior cloning is a phrasebook compiled from watching the interpreter for hours: nearly instantaneous lookup, fluent for the situations it saw, but with no machinery to reason when the exact final centimeter isn't in the book.
Why static beats temporal here. The expert is itself a function of \((q,\dot q,p_{\text{des}})\) — the MPC's input is the current state, plus a quadratic cost that has no history. Asking a GRU to learn this is asking it to ignore its own memory: extra parameters, harder optimization, no information gain. The Markov property of the expert is inherited by the optimal student.
The 2.9-cm wall. The student is rewarded for matching the expert's torques, not for closing the position loop. Tiny torque errors near the goal — where the expert is taking sub-millimeter corrective bites — wash out in the MSE but are exactly what determines the steady-state error. The fix is either an active correction signal (DAgger) or a hybrid that hands off to a precision controller near the target.
Limitations & Open Questions
- The precision gap. 64.16% at 2 cm vs 84.98% at 5 cm — BC captures the global trajectory but cannot eliminate terminal steady-state error without feedback.
- Open-loop training, closed-loop deployment. Vanilla BC suffers from distribution shift. The authors propose DAgger as the obvious next step.
- 3-DOF and a simplified dynamics model. MuJoCo handles Coriolis/gravity for the MPC, so the expert sees a clean double-integrator. Higher-DOF arms and stiffer non-linearities are explicitly left to future work.
- Linear MPC only. Cloning a non-linear MPC (and whether one architecture suffices) is flagged as future work.
- Architecture coverage. Transformers and Legendre Memory Units are named as untried but plausible candidates, especially as DOF grows.
- Sim only. No hardware deployment yet; sensor and actuator noise are emulated by Gaussian perturbations during training.
Takeaways
- BC of MPC works for 3-DOF arms. A small 4-layer MLP closes the loop at 84.98% success within 5 cm, with a 2.9 cm mean final error.
- ~3× faster, 25% less CPU than the MPC expert — and crucially, deterministic latency suitable for hard real-time control.
- Static beats temporal on this task: sliding windows and GRUs underperformed the static MLP. The Markov assumption holds.
- Depth > width for this regression.
[256,128,64,32]beat the wider[512,256,128,64]. - MSE > MAE as the BC loss — better test MSE and tighter variance, even though MAE wins on its own metric.
- The path forward is feedback. DAgger, hybrid precision controllers, and non-linear / higher-DOF MPCs are the obvious next experiments.
Links & Citation
Guegan, T., & Teo, W. J. D. (2026). Behavior Cloning of MPC for 3-DOF Robotic Manipulators. ICRA 2026 Workshop on Reinforcement Learning for Imitation Learning (RL4IL).