REMA project icon: a luminous lavender jellyfish, evoking trajectories drifting around a manifold.
IEEE Robotics and Automation Letters (RA-L), 2026

REMA: Efficient Online Robotic Reinforcement Learning
with Explicit Manifold Alignment

A plug-in manifold-based credit assignment framework that improves sparse-reward robotic RL by preserving informative failures and suppressing uninformative divergent rollouts.

Wenkai Guo1, Peizhi Tang1, Yukun Bai1, Ziming Yan1, Haoyuan Deng1, Guanxing Lu2, Shijie Li3, Ziwei Wang1†

1 School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
2 Shenzhen International Graduate School, Tsinghua University, China
3 Institute for Infocomm Research, A*STAR, Singapore
† Corresponding author

Paper (IEEE Xplore) Code coming soon Video BibTeX
REMA teaser: failed trajectories are evaluated by their distance to a reference manifold built from successful trajectories.
REMA penalizes failures distant from the reference manifold derived from successful trajectories, refining credit assignment for value learning.
1.2× Faster convergence in simulation
+7.4% Higher success rate in simulation
60.8% → 29.9% Human intervention reduced in real-world HIL RL
62.5% → 80.6% Autonomous success improved in real-world HIL RL

Abstract

Online reinforcement learning enables robots to improve through real-world interaction, but contact-rich manipulation remains difficult under sparse binary rewards. Conventional reward shaping often depends on goal-proximity heuristics and does not explicitly distinguish informative near-success failures from uninformative divergent trials. REMA addresses this credit assignment problem by aligning online trajectories with a spectral reference manifold constructed from successful demonstrations. Low-frequency DCT components capture task-relevant motion structure, allowing REMA to identify whether a failed trajectory is informative or off-manifold. Distant failures receive calibrated penalties that suppress noisy value-learning signals, while aligned near-misses remain useful for policy improvement. Across RoboMimic simulation tasks and real-world human-in-the-loop manipulation on Galaxea-A1 and Franka Research 3, REMA improves sample efficiency, reduces human intervention, and increases autonomous success.

Sparse rewards make all failures look equally bad

In contact-rich manipulation, binary rewards are easy to define but poor at assigning credit. A robot that almost grasps the target object and a robot that wanders in free space both receive zero reward. The first trajectory contains useful geometric information about the task, while the second mainly injects noise into value learning. REMA starts from this distinction: the quality of a failed trajectory should be judged relative to successful behavior, not only by its final reward.

Informative near-miss

Follows task-relevant geometry; useful for value learning.

Divergent failure

Leaves the successful-behavior manifold; should be suppressed.

Explicit manifold alignment for discriminative credit assignment

The core mechanism is deliberately simple. Successful trajectories define a reference manifold in end-effector task space. Each online trajectory is aligned and compressed into a spectral representation. Its normalized distance to the reference manifold determines whether the trajectory should be preserved as informative experience or penalized as an off-manifold deviation.

01

Build a reference manifold

REMA starts from a small set of successful demonstrations. Each trajectory is represented in end-effector task space, including position, rotation, and gripper state. The successful trajectories define the reference behavior structure for the task.

02

Compress motion structure spectrally

After geometric alignment, REMA applies the Discrete Cosine Transform to extract low-frequency motion components. These components summarize global task-relevant motion while filtering high-frequency execution noise.

03

Calibrate online trajectories

During online learning, each failed trajectory is compared with the reference manifold using normalized spectral distance. Off-manifold failures are penalized to reduce their impact on value estimation, while aligned failures are preserved as informative experience.

REMA pipeline: spectral manifold construction from successful trajectories, distance evaluation of online failures, and reward calibration based on alignment.
REMA constructs a spectral reference manifold from successful trajectories, evaluates online rollouts against it, and calibrates rewards based on manifold alignment.

Why REMA is plug-in

No auxiliary networks. No new policy architecture. No change to the underlying RL algorithm.

REMA acts after rollouts are collected. It calibrates the reward signal used for value learning, making it compatible with standard value-based online RL and human-in-the-loop RL pipelines.

More stable online learning with less human effort

Across RoboMimic Can and Square, REMA improves learning speed and final success compared with standard imitation and value-based RL baselines. On real robots, REMA reduces human intervention from 60.8% to 29.9% and improves autonomous success from 62.5% to 80.6% in human-in-the-loop RL, while using the same intervention equipment and offline data as the compared setup.

1.2×Simulation convergence
+7.4%Simulation success
60.8% → 29.9%Human effort
62.5% → 80.6%Real-world success

Simulation results

In simulation, REMA is evaluated on RoboMimic Can and Square. The experiments compare against behavior cloning, IBRL, RLPD, and SASR under matched seeds and hyperparameters. REMA improves early-stage learning and final convergence, showing stronger sample efficiency and lower variance across runs.

Learning curves on RoboMimic Can and Square: REMA outperforms BC, IBRL, RLPD, and SASR.
On RoboMimic Can and Square, REMA improves sample efficiency and achieves stronger final convergence compared with BC, IBRL, RLPD, and SASR.

Real-world human-in-the-loop training

In real-world human-in-the-loop RL, REMA is evaluated on Galaxea-A1 and Franka Research 3 across four tasks covering precision, contact sensitivity, distractors, and deformable-object manipulation. Both platforms use side-fixed and wrist-mounted cameras, and human interventions are provided using a 6D SpaceMouse. REMA uses the same offline data and intervention equipment as the compared HIL-SERL setup, differing only by applying manifold-based reward calibration to collected trajectories.

Place Toast

Pick toast from the toaster and place it on a plate. Galaxea-A1

Place Teacup

Grasp a target teacup among distractors and place it on a plate. Galaxea-A1

Insert Pipe

Perform tight-tolerance contact insertion. Franka Research 3

Fold Towel

Manipulate a deformable towel. Franka Research 3

Real-world human-in-the-loop training results on Galaxea-A1 and Franka Research 3.
Real-world human-in-the-loop training on Galaxea-A1 and Franka Research 3. REMA improves autonomous success while reducing human intervention under the same wall-clock budget.

Offline-to-online fine-tuning

Beyond learning from scratch, REMA also calibrates offline-to-online fine-tuning. Under a matched 30-minute online budget, REMA consistently improves over WSRL across the evaluated real-world tasks, with especially clear gains on the deformable Fold Towel task.

Real-world policy fine-tuning results (successes out of 20 trials)
TaskPretrained (Cal-QL)WSRL @ 30 minREMA @ 30 min
Place Teacup9/2013/2016/20
Insert Pipe11/2017(±2)/2019(±1)/20
Fold Towel6/2011(±3)/2014(±2)/20

Place Teacup on Galaxea-A1 is a single online fine-tuning run; Insert Pipe and Fold Towel on Franka Research 3 report mean ± std over three independent runs, each evaluated with 20 autonomous trials.

Videos

►

REMA overview video

Method summary and real-world human-in-the-loop training results — coming soon

►

Place Toast on Galaxea-A1

Task video coming soon

►

Place Teacup with distractors on Galaxea-A1

Task video coming soon

►

Insert Pipe on Franka Research 3

Task video coming soon

►

Fold Towel on Franka Research 3

Task video coming soon

Understanding the calibration behavior

Calibration dynamics

As online training progresses, the manifold threshold adapts with successful experience and the calibration rate declines. This shows that REMA is self-regulating: it strongly suppresses random early exploration, then relaxes as the policy becomes aligned with the successful-behavior manifold.

Manifold threshold and calibration rate during training on Can and Square; the calibration rate declines as learning progresses.
Learning dynamics on RoboMimic: the threshold adapts with online successes and the declining calibration rate shows the penalty is self-regulating.

Value landscape

REMA produces sharper value boundaries over task-relevant regions, whereas the baseline value landscape is more diffuse. This suggests that manifold-calibrated penalties help value learning become more sensitive to geometric task structure.

Normalized Q-value landscape over end-effector positions: REMA shows sharp value boundaries, IBRL lacks spatial differentiation.
Normalized Q-value landscape over end-effector positions at 20k steps on Can: REMA exhibits sharp value boundaries while the baseline lacks spatial differentiation.

Robustness and boundary cases

Ablation

Ablations show that a small penalty magnitude is important for preserving the optimal policy while suppressing uninformative failures. DCT-based spectral distance performs better than temporal alignment with DTW in the evaluated setting.

Ablation studies: penalty magnitude and distance metric comparison on RoboMimic.
Ablation studies on RoboMimic. Left: effect of penalty magnitude. Right: DCT spectral distance outperforms temporal alignment with DTW.

Failure mode

The spectral distance captures global motion shape, which makes it robust to execution-speed variability. However, it can underweight spatial outcome differences or localized contact irregularities. This motivates future extensions with multi-resolution or learned manifolds.

Illustrative misclassifications of the spectral distance on the Fold Towel task.
Illustrative boundary cases on Fold Towel: a spatially distant rollout judged on-manifold, and a near-success rollout judged off-manifold due to local hesitation.

Limitations

REMA captures global motion structure through low-frequency spectral features. This makes it robust to execution-speed variation but less sensitive to localized contact irregularities or qualitatively different success modes. Future extensions may use multi-resolution features, clustered manifolds, or learned trajectory representations when longer online-adaptation budgets justify additional representation learning.

Citation

@article{guo2026rema,
  title={Efficient Online Robotic Reinforcement Learning with Explicit Manifold Alignment},
  author={Guo, Wenkai and Tang, Peizhi and Bai, Yukun and Yan, Ziming and Deng, Haoyuan and Lu, Guanxing and Li, Shijie and Wang, Ziwei},
  journal={IEEE Robotics and Automation Letters},
  year={2026},
  url={https://ieeexplore.ieee.org/abstract/document/11668937}
}