ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models
ACCRUE 透過持續離策略強化學習改善機械人基礎模型,讓機械人在學習新任務時保留先前技能。使用 π0.5 + FastTD3,後續真實世界任務約需 4–12 分鐘機械人互動,達到 95% 成功率停止準則;相較單任務強化學習,AutoMate 和 NIST 組裝任務所需互動分別減少 31.0–68.5% 和 38.7–42.9%。
🤖 𝗥𝗲𝗮𝗹-𝗿𝗼𝗯𝗼𝘁 𝗥𝗟. 𝗙𝗲𝘄 𝗱𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻𝘀. 𝗡𝗲𝘄 𝘀𝗸𝗶𝗹𝗹𝘀 𝗶𝗻 𝗺𝗶𝗻𝘂𝘁𝗲𝘀.
Introducing 𝗔𝗖𝗖𝗥𝗨𝗘: Continual Off-Policy Reinforcement Learning for Robot Foundation Models.
We improve robot foundation model behavior with RL and carry that experience across tasks, helping robots 𝗹𝗲𝗮𝗿𝗻 𝗳𝗮𝘀𝘁𝗲𝗿 𝘄𝗵𝗶𝗹𝗲 𝗿𝗲𝘁𝗮𝗶𝗻𝗶𝗻𝗴 𝗲𝗮𝗿𝗹𝗶𝗲𝗿 𝘀𝗸𝗶𝗹𝗹𝘀.
🦾 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗱𝗶𝗿𝗲𝗰𝘁𝗹𝘆 𝗶𝗻 𝘁𝗵𝗲 𝗿𝗲𝗮𝗹 𝘄𝗼𝗿𝗹𝗱
For our real-world policies, all RL interaction comes from the physical robot. Each task starts with 𝗮 𝘀𝗺𝗮𝗹𝗹 𝘀𝗲𝘁 𝗼𝗳 𝗱𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻𝘀 to fine-tune the foundation model.
💻 𝗟𝗼𝘄 𝗮𝗱𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗚𝗣𝗨 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗼𝘃𝗲𝗿𝗵𝗲𝗮𝗱
During RL, the foundation model stays frozen. Only a compact actor–critic is updated to refine its reference actions.
🧠 𝗠𝗮𝗸𝗲 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 𝗰𝗼𝗻𝘁𝗶𝗻𝘂𝗮𝗹
We inherit that actor–critic for the next task and distill earlier behavior on a small fraction of prior-task replay.
⏱️ 𝗡𝗲𝘄 𝘀𝗸𝗶𝗹𝗹𝘀 𝗶𝗻 𝗺𝗶𝗻𝘂𝘁𝗲𝘀 𝗼𝗳 𝗿𝗼𝗯𝗼𝘁 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻
Using π0.5 + FastTD3, subsequent real-world tasks reach the 95% success stopping criterion in about 𝟰–𝟭𝟮 𝗺𝗶𝗻𝘂𝘁𝗲𝘀 𝗼𝗳 𝗿𝗼𝗯𝗼𝘁 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻.
Compared with single-task RL:
• AutoMate assembly: 𝟯𝟭.𝟬–𝟲𝟴.𝟱% 𝗹𝗲𝘀𝘀 𝗿𝗼𝗯𝗼𝘁 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻
• NIST assembly: 𝟯𝟴.𝟳–𝟰𝟮.𝟵% 𝗹𝗲𝘀𝘀 𝗿𝗼𝗯𝗼𝘁 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻
Both methods use the same stopping rule over the latest 20 episodes. These are robot interaction times, including online warmup, excluding demonstration collection and foundation-model fine-tuning.
🛡️ 𝗘𝗮𝗿𝗹𝗶𝗲𝗿 𝘀𝗸𝗶𝗹𝗹𝘀 𝗿𝗲𝗺𝗮𝗶𝗻
After each full sequence, the final policies achieve 𝟭𝟭𝟴/𝟭𝟮𝟬 𝘀𝘂𝗰𝗰𝗲𝘀𝘀𝗲𝘀 across six prior tasks (𝟵𝟴.𝟯% 𝗼𝘃𝗲𝗿𝗮𝗹𝗹). Each task is evaluated separately for 20 trials across the two independently trained suites.
We also evaluate ACCRUE separately in simulation using GR00T + FastSAC.
🎬 Watch the attached 𝟲𝟬-𝘀𝗲𝗰𝗼𝗻𝗱 𝗼𝘃𝗲𝗿𝘃𝗶𝗲𝘄, then explore more videos and interactive results on the project page.
📄 Paper: https://arxiv.org/abs/2610.18596
🌐 Project: https://accrue-rl.github.io/
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co