跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 16 天前AI 評分35

Horizon-Scalable Q-Learning:自動平衡單步與多步回報的離線價值學習方法

Horizon-Scalable Q-Learning

AI 導讀

Horizon-Scalable Q-Learning(HQL)提出一種方法,透過自動平衡單步與多步回報,提升長時程離線價值學習的準確度。只有當評論家對數據集動作的評分不低於行為克隆策略所抽取 N 個動作中的最高評分時,HQL 才延伸目標;否則退回單步目標,且無需動作密度。

正文

Published on Nov 7

Authors:

,

,

,

Abstract

Horizon-Scalable Q-Learning automatically balances one-step and multi-step returns by extending targets only when the critic values a dataset action at least as highly as the best-of-N actions from a behavior-cloning policy, improving long-horizon off-policy value learning.

Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small. Multi-step TD shortens this chain by following the dataset trajectory for several steps; however, since the behavior policy is often suboptimal, every followed action adds a bias, and no fixed number of steps suits a whole trajectory. In this paper, we propose Horizon-Scalable Q-Learning (HQL), a simple method that enables accurate long-horizon value learning by automatically balancing between one-step and multi-step returns. HQL extends a target through a dataset action only when the critic values it at least as highly as the best-of-N actions sampled from the BC policy, and otherwise falls back to a one-step target. We show that this rule is a principled off-policy correction that needs no action densities, which makes HQL compatible with expressive flow and diffusion policies. Experiments show that HQL avoids the failures of both one-step and multi-step TD on a combination-lock task across horizons, and outperforms prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.

View arXiv page View PDF Project page GitHub 2 Add to collection

Get this paper in your agent:

hf papers read 2611.10987

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.10987 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.10987 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.10987 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co