跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-12AI 評分44

TPPO: Tracking-Value PPO For Language Model Reasoning

TPPO: Tracking-Value PPO For Language Model Reasoning

AI 導讀

TPPO 提出追蹤價值的 PPO,透過累加各位置的標量輸出形成價值 logits,並以重建後的價值接受終端結果監督。相比 PPO,TPPO 在 Qwen3-4B-Base 和 Qwen3-8B-Base 的數學推理綜合評分分別提高 7.0 和 8.4 分,並改善分佈外泛化。程式碼已公開。

正文

Published on Oct 12

·

Submitted by

wang

on Oct 14

Authors:

,

,

,

,

,

,

,

Abstract

Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO (\method), a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. \method improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates. Our code is publicly available in https://github.com/LINs-lab/TPPO.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.16794

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.16794 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.16794 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.16794 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co