跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-13AI 評分39

DisaggKD:透過教師與學生分置及融合損失實現高效知識蒸餾

DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss

AI 導讀

DisaggKD 將教師與學生模型分置於不同節點,並以融合損失進行知識蒸餾,無須生成完整詞彙表 logits。在 Qwen3.5-35B-A3B 至 9B 配對中,分置配置達到與採用相同損失的共置配置相同 LM loss,且速度更快。較大型配對中,共置配置即使採用融合損失仍因教師權重和學生狀態佔滿記憶體而 OOM,分置配置則可訓練。

正文

Published on Oct 13

Authors:

Abstract

Knowledge distillation (KD) frameworks place teacher and student on the same GPUs, so per-GPU memory limits the teachers that fit. DisaggKD separates them across nodes: teacher replicas stream hidden states to the student, a KD loss over row windows and vocabulary shards equals the full-vocabulary loss in exact arithmetic without forming the full-vocabulary logits, and a prefetch ring paces producer and trainer. Closed-form memory and traffic models locate the memory limits of each layout. On the Qwen3.5-35B-A3B to 9B pair, the disaggregated layout matches the LM loss of the colocated run with the same loss and is faster than that run; our fused loss also lets a colocated run train on as few nodes at faster speed, while the split is memory efficient when the student spans several nodes. On the larger pairs, colocated runs with a fused loss still run OOM where the split trains, because teacher weights and student states fill the budget.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.17644

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.17644 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.17644 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.17644 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co