跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 16 天前AI 評分37

onóna:採用共享嵌入與解嵌入的雙嵌入方法

onóna: Twin Embeddings with Tied Unembeddings

AI 導讀

onóna 為 Transformer 語言模型提出雙表參數化方法,在保留輸入與輸出權重共享的同時提升輸出表達能力。測試涵蓋多種架構及 142M、330M 和 1B 規模;在 12 組比較中,11 組驗證損失與外部困惑度較低,10 組下游任務平均準確率較高。代價是增加一張嵌入表及一次詞彙投影。

正文

Published on Nov 7

Authors:

,

,

,

,

,

Abstract

A two-table weight-sharing method for Transformer language models improves output expressiveness and reduces validation loss and perplexity across several architectures and scales at the cost of an extra embedding table and vocabulary projection.

Weight tying reduces the parameter cost of language models, but the usual tied output head is limited to a single softmax. We introduce onóna, a two-table parameterization that increases output expressiveness while preserving input-output weight sharing. The Transformer takes the average of two embedding tables as input, and the output head combines the softmax distributions defined by those same tables. A gate conditioned only on the current input token determines the mixture weight. Its values can be precomputed for inference, and the Transformer body remains unchanged. We evaluate onóna across Dense, Gated DeltaNet hybrid, mixture-of-experts, and Attention Residual architectures at 142M, 330M, and 1B scales. Under matched token budgets, onóna lowers final validation loss and external perplexity in 11 of 12 comparisons and raises the average accuracy over eight downstream tasks in 10 of 12. The one exception on loss is the 1B MoE model. The cost is one extra embedding table and one extra vocabulary projection.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2611.10556

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.10556 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.10556 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.10556 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co