跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 27 天前AI 評分58

Metis-1.6 技術報告介紹兩款讓每個 token 經同一組 20 層兩次的推理模型

Think Twice: The Metis-1.6 Technical Report

AI 導讀

Metis-1.6 技術報告介紹作者從零訓練的兩款推理模型 Logos 和 Praxis,每個 token 都以相同權重經過同一組 20 層兩次。兩者主要採用 Mamba-2,只有 3 層使用 attention,且沒有位置編碼;Logos 有 10.23B 參數、每個 token 使用 2.36B,Praxis 有 2.94B 參數,4-bit 下可放入 2 GB。

正文

Author here. Metis-1.6 is two reasoning models I trained from scratch. The idea is in the title: every token runs through the same 20 layers twice, with the same weights. Logos has 10.23B parameters and uses 2.36B per token. Praxis has 2.94B and fits in 2 GB at 4-bit. Both are mostly Mamba-2, with attention in only 3 layers and no positional encoding.

On 1.02T training tokens, Logos ranks 2nd of 30 open models in math (67.5% on AIME 2025, 63.6% on HMMT February 2026) and sits on the Pareto frontier for both training data and compute. It trails in code, instruction following, long context and factual recall, and the paper reports those too.

Two results I didn't expect: the models learned to close the gates on a 1.77B-parameter hashed n-gram memory, so I removed it. And RL on Logos cost breadth, so 1.6 stops at SFT.

Weights are Apache 2.0, with MLX, FP8, NVFP4 and GGUF builds. Dromos is the inference engine I wrote for them.

Happy to answer questions.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co