跳到正文
螞蟻 inclusionAI:HuggingFace 新模型·· 29 天前AI 評分61

inclusionAI 發佈 Ling-3.1-flash-base 基礎模型檢查點

inclusionAI/Ling-3.1-flash-base

AI 導讀

inclusionAI 發佈 Ling-3.1-flash-base 基礎模型檢查點,總參數量為 560B,每個 token 啟用 25B 參數。它採用與 Ling-3.1-flash 相同的混合線性 MoE 架構,結合 KDA 和 Gated MLA,並在 47 項基準、八個領域進行評測。檢查點以 BF16 權重發佈,模型以 FP8 訓練,適用於持續預訓練、領域適配及後訓練研究。

正文

inclusionAI

🤗 Hugging Face   |   🤖 ModelScope

Introduction

We introduce Ling-3.1-flash-base, the base checkpoint of Ling-3.1-flash, released to support research and community-driven innovation. With 560B total parameters and 25B active parameters per token, the model provides a foundation for continued pretraining, domain adaptation, and post-training research.

Ling-3.1-flash-base shares the same hybrid-linear Mixture-of-Experts (MoE) architecture as Ling-3.1-flash, combining KDA with Gated MLA. As a base checkpoint, it is intended for further training and adaptation.

The model was trained using FP8. This release provides the BF16 base checkpoint produced after Warmup-Stable and Merge (WSM) checkpoint merging. BF16 refers to the precision of the released weights, while FP8 refers to the precision used during training.

Model Overview

Key Features

  • Sparse MoE architecture: 512 routed experts, with 8 routed experts and 1 shared expert activated per token, for a total of 25B active parameters per token.
  • Hybrid linear attention: 63 KDA layers and 9 Gated MLA layers combine linear attention with Gated MLA at a 7:1 ratio.

Model Specifications

Specification Value
Model Type Base checkpoint
Architecture Hybrid-linear MoE
Parameter Scale Total 560B, Activated 25B
Transformer Layers 63 KDA + 9 Gated MLA (7:1)
Number of Dense Layers 4
Number of Routed Experts 512
Number of Shared Experts 1
Number of Activated Routed Experts 8
Attention Heads 64
Hidden Size 4096
Expert Intermediate Size 1280
Dense Intermediate Size 10240
Vocabulary Size 157184

Architecture Diagram

Ling-3.1-flash architecture

Base Model Evaluation

We evaluate Ling-3.1-flash-base on 47 benchmarks across eight domains: language understanding, knowledge, code, mathematics, reasoning, multilingual capabilities, medical knowledge, and long-context understanding. The results below compare the model with DeepSeek-V4-Pro-Base and MiMo-V2.5-Pro-Base under the shot configurations shown.

Bold indicates the highest score in each row; underlining indicates the second-highest distinct score. Tied scores receive the same formatting. EM denotes exact match, and Acc denotes accuracy. Parameter scales are shown as total parameters and active parameters per token (A).

Domain Benchmark Shot Config Ling-3.1-flash-base DeepSeek-V4-Pro-Base MiMo-V2.5-Pro-Base
560B A25B 1.6T A49B 1.02T A42B
Language Understanding Race-High(EM) 0-shot 93.88 93.37 93.60
SquadV2(Acc) 1-shot 92.79 87.70 88.93
Belebele(EM) 0-shot 94.67 93.83 92.72
C3(EM) 0-shot 83.34 81.64 81.53
CCPM(EM) 0-shot 91.88 93.31 90.18
Knowledge ARC-c(EM) 0-shot 97.97 96.61 96.95
AGIEval(Acc) few-shot(0~5) 82.59 81.26 79.90
SimpleQA-Verified(Acc) 5-shot 52.40 52.10 32.60
C-SimpleQA(Acc) 5-shot 77.43 77.60 70.93
AA-Omniscience(Acc) 3-shot 37.67 48.16 36.45
MMLU-Pro(EM) 5-shot 76.71 75.28 71.26
CEval(EM) 5-shot 92.36 93.76 92.08
CMMLU(EM) 5-shot 91.60 90.52 88.33
GPQA-Diamond(EM) 0-shot 56.57 52.02 54.04
SuperGPQA(EM) 5-shot 55.84 53.19 48.65
Code HumanEval-Plus(Pass@1) 0-shot 79.88 57.93 82.93
CruxEval(Pass@1) 1-shot 80.50 69.06 71.25
MultiPL-E(Pass@1) 0-shot 70.55 55.94 64.74
MBPP-Plus(Pass@1) 3-shot 76.19 73.81 77.51
BigCodeBench(Pass@1) 0-shot 53.60 51.49 66.75
LCBench1(Pass@1) 3-shot 68.48 59.14 67.58
LCBench-cn1(Pass@1) 3-shot 60.83 57.35 63.36
Math MATH500(Acc) 4-shot 79.40 68.20 57.20
CollegeMath(Acc) 4-shot 73.41 63.15 66.61
FinanceReasoning(Acc) 3-shot 81.76 82.16 80.89
OlympiadBench(Acc) 3-shot 60.24 46.39 48.04
TheoremQA(Acc) 5-shot 63.56 61.92 60.27
OmniMath(Acc) 3-shot 65.31 34.89 40.13
Reasoning CommonSenseQA(EM) 5-shot 92.22 86.49 88.53
BBH(EM) 3-shot 90.25 89.98 89.80
KorBench(Acc) 3-shot 54.96 54.56 53.20
AutoLogi(Acc) 3-shot 67.13 64.81 66.23
ZebraLogic(Acc) 1-shot 30.80 20.30 28.90
Multilingual mARC(EM) 0-shot 92.98 89.10 88.55
MMMLU(EM) 0-shot 77.56 68.81 67.15
MultiGSM(Acc) 4-shot 87.66 85.87 86.33
HumanEvalXL(Pass@1) 0-shot 83.25 70.31 87.34
Med DiagnosisArena(EM) 0-shot 62.20 55.50 52.70
medmcqa(EM) 5-shot 78.08 80.33 77.50
medqa(EM) 5-shot 87.82 87.75 85.55
Long-context LongBench(Acc) 0-shot 51.90 17.70 15.66
LEval(Acc) 0-shot 85.09 58.34 61.49
ruler_16k(Acc) 1-shot 96.90 97.00 90.61
ruler_32k(Acc) 1-shot 95.87 95.57 89.10
ruler_64k(Acc) 1-shot 93.81 95.56 87.19
ruler_128k(Acc) 1-shot 88.64 95.06 85.79
ruler_256k(Acc) 1-shot 86.64 94.37 86.00
Note:
1 LCBench and LCBench-cn: January 2023–February 2025 (2301-2502).

Intended Use

Recommended use cases:

  • Continued pretraining.
  • Mid-training.
  • Supervised fine-tuning for domain adaptation.
  • Preference optimization and reinforcement-learning post-training.
  • Distillation research.
  • Long-context and MoE systems research.

Not recommended as-is for:

  • Direct end-user chat deployment.
  • Safety-critical applications without additional alignment and evaluation.
  • Production use without post-training and task-specific validation.

Usage

For fine-tuning examples, please refer to our ling-cookbook.

FAQ

For questions or feedback, please open a discussion in the model repository.

License

This model is released under the MIT License.

來源:螞蟻 inclusionAI:HuggingFace 新模型 · huggingface.co