跳到正文
DEV Community· Ikkun·· 10 天前AI 評分53

LFM2.5-2.6B Q4 在 livedoor 分類準確率為 10.8%,已作答答案有 93.1% 正確

The 2B model scored 10%. 93% of its answers were right

AI 導讀

LFM2.5-2.6B Q4 在 livedoor 日文新聞分類任務的整體準確率為 10.8%,但只回答 29/250 題,當中 27 題正確(93.1%)。測試以每項 250 題比較多個本地模型與 Jev 託管 API;在首 40 題把 token 預算由 512 提至 2,048 後,28 題有作答,11 題仍因達上限而無輸出,另有 1 題輸出無效標籤。

正文

I expected small local models to be worse than a hosted decision API at Japanese classification. I did not expect most of the gap to be something other than judgement. On the hardest of three tasks one model scored 10.8% overall while getting 27 of the 29 answers it produced right. The other 221 rows it never answered.

What I ran, and what the accuracy number counted

Three Japanese tasks, 250 identical rows each. livedoor is 9-class news-topic classification (mean 1,308 characters), rakuten is 2-class review polarity (165), chabsa is 3-class financial-sentence polarity (100). Chance is 11.1%, 50.0% and 33.3%. Every system got the same prompt asking for one label as JSON, and a row counted correct only if the parsed label matched gold. Unparseable output counted as wrong.

System livedoor rakuten chabsa
Jev (hosted API) 76.8% 94.4% 74.0%
Gemma 4 26B-A4B (local MoE) 65.2% 94.4% 75.2%
SemIf (Qwen3.5-4B) 68.8% 92.0% 64.0%
phi-4-mini 3.8B Q4 38.0% 89.6% 71.6%
MiniCPM5-2B Q4 36.4% 89.2% 61.6%
LFM2.5-2.6B Q4 10.8% 75.6% 26.4%

How many rows actually produced an answer?

This is the column I should have read first:

Model livedoor rakuten chabsa
phi-4-mini 3.8B 243/250 249/250 250/250
MiniCPM5-2B 197/250 250/250 250/250
LFM2.5-2.6B 29/250 199/250 87/250

Score only the rows that answered and the ranking moves. LFM2.5-2.6B's livedoor figure becomes 93.1% (27/29), and its rakuten figure 95.0%, above phi-4-mini's 89.6% and level with the hosted API's 94.4%. These models were not failing to judge. They failed to finish.

Why a small model returns nothing

I reproduced it on the mini-PC on 2026-10-26 (llama.cpp 60081bb, CPU-only) on the same livedoor row. With a 512-token budget the response body came back empty with finish_reason: length and 512 completion tokens spent. The tokens went to a populated reasoning_content field holding the model's step-by-step working: it ran out of budget before the answer.

Raise the budget to 2,048 and that row returns {"label": "MOVIE ENTER"}, correct, finish_reason: stop, 790 tokens. Same weights and prompt, temperature 0.

Raising the budget is not a complete fix. On the first 40 livedoor rows at 2,048: 28 answered (70%, against 11.6% at 512), but 11 still hit the cap and returned nothing, and one answered with an out-of-vocabulary label (IT Lifehack) that a strict parser rejects. Answer rate and answer validity are separate failure modes.

A reasoning-capable model spends tokens thinking, and a budget sized for the answer truncates it to nothing, which the harness scores as wrong. My earlier Gemma 4 E2B run shows the same shape at the format layer: 142 of 250 rows emitted valid JSON (56.8%), and 140 of those 142 were correct (98.6%).

Does a 2B model ever beat the API?

On accuracy, in my data, no. Paired McNemar on identical row ids, two-sided: phi-4-mini vs Jev −38.8pt on livedoor (p=1.2e-21), −4.8pt on rakuten (p=0.029), −2.4pt on chabsa (p=0.59); MiniCPM5-2B −40.4pt / −5.2pt (p=0.0072) / −12.4pt (p=0.0042); SemIf 4B −8.0pt (p=0.0012) / −2.4pt (p=0.070) / −10.0pt (p=0.014). The gap narrows as the task gets shorter and more lexical: 4.8 points on the 165-character polarity task, 39 to 40 on the long 9-class one.

Costs do not favour the small side either. Summed per-row latency over all 750 rows: MiniCPM5-2B 20 minutes, phi-4-mini 39, SemIf 4B 40, LFM2.5-2.6B 191.

Meanwhile 2,000 Jev requests across eight experimental arms produced zero unparseable responses. A closed-choice API cannot format-fail; a chat model can, and that difference dominates the table.

Measure answered rows before accuracy: a number averaging "wrong" and "silent" tells you nothing about which one you are buying. Background is in the zero-shot versus trained comparison and the CPU throughput limits.

Measured 2026-09-19 to 2026-10-26 on a Ryzen 9 7940HS mini-PC, llama.cpp 60081bb, Q4_K_M weights, temperature 0, 250 rows per task.

If you have sized a token budget for a local model's answers rather than its thinking, what budget actually cleared it?


Originally published at agentjournal.dev.

來源:DEV Community · dev.to