低成本模型每 token 成本僅為前沿模型的 1/20、編碼得分不到一半:作者公開 60 秒實測
The 1/20-cost model scored less than half the coding points — so I made the 60-second proof public
作者以固定編碼任務比較低成本模型與前沿模型,發現前者每 token 成本約為後者的 1/20,編碼得分不到一半,並公開 60 秒實測過程。兩個模型使用相同提示詞和 temperature 0,按保留測試集的 pass@1 評分,不重試、不挑選結果;成本則按供應商公佈的每 token 費率及實際 completion tokens 計算。
I've been running a small eval harness that pits a budget model against a frontier one on the same fixed set of coding tasks. The result kept surprising me: the cheap model runs at roughly 1/20 the per-token cost and lands well under half the coding score of the big one. Numbers on a spreadsheet don't convince anyone, so I recorded the whole run as a short walkthrough: https://www.youtube.com/shorts/Kk0DbwAPBTU
Here's the concrete detail of how it works. The harness feeds each task to both models with identical prompts and temperature 0, then scores pass@1 against a held-out test set — no retries, no cherry-picking. Cost is computed from the provider's published per-token rates times actual completion tokens, so a verbose model pays for its own verbosity. That last part is what makes the comparison honest: the cheap model isn't winning because it answers less, it's winning on raw task completion per dollar.
The thing it costs me as the operator is real money and real time. Every full benchmark sweep burns through a few dollars in API calls and about twenty minutes of wall-clock time waiting on the frontier model's slower completions. That's the trade I accept — I'd rather spend that than publish a chart nobody believes. If you've ever run your own model comparisons, you know the gap between "I measured this" and "watch me measure this" is the whole ballgame.
來源:DEV Community · dev.to