DEV Community· SparkLLM·· 4 小時前AI 評分47
在 Rockchip RK3576 和 RK3588 上運行 Spark-X2.5 1.7B
Running Spark-X2.5 1.7B on Rockchip RK3576 and RK3588
AI 導讀
Rockchip 的 RKNN-LLM v1.3.1 基準測試列出 Spark-X2.5 1.7B 在 RK3576 和 RK3588 上的首 token 延遲、生成速度及記憶體用量。
正文
How long do you wait for the first token, and how fast do the next ones arrive? Rockchip’s RKNN-LLM v1.3.1 benchmark lists both measurements for Spark-X2.5 1.7B.
| Board | Quantization | First token | Generation | Memory |
|---|---|---|---|---|
| RK3576 | W4A16 | 768.66 ms | 13.57 tokens/s | 1,028.58 MB |
| RK3588 | W8A8 | 386.23 ms | 15.61 tokens/s | 1,830.32 MB |
Rockchip used 128 input tokens and requested 64 new tokens, with the CPU and NPU running at their respective maximum frequencies. The two rows use different quantization formats. For a board you plan to deploy, start with its listed format and test your own prompt length. Record time to first token separately from generation speed.
來源:DEV Community · dev.to
