跳到正文
DEV Community· 于侃·· 9 天前AI 評分59

Prefill/decode 分離會把 KV cache 傳輸成本轉移到網絡鏈路

Your prefill/decode split has a network bill nobody puts in the diagram

AI 導讀

Prefill 與 decode 分置不同 GPU 池可減少 prefill 對同一 GPU 其他串流的阻塞,但 KV cache 跨機傳輸會進入請求關鍵路徑。

正文

A colleague's post this week made the case for splitting prefill and decode onto separate GPU pools: an 8,000-token prompt costs about 320 ms of pure compute-bound work, and during those 320 ms every other stream on that GPU freezes. Running prefill elsewhere fixes the tail. All of that is right.

The part missing from most diagrams of the split is the arrow between the two boxes, and that arrow has a size.

The KV cache is bigger than you think per token

The KV cache for one token is:

bytes_per_token = layers x kv_heads x head_dim x 2 (fp16)

For a 32-layer model with 8 KV heads at 128 dimensions:

32 x 8 x 128 x 2 = 65,536 bytes = 64 KB per token

That 64 KB is per token, per sequence. An 8,000-token prompt therefore produces a 512 MB cache that has to exist on the decode pool before a single output token can be generated. If prefill and decode are on different machines, those 512 MB move across the network, and it happens on the request's critical path.

What that costs in milliseconds

Link 512 MB transfer vs 320 ms prefill
10 Gb/s ~410 ms worse than doing it locally
25 Gb/s ~164 ms half the work you just moved
100 Gb/s ~41 ms the split clearly wins

At 25 Gb/s the transfer eats about half of the win. At 10 Gb/s the split is a net loss for long prompts, which is exactly the case it exists to fix. The saving is real; it just lives on a different cable than the one the blog post is about.

So the split is a network decision, not only a serving decision

Three practical consequences:

1. Measure cache size before you measure throughput. It is four numbers from the model config, and it tells you whether your interconnect can carry the workload at all. A model with fewer KV heads (grouped-query attention) is dramatically cheaper to disaggregate than one without: 8 KV heads instead of 32 is a 4x smaller transfer, which can flip the decision.

2. Long context is where the split helps most and costs most. The 320 ms prefill grows linearly with prompt length, and so does the transfer. They track each other, so the ratio barely improves - you are trading one linear cost for another, plus a constant.

3. Quantized or compressed KV caches are a serving feature and a networking feature at the same time. Halving the bytes per token halves the transfer, which is often worth more than the memory it saves.

A number to put on the diagram

If I were annotating one of those architecture diagrams, the label on the arrow would not be "KV cache". It would be the transfer time in milliseconds at the link speed you actually have:

transfer_ms = bytes_per_token x context_tokens x 8 / link_bits_per_second x 1000

If that number is a meaningful fraction of the prefill you removed, the split has not moved the work off the critical path - it has moved it onto a wire, where it is harder to see in a flame graph and much harder to shrink.

None of this argues against disaggregation. It argues for doing the arithmetic first, because the answer depends on an interconnect number that never appears in the model comparison table.

來源:DEV Community · dev.to