AI 的計量單位是瓦特:下一個運算瓶頸將轉移至何處
The Unit of AI Is the Watt: Where the Compute Bottleneck Moves Next
ArkNet Digital 的分析主張,AI 運算瓶頸正由 FLOPs 轉向記憶體頻寬、互連,再延伸至電力供應與電網。GB300 與 Vera Rubin 的對照中,FP8 dense compute 增長 3.5×,機櫃功率則只增長 1.4×;美國 2025 年投產的電力項目,從電網接駁申請至商業運作的中位時間為 61 個月。
The Unit of AI Is the Watt: Where the Compute Bottleneck Moves Next
Every AI roadmap I read still treats FLOPs as the scarce resource. They aren't. FLOPs are now the most abundant thing in the building. The constraint has already moved twice, and it's about to move again, out of the chip and into the power grid.
If you build AI products, this isn't trivia for hardware people. It decides what your inference costs, which architectures win, and which features you can afford to ship.
Here's the argument from first principles, using numbers the vendors publish themselves.
1. Compute is outrunning everything around it
Put Nvidia's current rack generation next to the one it replaces. The figures below come from Nvidia's Vera Rubin NVL72 spec page and a side-by-side compiled by Lightyard from Nvidia's datasheets. GB300 rack power comes from an OEM guide, since Nvidia doesn't print it.
| Per rack | GB300 NVL72 | Vera Rubin NVL72 | Growth |
|---|---|---|---|
| FP8 dense compute | 360 PFLOPS | 1,260 PFLOPS | 3.5× |
| HBM bandwidth | 576 TB/s | 1,400 TB/s | 2.4× |
| NVLink bandwidth | 130 TB/s | 216 TB/s | 1.7× |
| Rack power | 135 kW TDP | 187 kW (Max-Q) | 1.4× |
Read the growth column from top to bottom. Each resource grows more slowly than the one above it. Compute triples, memory bandwidth more than doubles, the interconnect grows by two-thirds, and the power you're allowed to pull grows by less than half.
That ordering is the whole story. Chips are getting cheaper and more capable faster than you can feed them, connect them or power them. The bottleneck isn't a single wall. It's a queue of walls, and the industry is walking down it.
2. Wall one: memory bandwidth (you're already here)
Start with the physics of generating a token.
During autoregressive decode, every weight in the model has to stream from HBM into the compute units for each forward pass. With FP8 weights that's one byte per parameter, and each byte does roughly two floating-point operations (a multiply and an add) for each sequence in the batch. So decode arithmetic intensity is about 2 × batch size FLOPs per byte, before you count the KV cache, which only adds bytes.
Now look at what the hardware wants. A single Rubin GPU is listed at 17.5 PFLOPS dense FP8 and 19.2 TB/s of HBM4 bandwidth (Nvidia). Divide one by the other and you get a ridge point of about 910 FLOPs per byte. On GB300 (per GPU: 5 PFLOPS FP8 dense, 8 TB/s) it was about 625.
Two conclusions fall straight out of that division:
- At small batch sizes, decode is bandwidth-bound by a wide margin. To keep a Rubin's tensor cores busy on weights alone, you'd need a batch in the hundreds of concurrent sequences per GPU. Real traffic, with KV-cache reads on top, sits far below that ridge for most of the day.
- The gap is widening. The ridge point rose roughly 45% in one generation. Each new chip makes small-batch, single-user, low-latency inference relatively worse at using the silicon you paid for.
A quick sanity bound: a 70B-parameter model in FP8 is about 70 GB of weights. At 19.2 TB/s, one GPU can stream those weights at most about 270 times a second. That's the hard ceiling on batch-1 tokens per second for that model on that chip, no matter how many PFLOPS the marketing slide shows. In practice you'll land below it.
This is why the inference stack of 2026 looks the way it does. Every technique that matters is a bandwidth trick wearing a different hat:
- Quantization (FP8, FP4): fewer bytes per weight.
- Batching and continuous batching: more FLOPs per byte loaded.
- Speculative decoding: a small model drafts, the big one verifies several tokens per weight pass.
- Mixture-of-experts: only part of the weights move per token.
- KV-cache compression, paging and offload: fewer bytes per token of context.
- Disaggregated prefill and decode: run the compute-bound phase and the bandwidth-bound phase on hardware sized for each.
If your team talks about "model quality" and never about bytes moved per token, it's optimizing the wrong variable.
3. Wall two: the interconnect (frontier labs are here)
Once a model, its KV cache and its parallelism strategy no longer fit inside one GPU, every token turns into a communication problem. Tensor parallelism means all-reduce traffic on every layer. Expert parallelism means all-to-all traffic. Long-context serving means cache shuffling.
That's why the NVLink domain, not the GPU, has become the real unit of compute. A Rubin NVL72 rack is designed as one 72-GPU machine with 216 TB/s of NVLink, while its scale-out networking is listed at 32.4 TB/s per rack (Nvidia). That's roughly a 7× cliff between talking inside the rack and talking outside it.
The architectural rule follows from that cliff: keep the chatty traffic inside the domain and send only the polite traffic across the network. Tensor and expert parallelism live inside the rack. Data parallelism and pipeline stages cross it. Model designs that respect that boundary run well, and designs that don't burn money waiting on the wire.
For anyone renting rather than building, the practical translation is short. When a provider quotes "GPUs," ask about the domain size and the scale-out bandwidth. Two clusters with the same GPU count can differ several-fold in useful throughput on a large MoE model.
4. Wall three: power (everyone is about to be here)
Now zoom out to the building.
Nvidia's own reference design puts about 40,000 Rubin GPUs in a 100 MW AI factory (Nvidia). That's roughly 2.5 kW of facility power per GPU once cooling and networking are counted. The chip is no longer the expensive, scarce input. The megawatt is.
The macro numbers agree. The IEA estimates data centers used about 415 TWh in 2024 and projects about 945 TWh by 2030 in its base case, slightly more than Japan's entire electricity consumption today (IEA, Energy and AI). It projects electricity use by accelerated servers growing about 30% a year.
And here's the part most AI strategy decks skip. In the US, the median power project that came online in 2025 took 61 months from interconnection request to commercial operation, up from 36 months in 2015 and 22 in 2008. About 2,061 GW of capacity was still waiting in queues at the end of 2025 (Berkeley Lab, Queued Up 2026).
Put those two clocks side by side. GPUs ship on a roughly annual cadence. New grid power takes about five years. When one input improves every 12 months and another takes 60, the slow one sets the price of everything.
That's why Nvidia now publishes three power profiles for the same rack (Max-P at 227 kW, Max-Q at 187 kW and MaxLPS at 156 kW), and describes MaxLPS as a way "to maximize AI factory output within a fixed data center power budget" (Nvidia Vera Rubin datasheet). When the world's leading chip company sells you the option to run its chips slower so you can fit more of them under your power cap, it's telling you what the binding constraint is.
5. The bottleneck ladder
Here's the mental model I use. Call it the bottleneck ladder:
- FLOPs: solved by every new generation.
- Memory bandwidth: where single-user inference lives today.
- Interconnect: where frontier training and large MoE serving live.
- Power and cooling: where data center builders already live.
- Grid, permits and transformers: where the whole industry is heading.
Two properties make the ladder useful:
- Each rung moves more slowly than the one below it. Silicon iterates yearly, packaging and memory every few years, buildings and grids on multi-year cycles. The higher rungs are where scarcity, and therefore margin, collects.
- The optimal software changes at every rung. At rung 2 you optimize bytes per token. At rung 3 you optimize communication patterns. At rung 4 you optimize tokens per joule, and that's a different objective from tokens per second.
6. What this means if you don't own a data center
Most companies will never buy a rack, but the ladder still sets their costs. Here's what I'd tell a CTO:
Price your product in joules, not tokens. Your provider's price per token is a lagging proxy for its power and bandwidth costs. Features that need long contexts, high reasoning budgets or batch-1 latency are expensive because they fight rung 2 and rung 4. Know which of your features do that.
Make latency a product decision, not a default. Batch-friendly workloads such as overnight analysis, async agents and bulk extraction ride the cheap side of the ridge. Real-time chat rides the expensive side. If a workflow can tolerate 30 seconds, design it to, and buy batch pricing where it exists.
Cut tokens before you cut models. Retrieval that sends 4,000 relevant tokens instead of 40,000 "just in case" tokens is a 10× change in bytes moved. That beats any vendor negotiation you'll ever have.
Route by difficulty. A small model drafting and a large model verifying is speculative decoding at the application layer. Most traffic in most products is easy. Pay frontier prices only for frontier problems.
Treat "AI compute is getting cheaper" with care. The price of a fixed capability falls fast. The price of frontier capability, served at low latency with long context, is pinned to the slowest rung on the ladder. Plan budgets for both curves.
The thesis, in one line
The last decade of AI was a race for FLOPs. This decade is a race for watts, bytes and wires. The winners will move the fewest bytes, per token, per joule, inside the smallest communication domain that fits the job.
The chip isn't the product anymore. The power-constrained system is. Design accordingly.
Michael, ArkNet Digital: AI systems, agent infrastructure and security engineering. arknet.digital
If you're sizing inference costs, or deciding between real-time and batch architectures for an AI feature, I'm glad to compare notes. Disagree with the ladder? Tell me which rung I've got wrong in the comments.
來源:DEV Community · dev.to