Bandwidth as the Price of a Token: Inference Economics, CXL, and Processing-in-Memory
Follow the chain to its end and it terminates in a dollar figure: the cost of generating a token. Because decode is memory-bound, that cost is governed by memory bandwidth and by how much memory sits near the compute. This is why serving optimizations are bandwidth optimizations in disguise, and why two escape routes, pooling memory over a fast fabric and moving computation into the memory itself, are drawing serious investment. My read: neither repeals the bandwidth wall, but both change where it binds. The token price is the bottom of the causal chain that starts at the GPU generation.
Knowledge check
Why is the cost of generating a token during LLM inference governed by memory bandwidth rather than by the accelerator's compute throughput?
Show the answer
Token generation in the decode phase reads the full model weights from memory for each token while performing very little arithmetic per byte, making it memory-bound. The token rate is therefore set by how fast weights can be streamed from memory, so bandwidth, not compute, determines throughput and hence the cost per token.