Cross-die interconnect fabric congestion emerges as a critical bottleneck in multi-GPU token decoding systems, exacerbated by memory coherency protocol latency as dynamic batch sizes scale. While NVLink-based systems achieve significantly higher bandwidth (900 GB/s) compared to PCIe alternatives (128 GB/s), the fundamental constraint shifts from computation to KV cache memory capacity and inter-GPU communication overhead, requiring architectural innovations in attention mechanisms and scheduling strategies to maintain latency SLOs.
Multi-GPU LLM inference systems face a nuanced hardware bottleneck landscape where traditional compute-centric performance metrics no longer capture system behavior. The transition from single-GPU to multi-GPU token decoding introduces two interrelated challenges: cross-die interconnect fabric congestion and memory coherency protocol latency. Understanding these constraints is essential for optimizing real-time inference serving with dynamic batch size scaling.
Contrary to conventional wisdom, computation is no longer the primary constraint in modern LLM inference. Research demonstrates that "the bottleneck is not compute — it is memory" [8], with KV cache management consuming approximately 1.6 GB per inference sequence in typical deployments [8]. This fundamental shift reframes multi-GPU architecture discussions: the inter-GPU communication fabric becomes critical precisely because it manages memory coherency and data movement rather than computational results.
The scaling challenge intensifies under dynamic batch sizing. Single-node deployments exhibit superior latency characteristics at small batch sizes but "performance degrades rapidly under increased" batch loads [2]. This degradation pattern suggests that interconnect saturation occurs at batch size thresholds where cumulative KV cache access patterns exceed fabric throughput capacity.
Modern GPU systems employ heterogeneous interconnect topologies. NVIDIA-based systems leverage NVLink, which achieves "900 GB/s while a PCIe machine doing the same thing is stuck at 128 GB/s" [16]. This sevenfold bandwidth advantage stems from NVLink's architectural design: "NVLink connections are designed for energy-efficient, tightly synchronized transfers with lower latency and higher bandwidth" [17]. Empirical measurements show NVLink achieving "80-90%+ of its theoretical bandwidth, while PCIe connections often topped out at 60-70%" [19], indicating efficiency differences beyond raw specifications.
AMD's approach presents alternative data paths: "This interconnect creates multiple paths for data movements between all processors in the system, CPU and GPUs, and can be utilized using various interfaces" [1]. However, architectural heterogeneity introduces complexity in optimizing coherency protocols across these diverse pathways.
Topological constraints compound interconnect limitations. In large-scale systems, "simplified intra-server PCIe topology" with "eight GPUs interconnected via NVLink" combined with "Host–GPU data" movement creates hierarchical bottlenecks [18]. Dynamic batch scaling exposes these hierarchical inefficiencies as all GPUs attempt simultaneous KV cache coherency updates.
Token decoding presents distinct memory access patterns compared to prefilling. During decoding, each new token generation requires reading the entire KV cache accumulated from previous tokens—an operation repeated for every output token. The latency of memory coherency protocols becomes deterministic in decode phases rather than probabilistic, as every token generation triggers synchronized KV cache lookups across participating GPUs.
Inter-GPU communication "is the foundation of distributed training. It's what keeps multiple GPUs aligned as they compute" [3]. In inference scenarios, this alignment requirement manifests as cache coherency barriers that serialize otherwise parallelizable operations. When batch sizes increase dynamically, the aggregate coherency overhead grows super-linearly due to contention on the interconnect fabric.
The "scaling of LLMs toward long-context inference has shifted the primary serving system bottleneck from computation to memory capacity" [6]. This observation directly contradicts assumptions underlying older multi-GPU scaling strategies. Modern optimization techniques address this through attention mechanism innovations rather than traditional parallelism.
Paged Attention represents one architectural response: "KV cache space is reduced to less than 4%, representing a 3x-5x improvement over prior systems. This enables larger batch sizes and directly" improves serving capacity [7]. By reducing KV cache memory footprint, these techniques proportionally reduce inter-GPU coherency traffic, alleviating interconnect congestion.
Alternative approaches like Multi-Query Attention (MQA) and Grouped Query Attention (GQA) similarly reduce data movement requirements. "Combining FlashAttention with sparse patterns and GQA can reduce prefill latency by 5-10× for million-token inputs compared to naive" implementations [15]. While primarily targeting prefill, these efficiency gains translate to reduced coherency protocol overhead across decode phases.
Real-time LLM inference systems must maintain Service Level Objectives (SLOs) for both Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT). Under dynamic batch scaling, the relationship between batch size and latency becomes non-linear due to interconnect saturation. "Predicted-Latency Based Scheduling for LLMs" demonstrates that "the predicted-latency model directly estimates TTFT and TPOT per server, allowing the scheduler to compute headroom against SLO targets" [12].
When interconnect fabric congestion increases coherency protocol latency, traditional queue-based scheduling fails to maintain SLOs. Disaggregation strategies offer partial remediation: "LAPS reduces prefill latency by over 30% compared to vanilla SGLang under prefill–decode disaggregation, and further decreases SLO violations" [13]. By separating prefill and decode processing phases across distinct GPU cohorts, disaggregation reduces simultaneous coherency traffic peaks.
Mixture-of-Experts (MoE) models introduce additional data movement complexity. "MoE models dynamically route each token to only a subset of experts, introducing substantial data movement overhead. Such overhead already" constrains scaling [11]. In multi-GPU deployments, MoE routing decisions interact poorly with coherency protocols, as token routing creates unpredictable memory access patterns across the interconnect fabric.
PCIe topology constraints become especially limiting in MoE scenarios. Systems restricted to older mining rigs or non-specialized hardware face severe interconnect limitations, as GPU "PCI slots" availability directly determines communication bandwidth [4]. This hardware reality underscores why interconnect infrastructure selection fundamentally bounds achievable inference throughput.
The attention mechanism itself introduces caching requirements that stress interconnect capacity. "Constraints emerge from GPU/TPU RAM size, bandwidth, and latency, as well as the need to support low prefill latency (time to first token)" [9]. These constraints interact multiplicatively: low prefill latency requires rapid KV cache population, while maintaining coherency across multiple GPUs demands careful synchronization.
Fast decoding techniques aim to address this through optimized attention computation: "Techniques such as flash attention, flash decoding, and multi-query attention can improve efficiency during training and inference. Quantizing" further reduces data volume [14]. However, these techniques address computation efficiency rather than coherency protocol latency, leaving the fundamental interconnect bottleneck partially unresolved.
The analysis reveals several critical design decisions. Systems supporting dynamic batch scaling must prioritize interconnect fabric bandwidth over raw compute throughput. NVLink-based architectures provide sevenfold bandwidth advantages over PCIe, justifying significant capital expenditure for specialized hardware. For budget-constrained deployments, architectural choices favoring lower batch sizes and disaggregated prefill-decode processing become necessary.
Operationally, batch size scheduling must account for interconnect saturation explicitly rather than treating it as a secondary concern. SLO management becomes interconnect-aware rather than purely latency-aware. Monitoring coherency protocol latency directly provides early warning signals for capacity exhaustion.
Cross-die interconnect fabric congestion and memory coherency protocol latency represent genuine hardware constraints that cannot be resolved through algorithmic optimization alone. While attention mechanism innovations (Paged Attention, GQA, FlashAttention) provide 3-10× efficiency improvements, these gains merely extend the operational window before interconnect saturation occurs. Dynamic batch scaling requires explicit interconnect management, disaggregation strategies, and hardware selection favoring high-bandwidth interconnects like NVLink. Future multi-GPU inference systems must treat interconnect design and coherency protocol latency as first-class architectural concerns rather than secondary implementation details.