Multi-chiplet GPU token decoding systems face critical memory bandwidth arbitration challenges that require coordinated priority scheduling, heterogeneous memory management, and dynamic quantization format switching to maintain throughput under real-time inference constraints. Current approaches leverage speculative decoding, KV-Cache optimization, and fine-grained memory controller scheduling to mitigate inter-chiplet communication overhead, though fundamental trade-offs between latency, power, and area remain unresolved across different chiplet granularities.
The intersection of heterogeneous memory bandwidth arbitration, priority scheduling, and multi-chiplet GPU architectures presents a complex optimization landscape for token decoding in large language model (LLM) inference. This analysis examines how hardware memory controllers must dynamically manage competing demands for bandwidth across distributed compute chiplets while supporting variable-precision quantization formats and maintaining real-time latency guarantees.
Memory bandwidth, rather than compute capacity, has emerged as the primary constraint in LLM token decoding workloads. Research indicates that storage bandwidth represents a fundamental limitation in LLM inference throughput [1]. This bottleneck intensifies in multi-chiplet systems where inter-chiplet communication introduces additional latency and consumes precious bandwidth resources [19].
The challenge manifests differently across inference phases. Prefill operations involve high-throughput computation over input sequences, while decoding produces single tokens with substantial memory-to-compute ratios. When both operations compete on shared memory infrastructure, bandwidth contention becomes severe. DualPath addresses this by introducing dual-path KV-Cache loading mechanisms that leverage decoding-engine bandwidth independently from prefill paths [2], fundamentally decoupling bandwidth consumption profiles.
Modern multi-chiplet GPUs employ hierarchical memory systems combining high-bandwidth on-chiplet SRAM, intermediate-speed HBM across chiplet boundaries, and tertiary storage through flash-based systems [3]. Hardware-based heterogeneous memory management strategies must intelligently map data placement and computation to exploit asymmetric memory architectures [16]. This requires memory controllers capable of understanding data access patterns and dynamically steering requests toward appropriately-tiered memory.
LPDDR memory deployment as a secondary tier demonstrates practical tradeoffs between cost and performance, enabling balanced inference throughput [17]. However, this architectural complexity demands sophisticated scheduling policies that prioritize requests based not merely on arrival time, but on their criticality to forward progress and quantifiable impact on end-to-end latency.
Memory controller chiplets must implement multi-level priority arbitration that considers: (1) token generation deadline criticality—later tokens in sequences have more flexible timing than tokens within active attention windows; (2) KV-Cache reuse opportunity—requests that unlock subsequent token computation should receive elevated priority; (3) memory tier appropriateness—steering requests to matching bandwidth-capacity tiers.
Speculative decoding frameworks provide one mechanism for priority-aware scheduling. FASER demonstrates that fine-grained phase management can minimize computational waste through dynamic condition serving, achieving 53% higher throughput and 1.92× lower latency compared to baseline approaches [11]. By predicting which speculative branches will be committed, the memory system can proactively prioritize their KV-Cache and model parameter loads.
BATS (Batched Attention-optimized Speculative Sampling) extends this concept by processing draft tokens in parallel, amortizing memory I/O costs of model parameters across multiple tokens [15]. This parallelization reduces per-token bandwidth pressure and enables more efficient controller arbitration across draft and verification phases.
Quantization format heterogeneity—combining FP8, BF16, and INT8 representations—optimizes the throughput-accuracy-efficiency frontier [8]. However, format switching introduces memory controller complexity: different quantization formats require distinct memory access patterns, cache line utilization characteristics, and compute-to-memory bandwidth requirements.
Dynamic format switching during inference requires memory controllers that can: adapt cache coherency protocols for mixed-format operations, adjust prefetch strategies based on anticipated format precision, and schedule bandwidth allocation accounting for format-specific memory efficiency. A token layer might use INT8 quantization for attention operations while maintaining BF16 precision for embedding layers, demanding controller intelligence to optimize each access pattern independently.
This scheduling challenge intensifies in real-time scenarios where format selection decisions must adapt within microsecond timescales based on intermediate accuracy metrics or per-token bandwidth pressure.
Chiplet-based architectures introduce fundamental trade-offs between granularity and interconnection overhead [6][10]. Simba's work establishes that tiling optimizations significantly improve data locality and reduce inter-chiplet traffic, with three primary strategies: replicating frequently-accessed model weights to minimize remote loads, temporal blocking to enhance KV-Cache reuse within chiplet boundaries, and spatial arrangement of compute and memory chiplets to reduce hop counts [19].
UCIe interconnect architecture provides standardized protocols for chiplet communication, but introduces latency and power costs inversely proportional to bandwidth provisioning [7]. Memory controllers must therefore model cache miss rates and network latency as functions of cache hierarchy and interconnection parameters [20], enabling feedback-driven optimization of request routing and prioritization policies.
Token decoding for interactive applications demands sub-100ms latency for individual tokens while maintaining throughout. This necessitates memory controller arbitration that prevents tail-latency scenarios where high-priority tokens experience queuing delays behind lower-priority requests.
HiFC's fine-grained block mapping approach, confining writes to high-performance pSLC zones, demonstrates hardware memory control mechanisms that stabilize latency and throughput under load [3]. Such techniques must extend to arbitration: priority scheduling must account for blocking probabilities—if a high-priority request might contend with multiple lower-priority operations, proactive demotion of lower-priority requests prevents convoy effects.
KV-Cache management represents the central operational concern in bandwidth arbitration. Distributed inference strategies including tensor parallelism split model weights across GPUs, but KV-Caches remain distributed according to sequence partitioning schemes [4]. KV-aware routing principles transform wasted recompute into accelerated throughput by optimizing cache reuse [5].
Memory controllers implementing KV-aware priority schemes elevate requests for KV-Cache entries that unlock attention computation over requests for model parameters that might be amortized across multiple tokens. SLED's framework demonstrates that speculative decoding with heterogeneous device support and reduced server-side memory consumption becomes possible when scheduling integrates KV-Cache locality awareness [14].
Integrating these mechanisms—heterogeneous memory arbitration, dynamic quantization, multi-chiplet locality optimization, and real-time priority scheduling—into unified hardware memory controllers remains an open research challenge. Current work addresses components individually: HiFC optimizes flash-based KV swapping latency; FASER optimizes speculative phase prediction; DualPath decouples prefill and decode bandwidth consumption.
However, synthesizing these techniques into a single memory controller architecture that simultaneously: adapts scheduling policies based on anticipated quantization formats, maintains KV-Cache locality across chiplet boundaries, provides priority guarantees for real-time tokens, and exploits heterogeneous memory tiers, requires architectural innovation beyond current implementations.
Memory bandwidth arbitration in multi-chiplet GPU token decoding demands controllers that transcend traditional FIFO or weighted fair queuing approaches. Future designs must integrate speculative prediction, KV-Cache-aware routing, format-adaptive scheduling, and quantization-aware prefetching into coherent priority arbitration frameworks. Achieving this while maintaining real-time latency guarantees and power efficiency across heterogeneous memory hierarchies represents the critical challenge for next-generation LLM inference accelerators.