Cross-layer cache coherency for disaggregated AI inference requires hardware-software co-design integrating network-aware coherence protocols with distributed token generation strategies. Contemporary approaches leverage asymmetric cache coherence models and prefill-decode disaggregation to balance consistency guarantees against latency and bandwidth constraints, though fundamental tradeoffs between strong consistency and distributed efficiency remain unresolved.
Disaggregated AI inference architectures—where computation spans multiple sockets, accelerators, and networked edge devices—present fundamental challenges for maintaining cache coherency while preserving the performance characteristics required for real-time token generation. Unlike traditional monolithic systems where cache coherence protocols [2] create the illusion of a unified memory space, disaggregated topologies must reconcile the physics of distributed systems with the algorithmic requirements of modern large language models (LLMs).
Disaggregated LLM serving separates prefill and decode phases across heterogeneous hardware to optimize throughput [12]. During prefill, input tokens are processed in parallel; during decode, a single token is generated sequentially. This architectural separation naturally distributes the KV-Cache—the key-value pairs cached during computation—across multiple memory tiers. However, disaggregation introduces a critical bottleneck: efficiently loading massive KV-Cache data from distributed caching tiers becomes a fundamental performance limiter [11].
The coherency challenge intensifies because these caches exist in heterogeneous address spaces. Traditional cache coherence protocols [3] coordinate updates across symmetric multi-socket systems where coherence remains largely transparent. In disaggregated systems, transparency breaks down; coherency becomes an explicit optimization target rather than an implicit guarantee [4].
Addressing this challenge requires integrated hardware-software strategies. Contemporary edge-AI systems achieve real-time processing by co-designing hardware accelerators with software optimization frameworks [6] [8]. For AI inference specifically, this manifests as token pruning and selective caching strategies paired with specialized compute units [7].
The asymmetric cache coherence model emerging from industrial practice [16] [20] preserves the host-device relationship between CPUs and accelerators by allowing both sides to implement caching agents. Unlike symmetric multi-socket protocols [18] that coordinate via distributed coherence directories, asymmetric protocols reduce coherence traffic by accepting weaker guarantees in accelerator-to-accelerator communication while maintaining strong host-device consistency.
Network topology profoundly influences coherence efficiency. Between-socket communication introduces latencies orders of magnitude higher than within-socket access, making naive broadcast-based coherence prohibitively expensive [1]. Cache-aware prefill-decode disaggregation (CPD) addresses this by routing computation such that coherence traffic aligns with network structure [13]. Requests for KV-Cache data preferentially traverse low-latency paths; requests that would traverse long-latency inter-device links are rerouted to local replicas or reformatted to exploit spatial locality.
Adaptive coherence protocols reduce network traffic through migratory policies [4], where data migrates toward consumers rather than maintaining centralized directory entries. This approach proved effective in traditional COMA (Cache-Only Memory Architecture) systems; disaggregated inference systems apply similar principles at the cross-device level.
However, fundamental constraints cannot be engineered away. The CAP theorem's distributed systems analog—that systems cannot simultaneously achieve low-latency writes, regional survivability, and strong consistency—applies directly [9]. In disaggregated AI inference, this manifests as a tradeoff between:
1. Strong cache consistency: Guaranteeing all caches see identical data, requiring synchronization overhead that inflates token generation latency
2. Latency optimization: Relaxing consistency to enable speculative execution and asynchronous cache updates
3. Availability under failure: Replicating KV-Cache data across accelerators, increasing storage and coherence complexity
Prefill-decode disaggregation exploits this tradeoff asymmetrically. During prefill—the latency-insensitive phase—stronger consistency is tolerable because throughput dominates. During decode—the latency-critical phase—weaker consistency suffices because each token generation is independent; different devices can generate tokens from slightly stale KV-Cache copies without correctness degradation [15].
Recent multi-accelerator edge systems leverage unified memory to sidestep some coherence challenges [19]. When accelerators share physical memory via unified addressing, hardware coherence mechanisms handle consistency transparently. However, this approach trades flexibility for efficiency; systems become tightly coupled, limiting disaggregation across geographically distributed edge nodes.
Intel's CXL standard [20] represents a middle ground, implementing asymmetric coherence via PCI-Express 5.0 transport. CXL enables accelerators to cache host memory while maintaining eventual consistency, reducing round-trip latencies compared to pure remote access while avoiding the overhead of maintaining perfect synchronization.
For token generation specifically, coherence challenges manifest during the decode phase. Multiple edge accelerators may maintain replicas of the same KV-Cache rows. When one accelerator generates a new token and appends its contribution to the KV-Cache, other replicas must be updated. Naive broadcast invalidation would serialize token generation; instead, systems must employ lazy consistency where replicas acknowledge updates asynchronously and subsequent token generations incorporate causally-dependent KV-Cache versions [15].
This requires application-level awareness of coherence semantics. LLM serving frameworks must explicitly manage which accelerators read from which KV-Cache replicas, transforming coherence from a transparent system property into a first-class architectural concern.
Several critical challenges remain unresolved. First, optimal coherence protocol selection across heterogeneous devices—CPUs, GPUs, TPUs, and neuromorphic accelerators—lacks principled design methodology [5]. Second, the interaction between coherence protocols and emerging LLM techniques like speculative decoding and tree search remains largely unexplored. Third, coherence for hierarchical KV-Cache layouts (main cache, auxiliary cache, disk spill) demands protocols spanning multiple consistency models simultaneously [11].
Further, the extensible nature of coherence toolkits [16] suggests that no single protocol will dominate; instead, future systems may select coherence strategies dynamically based on runtime inference patterns.
Cross-layer cache coherency for disaggregated AI inference demands rejecting traditional assumptions of transparent, system-wide consistency. Instead, effective designs accept heterogeneous coherence semantics, optimizing for specific inference phases and network topologies while maintaining application correctness through explicit synchronization at language-model boundaries. Hardware-software co-design—integrating asymmetric coherence protocols, network-aware scheduling, and token-generation-aware cache management—represents the practical path forward, though fundamental tradeoffs between consistency, latency, and availability ensure no universal solution exists [9].