Effective NUMA-aware token cache management for llama.cpp inference requires coordinating memory placement strategies with cache eviction policies to minimize cross-socket coherency traffic. While NUMA architectures inherently suffer from non-uniform access latencies, topology-aware allocation combined with adaptive eviction mechanisms can reduce memory bandwidth variability and improve sustained decoding throughput in multi-socket CPU systems.
Multi-socket CPU inference presents distinct challenges for language model serving, particularly in managing key-value (KV) caches across Non-Uniform Memory Access (NUMA) architectures. The llama.cpp inference engine operates on commodity hardware where memory access latency varies significantly based on socket affinity, creating opportunities for optimization through intelligent topology awareness and cache eviction strategies. This report examines the intersection of NUMA-aware memory placement and adaptive token cache eviction mechanisms for sustained decoding workloads.
NUMA systems fundamentally differ from Uniform Memory Access (UMA) architectures in that processors experience dramatically different access latencies depending on memory locality [2]. Each NUMA node contains local memory directly attached to its socket, providing fast access, while remote memory requires traversal of inter-socket links [1]. This architectural reality creates a performance gradient: accessing local memory is substantially faster than accessing memory local to another processor or shared memory regions [2].
The challenge intensifies when multiple copies of memory blocks exist across cache hierarchies in shared memory architectures [3]. Cache coherency protocols must manage consistent views across distributed caches, adding complexity and latency overhead [4]. For inference workloads that generate substantial KV cache data during token generation, this coherency traffic directly impacts sustained throughput and latency characteristics.
The KV cache is fundamental to efficient LLM inference, storing mathematical representations of previous tokens to avoid recomputation during autoregressive decoding [12]. Without caching, each new token generation would require reprocessing the entire context, making long-conversation inference prohibitively expensive. However, KV caches grow linearly with sequence length, consuming substantial memory in multi-turn conversations [16].
Token eviction mechanisms provide algorithmic strategies to bound KV cache growth [6]. These approaches selectively discard less valuable tokens while maintaining inference quality. The effectiveness of eviction depends critically on access pattern characteristics. Least-Recently Used (LRU) policies evict items not accessed for the longest duration, performing well when access exhibits temporal locality [7][9]. More sophisticated approaches like Token-PD achieve portfolio-optimal eviction, reducing peak memory by up to 60% and increasing throughput by 1.3× on multi-tenant traces [8].
Optimal cache performance in NUMA systems requires co-locating frequently accessed data with computing threads [1]. When KV cache data is placed on the same NUMA node as the socket executing attention computations, access latency remains low. Conversely, accessing remote KV cache data necessitates inter-socket hops, increasing both latency and bandwidth variability [1].
Hardware topology discovery becomes essential for implementing placement strategies. Tools like hwloc and Linux numactl utilities enable runtime discovery of NUMA topology [5], allowing inference engines to make informed allocation decisions. The granularity of placement decisions—whether at the page level, tensor level, or sequence level—determines how effectively systems can minimize remote accesses.
For llama.cpp specifically, feature requests indicate community recognition of NUMA optimization opportunities [13]. Expert allocation in Mixture-of-Experts models demonstrates that cached data dependency awareness can reduce cache miss penalties by 15-30% when experts are allocated to NUMA nodes containing frequently accessed data [13].
Cache coherency protocols enforce consistency across distributed caches when memory blocks are replicated [4]. In multi-socket NUMA systems, coherency traffic traverses inter-socket links, consuming precious bandwidth. For inference workloads with heavy KV cache access patterns, this coherency overhead can become substantial.
Minimizing cross-socket coherency involves strategic data placement to reduce the frequency of coherency messages. If a KV cache is placed on NUMA node 0 but primarily accessed by threads on NUMA node 1, every cache line update requires coherency communication. Conversely, placing cache data locally eliminates this overhead entirely.
Adaptive eviction policies can complement placement strategies by reducing total cache size. Smaller caches naturally generate less coherency traffic [10]. The interplay between eviction and placement becomes critical: aggressive eviction reduces memory footprint but may increase recomputation costs, while conservative placement maximizes local access but increases total bandwidth utilization.
Sustained decoding refers to maintaining consistent throughput and latency during long inference sessions. Memory bandwidth constraints often emerge as the primary bottleneck [16]. When NUMA nodes are forced to access remote KV cache repeatedly, effective bandwidth per socket decreases due to inter-socket link contention.
Optimizing for sustained decoding requires coordinating eviction policies with NUMA-aware placement. The decision to evict a token involves not just relevance metrics but also anticipated access patterns from the current NUMA node. An adaptive system might retain tokens that fit locally while evicting tokens whose predicted access frequency does not justify remote fetch costs [10].
The relationship between KV cache eviction and quantization provides additional perspective [10]. At certain scale thresholds, cache eviction becomes more efficient than quantization for reducing memory pressure. NUMA-aware systems might adjust this threshold based on available local memory, preferring eviction when local memory is constrained but quantization when bandwidth limitations dominate.
llama.cpp serves as the reference implementation for efficient CPU-based LLM inference [12]. The engine currently supports various optimization techniques but requires additional NUMA-awareness to fully exploit multi-socket architectures. KV cache handling, token generation scheduling, and memory allocation routines could be enhanced with topology-aware variants.
Storage hierarchies for KV caches present additional opportunities [17][18]. Rather than keeping all KV cache data in DRAM, sophisticated systems can offload less frequently accessed tokens to secondary storage or hierarchical memory systems. When combined with NUMA-awareness, such systems could maintain hot cache data locally while using remote memory or storage for colder data.
The emergence of new cache management architectures [20] demonstrates industry momentum toward specialized KV cache handling. These centralized management approaches reduce per-process overhead and enable sophisticated policies that coordinate across multiple inference requests.
Implementing NUMA-aware token cache eviction in llama.cpp introduces complexity. Hardware topology discovery adds initialization overhead. Adaptive eviction policies require overhead for measuring access patterns and predicting future token relevance. These overheads must be justified by throughput improvements.
The optimal eviction policy depends on workload characteristics. Interactive single-user inference with moderate sequence lengths may benefit differently from multi-turn conversations or multi-request batch processing. A practical system requires tuning parameters or adaptive mechanisms that adjust policies based on observed characteristics.
Coherent prefetching presents another consideration. If a token will soon be accessed from a remote socket, proactive prefetching might amortize the latency cost. However, prefetching incorrectly wastes bandwidth, so prediction accuracy becomes critical.
Adaptive token cache eviction combined with NUMA-aware memory placement offers meaningful optimization potential for llama.cpp inference on multi-socket systems. The core insight is that memory locality directly determines sustained decoding throughput: local accesses are fast and consume minimal bandwidth, while remote accesses create latency spikes and compete for limited inter-socket bandwidth. By coordinating eviction decisions with NUMA topology and carefully managing cache data placement, systems can substantially reduce coherency traffic and memory bandwidth variability. Implementation requires topology discovery mechanisms, adaptive eviction policies cognizant of access patterns, and integration within the llama.cpp execution model. The potential benefits—improved latency consistency, higher effective bandwidth, and better support for long-context inference—justify the architectural complexity.