Dynamic power budget allocation and thermal throttling prediction for multi-socket CPU LLM inference requires integrated hardware performance counter (HPC) monitoring to detect memory-boundedness and cache pressure, coupled with frequency scaling and core parking strategies that balance P99 latency requirements against thermal constraints in high-density AI workloads now reaching 600 kW per rack.
Large language model (LLM) inference on multi-socket CPUs presents a complex thermal and power management challenge as data center AI workloads drive rack power densities toward 600 kW [12]. Dynamic power budget allocation requires real-time visibility into workload characteristics through hardware performance counters, enabling intelligent frequency scaling and core parking decisions that maintain sustained token generation rates while preventing thermal throttling.
Modern CPU performance monitoring units provide the foundational telemetry required for dynamic power management in LLM inference. Hardware performance counters (PMUs) enable characterization of workload memory-boundedness and cache behavior—critical inputs for power budgeting decisions [3][4]. Specifically, counters measuring cache miss rates, memory bandwidth utilization, and prefetcher activity reveal whether workloads are compute-bound or memory-bound [1][5].
For multi-socket LLM inference systems, counter collection complexity increases significantly. L2 hardware prefetchers exhibit dynamic behavior dependent on cache occupancy, and prefetcher activity from logical processors sharing cache hierarchies creates interdependencies that static power models cannot capture [1]. This necessitates continuous counter sampling at sub-second granularity to detect phase transitions in token generation workloads, which alternate between embedding computations (compute-intensive) and attention mechanisms (memory-intensive).
Detecting memory-boundedness through counters enables targeted throttling policies: memory-bound workloads benefit less from frequency increases and more from core parking to reduce contention, while compute-bound phases justify aggressive frequency scaling [4]. This differentiation is essential because naïve frequency scaling across all cores wastes power on workloads already starved by memory bandwidth.
Existing hardware mechanisms on commodity processors can control cache and bandwidth consumption across multiple cores [2]. Hardware execution throttling—implemented through instruction replay suppression, instruction fetch stalls, or cycle counting mechanisms—provides fine-grained power control without OS overhead. Unlike software-driven throttling, hardware mechanisms respond in single-cycle latencies to thermal or power budget violations.
For multi-socket LLM inference, coordinated throttling across sockets requires shared visibility into package-level power delivery and thermal sensors. On systems with multiple sockets, uneven workload distribution causes hotspot formation: one socket may be computation-intensive while another handles embedding lookups, creating asymmetric thermal profiles. Hardware throttling mechanisms must account for inter-socket power coupling through the system interconnect (QPI/UPI bandwidth).
Core parking strategies complement hardware throttling by reducing leakage power and memory bandwidth pressure. Selectively parking cores in token generation phases (where token-level parallelism is limited) reduces contention for shared L3 cache and memory controllers while maintaining throughput [20]. However, core parking latency (C-state entry/exit overhead) must be amortized over token batches; parking strategies should operate at batch boundaries rather than per-token granularity to avoid excessive wake-up overhead.
LLM inference SLOs center on two latency metrics: time-to-first-token (TTFT) and inter-token latency (ITL), with P99 latency (tail latency) representing the critical constraint for user experience [6][8][10]. P99 latency requirements—typically 100-500ms ITL—demand predictable performance even under thermal throttling [7][10]. This creates a tension: aggressive frequency scaling to reduce P99 latency increases heat generation, triggering thermal throttling that catastrophically violates ITL SLOs.
CPU-based LLM inference shows promise for latency-critical scenarios where embedding and computation phases can be pipelined across sockets, reducing reliance on high-bandwidth interconnects [17][19]. However, CPU thermal limitations are stricter than GPUs due to lower heat dissipation per watt of compute. Power density in multi-socket systems—particularly with 48-128 core SKUs—concentrates heat in small die areas, necessitating thermal prediction before throttling becomes necessary.
P99 latency budgets must explicitly account for throttling-induced latency variance [9]. If thermal throttling reduces frequency by 20% under peak load, ITL SLO budgets must reserve capacity for this degradation or implement predictive throttling that preemptively reduces frequency to maintain consistent token latency.
Reactive thermal throttling—where the system reduces frequency only after exceeding thermal limits—cannot satisfy strict P99 SLOs because recovery latency is non-deterministic. Proactive prediction requires thermal models fed by real-time counter data and junction temperature sensors [11][13]. Thermal interface material (TIM) conductivity and hotspot power distribution are primary sensitivity factors [11], meaning thermal lag is workload-dependent: sustained memory-bound workloads heat differently than bursty compute phases.
Dynamic power budget allocation should implement a control loop:
1. Sample hardware counters (L3 miss rate, memory bandwidth utilization, CPI) at 10-100ms intervals
2. Classify workload phase as memory-bound or compute-bound
3. Project thermal trajectory by convolving power model with thermal time constants (typically 0.5-2 seconds for multi-socket packages)
4. If projected peak junction temperature exceeds throttling threshold minus safety margin, reduce frequency or increase core parking rate
5. Monitor actual temperature response; adjust model coefficients to reduce prediction error
Memory-bound phases can tolerate lower frequencies (1-2 GHz reductions) with minimal ITL impact because memory latency dominates; compute-bound phases require higher frequencies but generate less sustained heat due to L3 cache hits and reduced memory bandwidth. This suggests frequency scaling strategies that follow workload phase classification rather than fixed P-states.
Multi-socket systems introduce additional complexity: per-socket power budgets must be coordinated to avoid exceeding system-level power delivery limits [14]. If one socket approaches thermal limit, reducing its frequency may increase workload migration to the other socket, redistributing heat rather than reducing total power. Coordinated power capping across sockets—using a global control loop that adjusts frequency targets for all sockets based on maximum die temperature—provides more stable thermal behavior [12].
Mixture-of-Experts (MoE) and other sparse LLM architectures benefit from intelligent socket placement: expert layers can be assigned to specific sockets, enabling per-socket power budgeting aligned with architectural hotspots [16]. This facilitates more aggressive frequency scaling on lightly-utilized sockets while protecting thermally-constrained sockets.
The primary implementation challenge is counter sampling overhead: reading dozens of PMU counters across multiple sockets at high frequency creates contention for the counter hardware and memory bandwidth [5]. Selective counter sampling—measuring only the 4-6 most predictive counters for memory-boundedness classification—reduces overhead but may miss phase transitions.
Frequency scaling granularity and latency also matter: P-state transitions typically require 1-10ms, creating transient latency spikes during transitions. Batching frequency adjustments to occur at LLM batch boundaries (every 10-50 token generations) amortizes transition overhead but reduces responsiveness to rapid workload changes. Constant frequency selection (per-batch rather than per-token) simplifies controller design and improves cache locality.
Core parking decisions must balance leakage power savings (0.1-0.5W per core) against wake-up latency (0.5-2ms). For typical token generation rates (50-200 tokens/second), this amortizes favorably, but very high throughput scenarios may require cores to remain unparked.
Sustained LLM token generation on multi-socket CPUs requires tightly integrated hardware performance monitoring, workload phase classification, and proactive thermal management. Counter-driven frequency scaling adapts to memory- versus compute-bound phases, while core parking reduces power under phase constraints. Thermal prediction prevents reactive throttling from violating P99 latency SLOs. Implementation requires careful engineering of counter sampling, frequency scaling latency, and multi-socket coordination, but addresses the fundamental challenge of delivering consistent sub-100ms inter-token latency while operating at the thermal edge of multi-socket systems now facing 600 kW rack power densities [12].