Apto: Self-Adaptive CPU-GPU Hybrid MoE Inference on Shared PCs

Abstract

CPU/GPU hybrid inference enables Mixture-of-Experts (MoE) models to run on memory-constrained PCs by keeping all expert weights in host memory and executing activated experts across both processors. Existing hybrid systems, however, assume exclusive resource access and determine an upfront CPU/GPU partition, so CPU contention from colocated applications can severely imbalance the two execution paths. This paper presents Apto, a self-adaptive hybrid inference system that applies dynamic scheduling to expert execution. The CPU and GPU executors claim experts from opposite ends of a dual-ended queue as they progress, allowing the CPU/GPU work split to adapt to their execution progress without predicting contention. Apto further uses affinity-aware scheduling to expose each executor's preferred experts at its queue end and cancellable tail racing to mitigate final-expert stragglers. Compared with state-of-the-art GPU-only and hybrid baselines, Apto achieves average prefill and decode speedups of 2.96$\times$ and 1.52$\times$ under standalone execution, and 5.97$\times$ and 3.45$\times$ under concurrent CPU workloads. It also accelerates real-world coding-agent applications by 2.02$\times$ on average.