Building A CSFQ-Inspired Transport for Switched CXL Memory Pooling

Abstract

Emerging switched CXL memory pooling systems, albeit promising, suffer from significant performance interference due to the shared but performance-uncontrolled data path among concurrent memory streams between a host core and a remote DIMM. We systematically characterize a memory pooling appliance based on XConn's Apollo CXL switch and identify three issues: intra-host contention, in-fabric congestion, and unmanaged host-remote DIMM interaction. This paper presents a new transport layer, MemChannel, which provides the mchannel abstraction to manage end-to-end fabric bandwidth among competing memory flows and enable application-specific traffic for switched CXL memory pooling. Our key idea is to build a sender-driven, fabric-informed transport protocol, inspired by Core-Stateless Fair Queueing (CSFQ), that admits just the right amount of CXL requests to each mchannel based on the estimated core-to-CXL-DIMM bandwidth availability. To address CXL-induced idiosyncrasies, MemChannel introduces time-based rate control, host-side admission control, cross-host bookkeeping, new congestion signals, rate estimation based on the fluid model, and delay-based link-capacity adjustment. We build MemChannel from scratch and support unmodified applications. Evaluations over switched memory pooling demonstrate its effectiveness from performance-isolation, scalability, and multi-tenancy perspectives.