CXL Fabric Switching & Multi-Host Memory Pooling on Bare-Metal Servers

Traditional enterprise datacenter architecture binds physical dynamic random-access memory (DRAM) strictly to CPU sockets on individual server motherboards. In large-scale cloud hosting, in-memory analytics (such as Redis, Apache Spark, and SAP HANA), and AI inference deployments across Pakistan, this creates severe resource strandedness: one server starves for RAM while neighboring hypervisors leave 40% of their DRAM unallocated.

The advent of Compute Express Link (CXL 3.0 and CXL 3.1) changes this paradigm by introducing CXL Fabric Switching. By decoupling memory from compute and placing terabytes of low-latency DDR5 memory into a shared, switchable fabric, multiple independent bare-metal hosts on a Dedicated Server in Pakistan can dynamically borrow, pool, and release hardware memory segments over high-speed PCIe Gen5/Gen6 physical links with cache-coherent microsecond latencies.

Architectural Evolution: Direct-Attached CXL vs. CXL Fabric Switching

+-----------------------------------------------------------------------------------+
|                            CXL 3.0 Multi-Host Fabric                              |
|                                                                                   |
|  +--------------------+     +--------------------+     +--------------------+     |
|  |   Host Node 1      |     |   Host Node 2      |     |   Host Node 3      |     |
|  | (Xeon / EPYC CPU)  |     | (Xeon / EPYC CPU)  |     | (Xeon / EPYC CPU)  |     |
|  +---------+----------+     +---------+----------+     +---------+----------+     |
|            |                          |                          |                |
|            +-------------------+      |      +-------------------+                |
|                                |      |      |                                    |
|                     +----------v------v------v----------+                         |
|                     |     CXL 3.0 Multi-Port Fabric     |                         |
|                     |        Switch (Crossbar)          |                         |
|                     +----------+--------------+---------+                         |
|                                |              |                                   |
|            +-------------------+              +-------------------+               |
|            |                                                      |               |
|  +---------v-------------------------+          +-----------------v------------+  |
|  |    Multi-Logical Device (MLD)     |          |  Shared Dynamic Coherent     |  |
|  |    Pooled Memory Drawer (16TB)    |          |  Memory Pool (Low Latency)   |  |
|  +-----------------------------------+          +------------------------------+  |
+-----------------------------------------------------------------------------------+

1. CXL 1.1 / 2.0 (Single-Host & Simple Switching):

   • Direct-attached CXL Type 3 memory expansion boards inside a single PCIe slot.
   • Limited to point-to-point topologies or single-tier switches. Memory could only be assigned statically at server boot.

2. CXL 3.0 / 3.1 (Multi-Host Fabric Mesh):

   • Multi-Logical Devices (MLD): A single memory pool drawer can partition its physical capacity into up to 16 virtual memory devices, presenting separate cache-coherent segments to different physical servers simultaneously.
   • Back-Invalidation (BI) & Peer-to-Peer: Allows CPU hosts to access pooled memory without routing transactions through the originating root complex, dropping cross-chassis latency to under 150 nanoseconds.
   • Non-blocking Crossbar Fabrics: Replaces rigid PCIe point-to-point links with spine-leaf fabric switches capable of routing 64 GT/s per lane across PCIe Gen6 PAM4 signaling.

Linux Kernel Memory Tiering & Fabric Device Enumeration

In modern enterprise Linux kernels (6.6+ and 6.10+), the kernel discovers fabric-switched CXL devices through the CXL Subsystem (drivers/cxl) and the ACPI CEDT (CXL Early Discovery Table).

To verify detected CXL fabric switch endpoints on a Dedicated Server:

# Query active CXL topologies and fabric endpoints
cxl list -E -u

Diagnostic Output:

[
  {
    "endpoint":"endpoint2",
    "host":"mem0",
    "port":"port2",
    "decoder":"decoder2.0",
    "ranges": [
      {
        "start": "0x40000000000",
        "end": "0x43fffffffff",
        "size": "274877906944"
      }
    ]
  }
]

Here, mem0 is an external 256 GB pooled DRAM segment assigned to this specific server host by the upstream fabric controller.

Dynamic Memory Allocation via the CXL Fabric Manager

In an enterprise CXL datacenter fabric, memory is assigned to hosts dynamically using the Fabric Manager (FM) API via MCTP (Management Component Transport Protocol) over I2C or PCIe VDM:

# Instruct the CXL Fabric Switch to allocate a 512GB LD (Logical Device) slice to Host Port 3
cxl-fabric-cli assign-memory \
    --switch-id=switch-01 \
    --target-host-port=port-03 \
    --mld-pool=pool-alpha \
    --size=512G

Once assigned, the host server receives an ACPI hot-plug event (via native CXL driver notification). The Linux kernel immediately binds the new memory segment:

# Check NUMA node creation for pooled memory
numactl --hardware

NUMA Topology Output:

available: 2 nodes (0-1)
node 0 cpus: 0-63
node 0 size: 130842 MB
node 0 free: 120150 MB
node 1 cpus: none               <-- CPU-less Pooled CXL Fabric Node!
node 1 size: 524288 MB          <-- 512GB Allocated from Central Fabric Drawer
node 1 free: 524288 MB
node distances:
node   0   1 
  0:  10  22 
  1:  22  10 

Node 0 represents local direct-attached DDR5 DIMMs (distance 10, ~65ns). Node 1 represents pooled fabric memory (distance 22, ~160ns).

Application Binding for High-Throughput In-Memory Databases

For applications like Valkey, Redis, or Apache Cassandra, you can bind cold cache buffers to pooled CXL fabric memory while keeping hot query execution threads on local CPU-attached memory:

# Launch in-memory cache binding memory allocations primarily to CXL Node 1
numactl --membind=1 --cpunodebind=0 /usr/bin/valkey-server /etc/valkey/valkey.conf

Alternatively, enable Linux automated tiering via autonuma:

echo 1 > /proc/sys/kernel/numa_balancing

The kernel’s numa_balancing engine monitors page access frequency, transparently demoting cold pages from Node 0 to CXL Node 1 and migrating hot working sets back into local DIMMs.

Architectural Comparison: Datacenter Memory Expansion Models

For further technical insights into PCIe hardware architectures and low-level subsystem performance, explore our deep dives on CXL Memory Tiering with Linux memtier and PCIe Hot-Plug via ACPI (pcihp).