PCI Express 6.0 and CXL 2.0 are designed to transform servers

The industry association PCI SIG has approved the specification for PCI Express 6.0 in the final version 1.0. It is particularly important for servers and their components, especially in combination with Compute Express Link (CXL).

PCIe 6.0 doubles – as is usual with PCIe generation changes – the data transfer rate compared to PCIe 5.0 to 64 gigatransfers per second, so that a single lane (PCIe 6.0 x1) transmits up to 8 GB/s in each direction. Current PCIe 4.0 SSDs require four lanes for this.

PCIe super fast

An x16 slot achieves 128 GB/s per direction with PCIe 6.0. In order for DMA transfers to be possible at this speed, you need four DDR5-4800 memory channels, which together transfer more than 150 GB/s.

In other words: At 32 GB/s, PCIe 6.0 x4 is faster than a DDR4-3200 memory channel with 64 data signal lines. This enables new architectures such as disaggregated servers, which can be flexibly interconnected from resource pools depending on the computing requirements.

Samsung is already planning CXL memory modules for CXL 1.1 based on PCIe 5.0. The advanced capabilities of CXL 2.0 then make it easier to manage persistent storage such as Optane DC memory or other storage-class memory in pools accessible to multiple server nodes.

Faster PCIe also makes sense for future SSDs, for AI computing accelerators and for Ethernet adapters with 800 Gbit/s per port (800GE).

CXL could also accelerate the use of new concepts such as RAM with computing functions or computational storage, as well as SmartNICs or data processing units (DPU, IPU, AWS Nitro).

PAM4 modulation and FLITs

In order to achieve the extreme data transfer rates, PCIe 6.0 brings serious changes on the physical level compared to its predecessors. Four-stage pulse amplitude modulation (PAM4) is used as the modulation method, which now transmits two bits per transfer instead of one (0 or 1), i.e. works with four voltage levels. In order to compensate for the associated higher susceptibility to interference, Forward Error Correction (FEC) is used as error correction. This in turn requires data packets of the same length, so-called Flow Control Units (FLITs).

However, PCIe 6.0 versions remain backward compatible with older PCIe versions.

Server-Roadmap

For 2022, Intel has announced the fourth generation of the Xeon-SP as “Sapphire Rapids” with PCIe 5.0 and CXL 1.1. The Core i-12000 “Alder Lake” with PCIe 5.0 is already on the market for desktop PCs, but without CXL. The IBM Power10 also masters PCIe 5.0.

In order to process the high data transfer rates, future server processors will have more DRAM channels (AMD Genoa with Zen 4, DDR5 and PCIe 5.0 around 12 instead of 8) or fast HBM2E RAM (Intel Sapphire Rapids).

Intel has announced the next-but-one generation of server processors called Granite Rapids for 2023; whether it will bring PCIe 6.0 and CXL 2.0 is unknown.

The once-competing Gen-Z interconnect concept is now history. ARM, AMD, IBM and Nvidia have also been working on CXL since 2019, while maintaining their own coherent interconnects such as CCIX, InfinityFabric, OpenCAPI and NVLink.

(queue)