PCI Express 6.0 and CXL 2.0 are designed to transform servers

Share your love

The industry association PCI SIG has approved the specification for PCI Express 6.0 in the final version 1.0. It is particularly important for servers and their components, especially in combination with Compute Express Link (CXL).

PCIe 6.0 doubles – as is usual with PCIe generation changes – the data transfer rate compared to PCIe 5.0 to 64 gigatransfers per second, so that a single lane (PCIe 6.0 x1) transmits up to 8 GB/s in each direction. Current PCIe 4.0 SSDs require four lanes for this.

An x16 slot achieves 128 GB/s per direction with PCIe 6.0. In order for DMA transfers to be possible at this speed, you need four DDR5-4800 memory channels, which together transfer more than 150 GB/s.

Samsung CXL Memory Expander mit DDR5-RAM

(Image: Samsung)

In other words: At 32 GB/s, PCIe 6.0 x4 is faster than a DDR4-3200 memory channel with 64 data signal lines. This enables new architectures such as disaggregated servers, which can be flexibly interconnected from resource pools depending on the computing requirements.

Samsung is already planning CXL memory modules for CXL 1.1 based on PCIe 5.0. The advanced capabilities of CXL 2.0 then make it easier to manage persistent storage such as Optane DC memory or other storage-class memory in pools accessible to multiple server nodes.

Faster PCIe also makes sense for future SSDs, for AI computing accelerators and for Ethernet adapters with 800 Gbit/s per port (800GE).

CXL could also accelerate the use of new concepts such as RAM with computing functions or computational storage, as well as SmartNICs or data processing units (DPU, IPU, AWS Nitro).

Data transfer rates from PCIe and RAM
interface transfer rate

4 channels DDR5-4800

154 GByte/s

PCIe 6.0 x16

128 GByte/s

2 channels DDR5-4800

77 GByte/s

PCIe 6.0 x8

64 GByte/s

PCIe 5.0 x16

64 GByte/s

1 channel DDR5-4800

38 GByte/s

PCIe 6.0 x4

32 GByte/s

PCIe 5.0 x8

32 GByte/s

PCIe 4.0 x16

32 GByte/s

1 channel DDR4-3200

26 GByte/s

PCIe 6.0 x1

8 GByte/s

PCIe 4.0 x4

8 GByte/s

PCIe 3.0 x8

8 GByte/s

maximum transfer rate per direction

PAM4 modulation transfers 2 bits per transfer

(Image: Intel)

In order to achieve the extreme data transfer rates, PCIe 6.0 brings serious changes on the physical level compared to its predecessors. Four-stage pulse amplitude modulation (PAM4) is used as the modulation method, which now transmits two bits per transfer instead of one (0 or 1), i.e. works with four voltage levels. In order to compensate for the associated higher susceptibility to interference, Forward Error Correction (FEC) is used as error correction. This in turn requires data packets of the same length, so-called Flow Control Units (FLITs).

However, PCIe 6.0 versions remain backward compatible with older PCIe versions.

In March 2021, Intel CEO Pat Gelsinger announced the Xeon SP generation “Granite Rapids” for 2023.

(Image: Intel)

For 2022, Intel has announced the fourth generation of the Xeon-SP as “Sapphire Rapids” with PCIe 5.0 and CXL 1.1. The Core i-12000 “Alder Lake” with PCIe 5.0 is already on the market for desktop PCs, but without CXL. The IBM Power10 also masters PCIe 5.0.

In order to process the high data transfer rates, future server processors will have more DRAM channels (AMD Genoa with Zen 4, DDR5 and PCIe 5.0 around 12 instead of 8) or fast HBM2E RAM (Intel Sapphire Rapids).

Intel has announced the next-but-one generation of server processors called Granite Rapids for 2023; whether it will bring PCIe 6.0 and CXL 2.0 is unknown.

The once-competing Gen-Z interconnect concept is now history. ARM, AMD, IBM and Nvidia have also been working on CXL since 2019, while maintaining their own coherent interconnects such as CCIX, InfinityFabric, OpenCAPI and NVLink.

More from c't magazine

More from c't magazine

More from c't magazine

More from c't magazine


Article Source

Read Also   Today's Free Fire Codes September 2, 2021; all free rewards
Share your love