Check out our coverage of DesignCon 2024.
Connectivity is becoming a bottleneck in the age of AI. To unclog the interconnects between processors, accelerators, and memory, high-speed serial interfaces based on the PCIe Gen 6 bus are in the pipeline.
But testing these high-speed digital signals as they travel out of the IC package, over printed circuit boards (PCBs) and backplanes, and through the connectors, cables, or other interconnects in the system, is a tall order at the best of times. PCIe Gen 6 is also the first PCIe standard to use PAM4 signaling, giving it data rates of 64 gigatransfers per second (GT/s), double the speed of the PCIe Gen 5 bus. The more complex signaling and faster data rates mean that it’s more of a challenge to pinpoint the source of signal-integrity problems.
Teledyne LeCroy is lending a hand to companies at the forefront of high-speed connectivity with its latest “CrossSync PHY” technology. While it rolled out CrossSync PHY several years ago to connect its digital oscilloscopes and protocol analyzers to each other in real-time, the company is making upgrades to perform full physical-layer to protocol-layer analysis of high-speed serial interfaces based on PCIe Gen 6.
When testing high-speed interfaces at the physical layer (PHY), modern oscilloscopes are unmatched. They’re being upgraded with larger bandwidths, higher resolution, and more memory to capture the intricate details of high-speed signals based on PCIe and other forms of high-speed connectivity. Thus, they can more accurately locate potential problems related to signal integrity (SI) and power integrity (PI) and give you insights to reduce electromagnetic interference (EMI) that hurts high-speed digital systems.
But even the latest oscilloscopes can struggle to run the wide range of tests used to evaluate high-speed serial interfaces. They tend to be incapable of capturing protocol data transfers, because memory runs out before the protocol event occurs. So, it’s necessary to use a separate protocol analyzer to view protocol events that take place in the channel between devices and analyze them over time to figure out whether they can communicate with each other and to root out any problems.
Teledyne said it uses CrossSync PHY to connect the oscilloscope with a protocol analyzer in real-time and correlate the traces stemming from both systems, giving engineers deeper insights into high-speed serial links.
Is It the AI Era or the PCIe Era?
Today, PCIe is the most widely used high-speed serial interconnect bus in the data center because of its high bandwidth and low latency. PCIe is used inside the server itself to connect the CPU to the GPUs, to networking cards, AI accelerators, and memory and storage. It’s also frequently tasked with attaching CPUs to accelerators and memory sitting in the same column of servers via board-to-board connectors or cables.
As AI transforms the insides of the data center, the demand for high-speed serial connectivity is climbing. Thus, companies are upgrading to PCIe Gen 5, which doubles the data rate of PCIe Gen 4 from 16 GT/s to 32 GT/s, with plans for PCIe Gen 6 further in the future. The PCIe Gen 6 standard is unique in that it’s the first PCIe standard to use more advanced PAM4 signaling, giving it the ability to double the throughput of PCIe Gen 5, up to 64 GT/s at the same channel bandwidth, while being backwards compatible.
But the more complex signaling and faster data rates bring other problems into play. Specifically, these factors inevitably impact signal integrity due to losses that limit the distance a signal can travel through the IC package, the copper traces embedded inside the circuit board, and the connectors and cables weaving throughout the system, which all weaken the signal as it runs through them.
The complexity of the serial interface itself is also on the rise. That’s primarily due to dynamic link equalization, negotiation, training, and handshaking operations to handle faster data-transfer rates, said Teledyne. The latest chips using PCIe Gen 6 use transmit and receive equalization, with required forward error correction (FEC). Since FEC takes a toll on latency, PCIe Gen 6 adds flow control unit (FLIT) encoding to reduce it.
The oscilloscope is the primary tool to test and troubleshoot high-speed digital interfaces, such as PCIe, as signals travel over the PHY. Before building a full physical prototype to test and debug, engineers are also increasingly adopting software tools to simulate high-speed IC packages and PCB interconnects (along with EMI) and optimize the SI and PI of the devices.
The other part of the test and measurement process is protocol validation. Protocol analyzers help determine whether the microprocessor (MPU), accelerator, or other device can communicate safely with another device at the speeds required by the PCIe standard. Instead of probing specific points on a PCB, these tools are paired with interposers to grant access to live signals in the device being tested.
Testing from Physical Layer to Protocol Layer
As AI ramps up across the technology industry, physical- and protocol-layer validation of processors, accelerators, and memory that use PCIe for high-speed connectivity is becoming vital.
The signals traveling over the PCIe bus tend to be interrupted by electrical events or a protocol interaction, or a combination of both. But correlating events in the physical and protocol layers and figuring out the root cause of the problem poses a challenge.
By merging the oscilloscope’s functions with a protocol analyzer in real-time, CrossSync PHY grants the ability to view and evaluate traces from both tools at the same time, giving more insight into protocol-directed electrical behaviors. Capturing the entire protocol stack makes it possible to identify and resolve interoperability issues faster.
In between the oscilloscope and protocol analyzer is the company’s new Card Electromechanical (CEM) interposer. The device being tested is mounted to the interposers, which exposes connections for testing. The interposer adds points to probe for the reference clock, sideband signals, voltage and current levels in the power rail, and data signals for the protocol analyzer to identify and isolate connectivity issues in PCIe Gen 6 links. Teledyne said CrossSync PHY uses software to unite the system’s building blocks.
With the CrossSync PHY, engineers can use the navigation bar displayed on the front of the oscilloscope to clarify the timing relationship between signals captured by the protocol analyzer and oscilloscope.