Fan-Out in Systems Using the Jasper Forest CPU
Layout without Jasper Forest
Layout with Jasper Forest
Intel recently launched Xeon C5500/C3500 "Jasper Forest," its latest CPU. Targeted at the embedded, communication, and storage markets, Jasper Forest will offer 16 PCI Express 2.0 (PCIe Gen 2) lanes and one non-transparent (NT) port. Traditionally, on-board PCIe switches have been used for expanding the number of PCIe lanes as well as providing NT bridging functionality. With 16 lanes and an NT port integrated into the Jasper Forest CPU, a designer may think the need for on-board PCIe switches may be obviated in Jasper Forest systems. However, this often is not the case.
Until now, the NT functionality was only available via a PCIe switch. Jasper Forest’s NT port will be configurable as either a x4 or x8 port and will support PCI’s NT bridging functionality for host isolation. In a PCIe hierarchy, there can be only one host for each endpoint. For multi-host architectures with multiple CPUs, a NT port is needed to isolate the host hierarchies and ensure there is no contention over the endpoints. The NT port allows systems to isolate host memory domains by presenting the processor subsystem as an endpoint rather than another memory system. Base address registers are used to translate addresses; doorbell registers are used to send interrupts between the address domains; and scratchpad registers, accessible by both CPUs, allow inter-processor communication.
Although Jasper Forest addresses the need for an NT port, the need for a PCIe switch remains. Jasper Forest’s 16 lanes can be grouped into four (x4x4x4x4), three (x8x4x4) or two (x8x8) ports. In many applications, this is already an insufficient amount of PCIe lanes and ports. Moreover, if one of these ports is used as an NT port, the need for more lanes – and therefore a switch – is intensified. CPUs’ Evolution in PCI Express Systems
In a traditional PCIe backplane usage model, one large PCIe switch (see Figure 1) is used on the switch fabric to fan out to all the processor and I/O blades. Since the processor blades have CPUs on-board, a smaller PCIe switch with an NT port is used to isolate the CPU domain on each processor blade. A smaller eight-lane switch is typically used as a simple NT buffer, utilizing a x4 link on both the upstream and downstream sides of the switch.
In the past, the CPU would connect to a northbridge, which would connect to the memory and graphics display port. Furthermore, the southbridge would serve as the I/O hub providing all the necessary I/O ports. Hence, providing a large number of PCIe ports was not required by CPUs.
But going forward, the northbridge has disappeared and the southbridge only offers a limited number of PCIe lanes. As a result, the burden of providing the necessary PCIe ports to connect to memory, graphics and I/Os falls on the CPU.
Moreover, the need for I/Os on a system only continues to grow. As designers try to differentiate their products and create specialized systems targeting niche market segments (e.g. high-end storage, super computing), today’s systems are often pumped full of endpoints and support numerous interfaces.
As a result, even though Jasper Forest’s integrated NT port will reduce the need for switches that provide host isolation, it will create a need for additional PCIe ports. Using one of Jasper Forest’s four PCIe ports (or two PCIe ports if using x8 links) as an NT port does not leave enough lanes on the CPU to fan out to other endpoints such as memory, ASICs, or other I/O (SATA drives, NICs, HBAs).
Figure 2 shows an example where a second switch (24 lanes) is used to fan out from the Jasper Forest CPU to its endpoints. In this example, one of the CPUs ports is being used as an NT port (could be x8 or x4) and the other PCIe port is used as a x8 link to the PCIe switch. The switch then fans out to four endpoints via x4 links (could be a mix of x8s and x4s as well). A larger (e.g. 32-lane) or smaller (e.g. 16-lane) switch can also be used, depending on how many I/Os need to be connected to the CPU.
In the backplane (or I/O module), a 96-lane switch is still used for fan-out purposes to provide a connection between all the CPU blades. Ninety-six-lane switches usually also support a large number of ports (e.g. 24) and therefore can support a large number of I/Os (capable of fanning out to 24 blades/modules via x4 links) and is ideal for fan-out applications.
Although Jasper Forest’s integrated NT port may eliminate the need for PCIe switches in some basic applications, in many cases it will create the need for additional fan-out via on-board PCIe switches. Furthermore, since the Jasper Forest’s integrated NT port can only support a maximum link-width of x8, applications requiring a x16 NT link will still require a PCIe switch. Finally, PCIe switches from providers such as PLX Technologyy – the only switch vendor with Gen 2 switches with an integrated NT port, configurable as x1, x2, x4, x8, or x16 -- are loaded with features targeted at enhancing overall system performance, as well as debug capabilities, thus offering more than just fan-out functionality.