Start the coffee. Embedded designers are in for the long haul when it comes to implementing PCI Express, but the rewards are worth it.
Parallel PCI and PCI-X, hampered by its wide bus that makes layout difficult, no longer is the speed champ. Winning with a technical knockout of performance and new chips is... PCI Express. But will dealing with this high-speed serial technology be easier or harder?
PCI Express looks like the holy grail (Fig. 1). Compared to PCI/PCI-X, it is fast. It uses fewer pins, has more features, and offers backward-compatibility with PCI/PCI-X software.
So what's the problem? Generally, many designers are simply ill-prepared for PCI Express' quantum leap in board and system design technology. Embedded designers will find that its 2.5-GHz speed is a major problem—it's almost two orders of magnitude different from what they're accustomed to. (PCI tops out at 133 MHz.) In addition, routing and power issues persist. Furthermore, analog designers will have to meet new, more restrictive requirements.
This leaves two alternatives: Buy something, or bite the bullet and learn how to design to the new standard. The former is what most PCI Express systems employ now. Motherboards feature PCI Express sockets. And PCI Express cards are readily available, though the number of choices is limited but growing. Thus, system designers can simply select products, plug a system together, and get a working final product. Such support also will be needed for Advanced Switching, PCI Express' sibling (see "Advanced Switching Waits In The Wings," p. 60).
The other alternative is the more arduous route—engineers designing boards to plug into a PCI Express motherboard or designing custom embedded systems. To alleviate part of the problem, one could use modules like those based on COM (computer on module) Express. Kontron, PFU Systems, and Radisys are three companies that provide COM Express modules complete with processors, Gigabit Ethernet, and, of course, PCI Express (Fig. 2).
Next, developers must build a carrier board that links the modules to PCI Express devices on the carrier board, thereby cutting the design problem in two. Either way, a knowledge of PCI Express is needed, which means comprehending very high-speed analog design.
Power and clock distribution, less critical aspects of PCI and PCI-X system design, have become quite significant. Systems that could be built on four-layer boards now require eight-layer boards with multiple ground and power planes.
The PCI Special Interest Group (SIG) is responsible for the PCI Express specification. The SIG also is a source of information for PCI Express designs. (Some information is available only to members). Specifications address all aspects of PCI Express, from the physical design (even including connectors and cabling details) to the wire protocol. On top of that, PCI Express design recommendations can be beneficial to designers.
Design-related documents that cover jitter modeling and bit error rate (BER) also come in handy. The amount of jitter determines how well devices will interoperate. PCI depends more on clock skew, while PCI Express is affected more by jitter because the clock is embedded in the data (Fig. 3).
PCI Express utilizes a 100-MHz clock and generates the 2.5-GHz serial stream via a phase-lock loop (PLL) (Fig. 4). The quality of the serializer/deserializer (SERDES) is key to a PCI Express system's success. However, it will only be as good as the transmission line between the transmitter and receiver. Tolerances on the circuit board between two PCI Express devices are critical.
Tom Tinory, StarGen's director of Board Product Development, notes that the design of power planes supporting PCI Express devices can be tricky and complex. Regulated power must be supplied equally to these devices for the SERDES, receiver termination, and the analog voltage when driving the PLL. It may be possible to "carve" these power-plane splits onto a single power plane. But in many cases, there's a need for two or three planes.
PCI Express electrical design rules require high-speed differential signals to be surrounded by ground above and below. Combining this rule with the complexity of power-plane splits drives up the number of layers required for electrically clean and compliant designs.
PCI EXPRESS SERDES High-speed SERDES designs aren't new, but they have been found in devices like FPGAs from Altera and Xilinx. They also show up in Gigabit Ethernet designs. This gives companies working in these areas a head start in dealing with the SERDES design complexities."Designers need to do their homework before they start," says Jean-Marc Patenaude, director of Logic Platform Solutions for Rambus. "They should not assume they can do it all themselves."
Patenaude knows about supporting the client and about high-speed SERDES. Rambus developed a seven-step support process along with signal-integrity analysis software. The company's SERDES design incorporates more testability features, such as ac JTAG testing, compared to the standard JTAG support. The ac tests can check on bit-error-rate (BER) eye diagram data.
Rambus isn't alone in this effort. Companies such as Mentor Graphics and Synopsys provide similar services. Even companies like LSI Logic are getting involved in their clients' board designs.
LSI Logic retains a good deal of SERDES expertise from its Fibre Channel work. Serial ATA (SATA) and Serial Attached SCSI (SAS) are two new, high-speed serial interfaces that LSI Logic is melding with PCI Express. Harmel Sangha, LSI Logic's director of CoreWareIP Marketing, noted that the company also created its own package model and Spice interface design tools to check signal integrity in high-speed designs. This is important in an environment that may have as many as 64 SERDES.
BRIDGES AND SWITCHES PCI Express systems are built from four components: masters, end-nodes, bridges, and switches (Fig. 5). PCI/PCI-X does not include switches. Masters are typically processors or processor support chip sets. End-nodes consist of devices like Ethernet interfaces. Bridges provide links between PCI Express and other technologies (e.g., PCI). And switches link things together, including other switches. (Although in many cases, a single switch is all that's required.) Many processor chip sets also incorporate a switch.Companies like PLX Technology and IDT provide a range of bridges and switches. PLX Technology developed a PCI Express-to-PCI bridge, the PEX 8111, and a PCI Express-to-PCI/PCI-X bridge, the PEX 8114 (Fig. 6). IDT's family includes the PEB20N1 and PEB20N2, which support PCI-X 2.0 Mode 1 and 2. These bridges will be used primarily to link legacy designs to PCI Express. Routing for a single lane is much easier than multilane designs, as is keeping the bridge chip close to the connector or switch that supports it.
Both IDT's and PLX Technology's switches support nontransparent bridging. The bridge then can be used as a master in multimaster applications such as cluster servers.
IDT's PES12N3 and PES24N3 are 12- and 24-lane dual-port switches. The port specifications translate to three ports with up to x8 lanes per port. PCI Express can autonegotiate all the way down to a x1 port.
PLX Technology's parts include the PEX 8516 and PEX 8532, with 16 and 32 lanes, respectively. The PEX 8516 supports up to 16 lanes and can have up to eight downstream ports where each can autonegotiate down to one lane if necessary. These chips usually wind up in higher-end systems with wide, multilane connections, which tend to be more difficult to lay out.
While designing with PCI Express can be more difficult, there are some benefits other than high performance and hot-plug capability that will keep you in the game. For example, unused lanes needn't be terminated, because the SERDES for these lanes are turned off. This means the pins won't be used. This also aids in reducing power consumption. Autonegotiation can determine the number of lanes used.
Switch chips typically come in a high-density, ball-grid-array (BGA) package. One question concerning this packaging is the ability to cleanly escape the BGA fields with differential signals. The small-geometry package leads to a smaller system footprint. But the typical 1-mm pitch BGA is very difficult to route when it comes to differential pairs. This is particularly true with PCI Express, which requires a 100-Ω differential impedance. Most chip designs try to use the outer BGA pins for the PCI Express signals. Yet this isn't always an easy task when there's a large number of overall lanes, as is the case with switch chips.
GETTING TESTY Two new types of test systems will be needed for PCI Express—one to address the analog aspect of the design, and the other to address the protocol. Agilent and Tektronix already offer tools in these realms.Tektronix addresses signal testing with tools like its Data Timing Generators and RT-Eye Serial Data Compliance and Analysis Software. Many of these tools will be more familiar to RF designers, but now they must be incorporated into the typical digital board designer's toolkit.
Agilent's E2960A Protocol Analyzer and Protocol Exerciser for PCI Express looks at the interface at a higher level. Unfortunately, it means a design must incorporate the appropriate test points so that cabling can be attached. This becomes less of an issue when working with prototypes, though.
Still, even adding test points can affect the functionality of a PCI Express link. Therefore, it pays to do the testing as well as have in-house or solicited support for the design. Luckily, most designs will deal with PCI Express chips from vendors who already tested the protocol support. PCI-SIG regularly has "plugfests," in which chip and system vendors can demonstrate interoperability.
Tools like Altium's Protel 2004, Cadence's Allegro system, and others for pc-board layout already target high-speed differential designs. However, with PCI Express, the tools will require further tuning. For example, some PCI Express chips will autonegotiate pins as well as speeds. In some cases, the transmit pairs of two chips could be connected together. Then the chips would switch one set to the SERDES receiver within the chip with autonegotiation support.
It's a great feature, but pc-board layout tools can't handle it well when it comes to autorouting. Essentially, pin 1 and 2 on chip A can be connected to pins 5 and 8 on chip B, where the choice is left to the layout program or designer. It also complicates schematic capture. For now, pc-board layout programs will continue to have fixed pin usage.
High-speed test and analysis tools are embedded within pc-board design products, but many digital designers may be unfamiliar with these features. It's time to dust off the manuals and do some reading before diving into PCI Express designs.
Basically, the bottom line for developers is this: Get to know the design requirements and follow them religiously. High-speed serial systems are very unforgiving in some areas and extremely flexible in others. The differences in design, implementation, and testing are unlike anything encountered with a PCI design. Likewise, the performance jumps from ISA to PCI to PCI-X are trivial compared to the jump to PCI Express.
PCI Express board and system design will simplify as time goes on. Tools will improve to meet designer needs, and knowledge about PCI Express' idiosyncrasies as well as its design implementation will steadily expand. For now, most developers are crouched at the starting line. The benefits of PCI Express are significant, and it's the trend for new system designs, so don't get left in the dust. Prepare now and seek assistance.
Compatibility: software compatible with PCI/PCI-X
Link speed: 2.5-Gbit/s, full-duplex, 4-wire
Links: x1, x2, x4, x8, x16, x32
Signaling: low-voltage differential signaling (LVDS)
Clock: 8b/10b encoding scheme
Features: hot-swap, low latency, virtual channels, quality of service (QoS)
Advanced features: nontransparent bridging allows for multiple root devices
www.amd.com
Agilent
www.agilent.com
Altera
www.altera.com
Altium
www.altium.com
ASI SIG
www.asi-sig.org
Cadence
www.cadence.com
Elma Bustronic
www.elma.com
ExpressCard
www.expresscard.org
Integrated Device Technology
www.idt.com
Intel
www.intel.com
Kontron
www.kontron.com
LSI Logic
www.lsilogic.com
Mentor Graphics
www.mentor.com
PCMCIA
www.pcmcia.com
PCI Industrial Computer Manufacturers Group
www.picmg.org
PCI-SIG
www.pcisig.com
Pentair Enclosures
www.pentairenclosures.com
PLX Technology
www.plxtech.com
PFU Systems
www.pfusystems.com
Radisys
www.radisys.com
Rambus
www.rambus.com
Synopsys
www.synopsys.com
Stargen
www.stargen.com
Tektronix
www.tek.com
Xilinx
www.xilinx.com
