The Internet has driven the bandwidth explosion in today's networks, challenging designers to keep pace with a seemingly endless thirst for bandwidth. Due to the massive amounts of data transmitted between the port cards and the core switching/routing function, the backplane of any switch/router can be the congestion point of the network. So as data rates and port counts rise, high-speed serial interconnects are migrating to the backplane. Serial-backplane architectures offer advantages in pin-count and noise reduction, hot-plug capability, and flexibility in design architecture. Serial signals can be routed on STP cable, copper traces, fiber-optic cable, or any combination of the three.
The design engineer faces new challenges, however, in the transition from the traditional parallel backplane to the high-performance serial backplane. With the ever-growing data rates of serial transmissions, the line between digital and analog signaling is blurred. Maintaining signal integrity requires paying increased attention to it in board design and component selection. Transmission-line effects, radiated signals, and jitter must all be understood and designed out. Carefully consider the critical issues of controlled-impedance traces and connectors, differential routing, termination techniques, and dc balance.
This article identifies critical serial-backplane design requirements, while pointing out some of the pitfalls commonly found in those designs. Also, various serial-backplane architectures are examined, and high-frequency design techniques are discussed. For engineers who want access to more detailed equations, references are included.
When calculating the bandwidth requirements for existing and future switches and routers, it becomes clear that the backplane's throughput must be increased. In traditional shared-media backplane architectures, like those found in mainframe computers and first-generation switches and routers, data can only be transmitted one card at a time. The remaining port cards in the switch are "blocked."
The same limitation exists in loop architectures. As port counts and bandwidth requirements increase, the parallel-backplane bus architecture becomes too wide and/or fast to be cost-effective. Serial-backplane architectures solve this problem by reducing the number of signal traces routed to the central switch.
A general rule for selecting a serial-backplane approach is to pick a total bandwidth exceeding 3.2 Gbits/s. A serial design also should be considered if the bandwidth requirements between the port card and central switch exceed 1 to 2 Gbits/s. The advantages include:
- Fewer signal traces. This reduces the connector size, the connector insertion force, and the complexity of routing multiple backplane traces.
- Reduction of simultaneous-switching-output (SSO) noise. The high-power, high-slew-rate parallel-bus drivers used in parallel backplanes generate large amounts of EMI and SSO noise. The differential drivers in serial-backplane designs don't suffer from these problems.
- The ability to hot plug.
Serial-backplane applications in the wide-area network (WAN) include new backbone terabit switches/routers, traditional digital crossconnects, and DWDM crossconnects. In the local-area network (LAN), they comprise high-bandwidth, high-port-count ATM, Ethernet, and Fibre Channel switches. Figure 1 shows a block diagram for a serial-backplane switch/router.
The designer must make a number of tradeoffs when architecting the serial-backplane system.1 First, there's the issue of choosing either a single or multi-chassis. A single chassis (Fig. 2a) allows the use of low-cost copper traces. On the other hand, a multi-chassis (Fig. 2b) design is more expandable. But it could require a fiber-optic interconnect.
Then there's the operating frequency, which must account for the distance and bandwidth of the serial transmission path. The choice of architectures also comes into play here. A partially blocking CLOS (Fig. 3) is easily expandable and typically lower in cost, and it consumes less power. But a nonblocking architecture provides the best connectivity.
Lastly, make sure you consider the number and placement of repeater circuits. Expansion architectures may require repeater circuits to restore signal integrity.
Parallel digital crossconnects or "grooming" switches demand a serial connection between the port card and the core "grooming" switch fabric. Because the switch matrix operates on parallel data, the serial backplane requires serializer/deserializer (SerDes) devices on both the port card and the switch-fabric card.
This scenario offers key tradeoffs for the serial-backplane designer, the first of which is operating frequency. This frequency must account for the distance and bandwidth of the serial transmission path. Also take into account the SerDes integration levels, which may require single, dual, or quad SerDes devices. The switch fabric will need higher levels of integration.
Finally, the designer must select an encoding scheme for the serial data routed across the backplane. Traditionally, 8B/10B is used for its excellent dc-balance characteristics and acceptance in serial-link standards, such as Fibre Channel and Gigabit Ethernet.
Packet-switched applications reconfigure connections much more frequently than circuit-switched networks. This need for rapid reconfiguration places pressure on the designer to do the following:
- Determine the optimum packet size to transit the backplane
- Rapidly reconfigure the switch core to maximize switch throughput
- Find a fast-acquisition phase-locked loop (PLL) if a serial-crosspoint-switch architecture is chosen. Acquisition time is the number of bit times required for the receive PLL to recover from the phase discontinuity introduced when the crosspoint is switched.
Let's go over the basics. Signal integrity is ultimately measured as a bit-error rate (BER), or number of bit errors per unit interval of time. Target BERs for serial-backplane systems are 10-14 or even 10-15 errors/s. Figure 4 shows the general relationship between eye opening versus BER at the input to a serial-backplane receiver. In a good design, the data eye across the backplane will have a 70% opening at the receive input, though 80% can be achieved in careful designs. The major contributors to poor signal integrity on high-speed serial traces are transmission-line reflections, crosstalk and noise, and jitter.
Transmission lines are signal wires with a defined relationship to a ground plane. A rule of thumb is that a signal with a rise (or fall) time shorter than 6 times the round trip delay time should be treated as a transmission line. (In the literature, the multiplier ranges from 2 to 10, but 6 seems to be the most common number.) On glass-epoxy pc boards (*r = 4.4), the speed at which signals travel is approximately 0.18 ns/in. Thus, a signal trace is a transmission line when:
TR or TF < 6 * 0.18 (ns/in.) * 2L (in.)
where L = one-way trace length
Say a 1-Gbit/s signal has rise/fall times of 300 ps. It requires termination if the one-way line length is >=1/8 in. Some references supply a more in-depth analysis of this issue.2, 3
Impedance mismatches in transmission lines produce signal reflections that change the signal's rise- and fall-time characteristics. To avoid reflections, the line must be terminated with an impedance equal to its own characteristic impedance. This is accomplished either at the far (receiving) end of the line with a parallel termination, or at the near (transmitting) end with a series termination.
A series termination allows the signal to reflect once at the far end of the line, and then kills any further reflections at the near end. This type of termination may be used with digital signals. Serial-backplane signals are essentially analog, however, and should always terminate at the far end. A simple termination for high-speed differential signals is a line-to-line resistor at twice the impedance of either single line.
When laying out a board, first consider high-speed I/O. Termination resistors and output pull-down resistors should be placed as close as possible to their respective I/O. Typically, ac coupling capacitors are included in the high-speed signal path to avoid dc bias mismatches and prevent overvoltage and latch-up while the chip is powering up. There's no overwhelming reason to place the coupling capacitors close to either the transmitting or receiving part. If a signal is routed through a connector, the ac coupling capacitors are often laid out close to the connector.
Keep differential-pair traces the same length to minimize skew, and use either chamfered or rounded corners to reduce electromagnetic interference (EMI) and impedance mismatches. Dielectric and skin loss must be taken into account when creating the board stack-up. As signal speed increases, skin loss rises. Traces must be made wider to minimize signal attenuation. But wider traces require a thicker board to maintain the line's required characteristic impedance.
Figure 5 offers three basic printed-circuit-board structures: microstrip, stripline, and dual stripline. Formulas for controlling impedance and propagation delay are available for each type of structure. But the best method for ensuring good impedance control is to work with the board manufacturer during layout.
As for power-supply considerations, separate power and ground planes are recommended in high-speed applications. The use of dedicated power planes minimizes IR bus drops, while providing a low ac impedance path. Full ground planes provide shielding and ensure that the current return path is unobstructed. Avoid segmented ground or power planes, because ac return paths may be forced to route around the slots in these planes. This has the doubly bad effect of increasing the area of the ground loop and possibly creating a slot antenna.
In high-speed devices, voltage levels are typically derived from the VCC rail. Power planes should be isolated from switching supplies using ferrite beads. Take special care to bypass analog power and ground pins, since noise on these lines will translate directly to jitter. Ideally, each power pin should be individually bypassed with a low-impedance capacitor.
Jitter is deviation from the ideal timing of an event that affects the synchronization of clock-recovery circuits. It's divided into random, deterministic, and frequency-dependent components. Random jitter is essentially thermal noise. It's Gaussian in nature, and typically specified in picoseconds on a root-mean-square (rms) basis. Deterministic jitter is caused by bandwidth limiting in the data path, and adds linearly. Frequency-dependent-jitter is brought about by noise, typically due to switching power supplies. Figure 6 shows the sources of jitter in a link design.
As the frequency of a signal increases, it is more easily broadcasted. Guidelines for controlling radiation effects, such as crosstalk, noise, and EMI, include:
- Avoid 180° turns, 90° corners, and slots in power and ground planes. All of these create antenna effects.
- Keep high-speed differential traces short, of equal length, and close together to decrease noise susceptibility and EMI.
- Minimize ground loops by placing signal planes adjacent to power and/or ground planes. Here are the common issues affecting the return-current path: poor connector-pin assignment, breaks in power or ground planes adjacent to a signal plane, and poor shield grounding of copper cables.
If noise, crosstalk, or EMI is a concern in a design, keep sensitive high-speed traces in stripline environments. That will provide the best performance.
Once you've mastered the basics, all that remains is putting a system together. Components in the serial data path include transmit and receive serializer/deserializers, backplane connectors, and the crosspoint switch matrix. In laying out the port card, place the serializer as near to the backplane connector as possible. Hand-route the high-speed traces; it shouldn't be difficult to keep them 3 in. or less in length. For convenience, make these traces microstrip structures to aid component placement and facilitate board debugging.
The backplane connector's impedance determines the required characteristic impedance of the signal traces. This, in turn, is used to figure out the board stack-up. Serial-backplane connectors incorporate shielding techniques and good impedance control to maximize the signal integrity of high-speed signals.
In a 19-in. rack-mounted system, the maximum trace length across the backplane is approximately 24 in. This can be negotiated with good signal integrity at 1.25-Gbit/s data rates. Backplane traces are implemented as striplines. To avoid crosstalk on these long signal runs, spacing between adjacent signal channels is typically 2.5 to 3 times the trace width. Add copper-poured grounds between the signal channels to increase crosstalk immunity.
Board-material selection can improve both signal quality and density. At signal speeds of less than 2.5 Gbits/s, FR4 will perform adequately across a 19-in. backplane. Speeds ranging from 2.5 to 5 Gbits/s may benefit from alternative board materials. For signal rates above 5 Gbits/s, look to alternative interconnect technologies, such as fiber-optic interconnects.
The following are the signal-integrity issues associated with the crosspoint matrix: channel-to-channel skew, rise-/fall-time degradation (deterministic jitter), signal attenuation, and crosstalk.
Skew can be controlled in the layout of the backplane and switch-card traces. If required, repeaters may be employed on the switch card to improve jitter and restore signal levels. Examine these issues prior to final layout through breadboarding and simulation.
The selection of a transmission medium, copper versus optical, is dictated by distance, cost, and signal speed. Interchassis signaling may be accomplished over equalized STP cable at signal speeds ¾1.25 Gbits/s. At data rates of 2.5 Gbits/s and above, signaling between chassis is typically optical. This provides better signaling characteristics, as well as flexibility in system architecture. The cost of implementing a fiber-optic interconnect, however, is 25% to 50% higher than copper. The cost delta is in the optical- to-electrical-conversion and associated analog components.
Since the bandwidth through fiber-optic cable is virtually unlimited, its use in serial-backplane systems allows multiple architecture configurations. Placing port cards and switch cards in separate chassis provides a path to modular scalability. Mesh architectures, which employ crosspoint switches on the port cards, eliminate the independent switch card.
Using today's fiber-optic interconnect technology, aggregate channels prior to transport between chassis to minimize cost. In the near future, parallel optical fiber will allow multiple optical connections through a small-form-factor connector. This will bypass the requirement for a copper backplane.
References:
- Palkert, Tom, and Spehn, Dick, "Designing A High-Bandwidth ATM Switch," Electronic Design, May 13, 1996; p. 112.
- Wadell, Brian C., Transmission Line Design Handbook, Artech House Inc., 1991.
- Johnson, Howard W., and Graham, Martin, High-Speed Digital Design: A Handbook of Black Magic, Prentice Hall PTR, 1993.