Bus systems based on standards like PCI or H.110 are widely deployed in communications equipment. But such architectures will eventually run out of gas. It's true that PCI-based bus systems have progressively moved toward higher bandwidths—from 32-bit, 33-MHz systems at 133 Mbytes/s to 64-bit, 133-MHz systems at 1.066 Gbytes/s. With each of these advances, though, devices remain connected to a bus that, by its very nature, is shared. So, each device must wait its turn. This calls for some kind of arbitration because only one device can talk at once. Also, bus systems just don't meet next-generation requirements for converged data, voice, or video networks.
Fortunately, there's an alternative to simply cranking up the clock further or widening the bus beyond 64 bits. StarGen is introducing a switched, point-to-point approach that overcomes many limitations of the bus-based architectures. To boot, the new architecture preserves compatibility with older bus-based systems.
The StarGen switch fabric fulfills a role that's roughly coincident with the backplane (Fig. 1). Furthermore, it provides a compatible migration path from existing bus-based standards. Applications for the StarGen switch fabric include DSL access multiplexers (DSLAMs), voice-over-IP (VoIP) gateways, and edge-routers/switches. The switch fabric may find other applications in cable head-end systems, wireless base stations, and computer telephony platforms.
A star and a PCI fabric bridge are being introduced as basic building blocks of the StarGen switch fabric. These devices comprise the first phase in fulfilling StarGen's objective of establishing an open, widely adopted interconnect standard. By providing open access to the technology specification, third parties will be able to develop complementary products and expand market coverage.
The PCI fabric bridge interfaces 64- or 32-bit PCI buses operating at 66 or 33 MHz to the StarGen switch fabric (Fig. 2). Designated as the SG 2010, it converts the PCI protocol into the StarGen protocol, creating frames that are essentially packetized transactions. Those frames are routed out over one of the two serial links on the bridge. The links can be used either independently or bundled. When used independently, they provide two independent connections in a fabric. When bundled, they operate as a single, fast link to achieve 5-Gbit/s, full-duplex communication (see "8B/10B Encoding,").
The 2.5-Gbit/s, full-duplex links comprise four aggregated 622-Mbit/s LVDS pairs. A serializer/deserializer (SERDES) is used on each link to convert a parallel, on-chip data stream to the transmitted serial stream off chip. The SERDES reverses the process to receive the incoming serial data stream, recovering the embedded clock and converting it back into a parallel data stream. Power consumption is approximately 2 W.
The PCI fabric bridge performs two principal functions. It supports legacy-address-routed traffic, which provides 100% compatibility with existing PCI software. It also provides a fabric-native path and multicast routing capability. Plus, the PCI bridge fabric can isolate address ranges and perform address translations. It's 5-V tolerant and compliant with the PCI local bus, rev 2.2 specification.
The PCI fabric bridge supports both "transparent" and "nontransparent" bridge functions. As a transparent bridge, it adheres to the PCI-to-PCI Bridge Architecture Bus Specification of the PCI Special Interest Group. As a nontransparent bridge, it can be used to distribute processors around a PCI hierarchy with each processor assigned an independent local address space.
The PCI fabric bridge can connect to local bus interrupts and transport those interrupts across the fabric. When they arrive at another bridge at the other edge of the fabric, these interrupts can be converted to interrupt assertions at that local bus.
The PCI fabric bridge interfaces to the star fabric through two 2.5-Gbit/s, full-duplex serial links. The SERDES converts the serial data streams to parallel streams and passes them to the link buffers. The star protocol frames are converted to PCI traffic and transferred to the PCI buffers, and then on to the PCI bus through the PCI interface. For traffic flowing from the PCI to the fabric, the process occurs in reversed order. The bridge also interfaces to an SROM to preload configuration registers, and an EPROM interface capable of providing BIOS code.
The star performs the switching function within StarGen's switch fabric (Fig. 3). Designated the SG 1010, it provides six serial links. Each link comprises four aggregated 622-Mbit/s, LVDS differential pairs. Each is capable of 2.5-Gbit/s, full-duplex communication, for an aggregate bandwidth of 5 Gbits/s per link. Two of these links can be bundled to create a 5-Gbit/s, full-duplex link to another device.
Every link has a SERDES for converting the serial data streams to parallel form for on-chip transport. Separate buffer queues are maintained at each outgoing port for each incoming port and each class of service. This design prevents head-of-line blocking. Multicast tables are maintained in the switch for sending single data streams to many destinations. Traffic other than multicasting uses source-based routing and doesn't require routing tables within the star. Initialization is supported through the SROM interface.
The switch is nonblocking and can simultaneously transmit and receive on all links. In other words, traffic on one port doesn't block traffic on another port. Power consumption is approximately 2.5 W.
The StarGen switch fabric can free telecommunication systems from the need for independent buses to handle voice and data buses, as is the case with H.110 and PCI. Voice traffic is eight deterministic bits of data every 125 µs, something not particularly demanding from a bandwidth point of view. But when someone tries to move a 100-Mbit file, you don't want that to hinder voice delivery. The challenge then is to deliver real-time data traffic in the presence of asynchronous, best-effort, delivery traffic that's so common in the computing/data-networking world. You want deterministic delivery of this isochronous traffic (voice) even though some devices in the system are trying to use every bit of bandwidth.
The StarGen architecture focuses on solving this issue in the backplane of the communications equipment. It provides simultaneous support for both asynchronous and isochronous traffic. Within the backplane, the StarGen approach makes a distinction between the data payload, whether it's packet-oriented, IP traffic, or QoS-oriented traffic requiring real-time delivery, like voice or TDM traffic. StarGen's architecture is said to be unique with regard to its ability to support both kinds of traffic.
The StarGen architecture defines seven classes of service. The initial products implement four. First, asynchronous service is the best-effort delivery associated with control and data that's non-QoS traffic. Next, isochronous service is used for TDM or media traffic where real-time delivery guarantee requirements are associated with it. Then there's a multicast class so one source can deliver information to multiple end points. Finally, there's a class for handling high-priority, provisioning traffic, such as error notifications, fabric maintenance transactions, and interrupts.
Even though both the star and PCI fabric bridge devices can support four classes of service, they are designed to be forward compatible. Therefore, they will interoperate with future devices likely to implement additional classes of service.
Frequently, there will be separate interconnect infrastructures, one for control and one for data. The StarGen approach can unify them into a single interconnect. By doing away with the necessity for two different interconnect structures, it enables engineers to design both cost-effective and highly reliable integrated systems.
The StarGen architecture supports three routing methods: standard PCI addressing or address routing, path routing, and multicast routing.
One of the obstacles in the CompactPCI space is the restraint on the number of slots for a single chassis. A 19-in. rack has space for 20 or 21 slots, but you can't use them all because the limitations of PCI basically constrain you to no more than two eight-slot PCI buses.
Also an obstacle is the fact that many applications are pushing requirements for uplinks with higher data rates beyond OC-3 to multiple OC-3s or OC-12. This means more performance per card. StarGen supports such performance on a per-slot basis. For example, H.110 has the capacity of 4000 time slots, which typically means 2000 calls for an entire backplane. But with the StarGen architecture, you can realize that much bandwidth in just one slot.
A system based on a switch architecture employing stars and fabric bridges presents a large constellation of topological options. The largest nonswitch solution employs three bridges (Fig. 4a). A single-star configuration can be configured with six fabric bridges (Fig. 4b). A redundant fabric can be realized by adding a second star (shown by dotted lines in Fig. 4b) and connected to the second fabric link on each of the six bridges. Redundancy also can be achieved by the configuration shown in Figure 4c. In both of these configurations, any link or any switch can fail, yet full connectivity will be maintained. Clearly, with only 10 stars, 30 PCI bus segments can be interconnected through the fabric (Fig. 4d). Larger topologies of up to hundreds of bus segments are achievable.
Traditionally, a retry is employed whenever a node is too busy or doesn't have sufficient buffer space. For instance, in PCI, if a master tries to deliver data to a target that doesn't have space in its buffers, the master still attempts the transaction. The master doesn't really know when the target has space to accept the transaction, so it continually tries and retries the transaction until the target has space. But retries consume bandwidth needlessly and don't make efficient use of the fabric's resources.
Use of StarGen's architecture completely eliminates retrying. Instead, a credit-based mechanism is implemented between two nodes for the transfer of traffic. A sender maintains a set of counters for each class of service and for each output port in the next hop.
Before it can send traffic, the sender checks to make sure that it has sufficient credit at the next node—enough buffer space so the traffic can travel all the way to the output port of the next switch. If it doesn't have credit, it won't attempt to send the transaction. Rather, the sender will hold the transaction until sufficient credit is available. Once the transaction is sent, the sending node decrements its credit. In turn, when the receiving node is able to forward the traffic, it sends a credit back, which causes the credit to be incremented in the sending node. This adds up to a closed-loop system that enables the transfer of data without the presence of retries.
Comparing With InfiniBand Instead of using high-speed technologies, StarGen lets designers configure designs with much lower costs. The StarGen physical layer significantly differs from that of InfiniBand. Although it delivers the same raw data rate, StarGen uses four 622-Mbit/s, LVDS differential pairs, whereas InfiniBand uses a single differential pair running at 2.5 Gbits/s.Running at 622 Mbits/s accords StarGen some significant advantages. It's a lot easier for a designer to work with this system. At 2.5 Gbits/s, it's very difficult to get the signal off of the silicon, through the bond wire, out of the package, across the board, down a cable, and then successfully passed through the same sequences of interfaces in reverse order at the receiving end. But below about 800 MHz, this becomes much easier. At 622 Mbits/s, StarGen enables low system-design costs by avoiding exotic pc-board materials and connectors. Furthermore, it's relatively easy to drive the 622-Mbit/s LVDS signal off the package, through standard FR-4 board construction, through the existing CompactPCI connectors, and down a backplane—or down CAT5 cables with RJ-45 connectors.
Price & AvailabilityThe SG 2010 PCI fabric bridge and the SG 1010 star both come in 272-pin PBGA packages. Each costs less than $50. Prototypes will be available this quarter, with quantity shipments by the second quarter.StarGen Inc., 201 Boston Post Rd. West, Ste. 200, Marlborough, MA 01752; (508) 786-9950; fax (508)-786-9785; Internet: www.stargen.com.