This article is part of the TechXchange: Silicon Photonics.
Broadcom is partnering with Tencent to accelerate the adoption of high-bandwidth co-packaged optics switches. The company said it will deploy its first switch with optics inside, called Humboldt, in the cloud giant’s data centers next year.
The switches are based on a new architecture announced in early 2021 that brings co-packaged optical interconnects inside a traditional switch. The technology—also called CPO, for short—effectively shifts the optics and digital signal processors (DSPs) out of the pluggable modules placed at the very front of the switch, uses silicon photonics to turn them into chiplets, and then puts them in the same package as the ASIC.
Broadcom said the ability to connect fiber-optic cables directly to the switch brings big advantages. The company revealed that Humboldt offers 30% power savings and costs 40% less per bit compared to systems relying on traditional switches.
In the new Humboldt switch, a 25.6-Tb/s Tomahawk 4 ASIC that contains 100-Gb/s SerDes is directly coupled to four 3.2-Tb/s Silicon Photonics Chiplets in Package (SCIP) optical interfaces. The silicon-photonics chiplets connect 16 MPO connectors on the front of the switch. While the silicon-photonics engines supply 12.8 Tb/s of the total 25.6 Tb/s, Humboldt uses electrical IO to interface with 32 QSFP112 ports that run at 400 Gb/s to supply the remaining bandwidth.
“In co-packaged optics, what you have is the same switch and then that is directly attached to the optical engines. We directly drive from the switch, and so it provides a very elegant and very power and cost-optimized solution,” said Manish Mehta, VP of Optical Systems Marketing and Operations at Broadcom.
The company hopes the partnership with Tencent will lend more credence to the concept of co-packaged optics in high-bandwidth switches.
Copper Hits Bottlenecks
As bandwidth grows with artificial intelligence (AI) and other workloads, cloud giants are increasingly using the power of photons to string together tens of thousands of servers in their rapidly expanding data centers.
“Customer demands for higher bandwidth have pushed Tencent to perpetually scale our network,” said Sage Zoe, Cloud VP, Tencent.
According to Broadcom, on average, the world’s largest cloud-scale data centers come packed with more than 10,000 switches, largely powered by Broadcom’s Tomahawk chips. To keep up with the need for speed in massive data centers, the switches house more than a million optical interconnects that fling data at a fraction of the power use and cost due to the limits of electrical interconnects at high speeds.
“As you shift to higher speeds, you’re significantly limiting the reach of passive copper even if you change the connectivity architecture versus what it is today, so a larger percentage of the links will have to move to optical,” said Rajiv Pancholy, head of hyperscale strategy at Broadcom (Fig.1).
But the problem co-packaged optics promises to solve is not port performance, but rather power consumption. Heat is also limited, opening the door to higher port densities and, as a result, giving a boost to bandwidth.
Today, switches depend on optical transceivers that are plugged directly into the front of the switch, converting electric signals into light and propelling them over fiber-optic cables radiating from the switch.
The optics are connected to the SerDes ports at the heart of the switch using electrical IO. When a signal leaves the switch, it travels over copper traces on the circuit board before it enters a series of connectors that lead to the optical transceivers. The pluggable module contains a DSP that runs signal processing to make sure the signal is not scrambled at high speeds as it travels from the switch silicon on the board to the optics.
But as performance-hungry cloud giants move to higher bandwidths, electrical IO has become a bottleneck. Connecting the SerDes to optics with electrical interfaces is becoming more complicated and power-hungry.
Broadcom has been bracing for these problems to boil over for years. The company has decades of research and development (R&D) in optics behind it, covering a wide range of optical technologies, including lasers.
In 2019, it hired Alexis Bjorlin, one of the stars in the world of networking hardware, to lead its optical systems business. She pushed the unit to develop a silicon photonics platform that could leverage many of its existing optical building blocks. (Fig. 2). In a sign of the important role that silicon photonics will likely play in the future of data centers, Bjorlin left Broadcom in late 2021 to become Meta’s VP of infrastructure hardware.
Shorter Distance, Less Power
By reducing the distance that generally separates the switch silicon from the optics to mere millimeters, Broadcom said Humboldt cuts down on the amount of power required to run all of these optical transceivers.
Power is a major challenge for companies such as Tencent that are building huge data centers to power their clouds and various other services. Data centers could account for 15% of global electricity demand by 2025.
The DSP in a standard pluggable optical transceiver runs signal processing to scrub out impairments that scramble the signal as it traverses the circuit board and multiple connector discontinuities in a standard switch system—and it burns through lots of power to do so.
By removing slots for pluggable modules, Humboldt creates more space to pack more optical ports on the face of the switch, boosting bandwidth density. Alternatively, instead of increasing density, you can also space out the connectors farther apart to improve air cooling, said Broadcom’s Mehta.
At the switch level, Humboldt consumes half the power of a standard pluggable module per port, according to the company. Mehta said that pluggable modules packaged with a 7-nm DSP to support a bandwidth of 800 Gb/s runs on roughly 16 W. To supply the same amount of bandwidth, he said the Humboldt switch that Tencent plans to bring to its data centers expends 7 W.
Broadcom promises co-packaged optics will reduce cost per bit compared to traditional switches as more energy-efficient optical engines replace pluggable modules packed with DSPs and other expensive silicon.
Broadcom said it sees customers spending about 10X more on optics than switch chips, which can cost thousands of dollars each. With co-packaged optics, optical modules that propagate from the faceplate of the switch are no longer required. Depending on the bandwidth, these pluggable modules can cost several hundred dollars each.
But Humboldt is not without its complexities. While the optical engines are co-packaged, each of the four interfaces in the package is paired with a passively cooled continuous wave (CW) laser to supply constant light to them.
For all the virtues of co-packaged optics, the technology has been plagued by doubts for years. Other industry insiders have warned that it is too early for switches with optics inside to compete with traditional hardware, citing the challenges related to reliability and the staying power of electric interconnects at high speeds.
Bound for the Cloud
Broadcom executives said the fact that one of the world’s largest technology companies is willing to use Humboldt is evidence that the underlying technology is ready to roll (Fig. 3).
“Once companies see that it’s deployed, that’s when co-packaged optics will start to become part of the conversation,” Mehta pointed out. “For now, it’s a lot of questions like, ‘How will you get it to work? Will it work?’”
The Humboldt switch Tencent is deploying in its cloud data centers features a combination of co-packaged optical and copper ports. Broadcom said this gives Tencent the flexibility to use the Humboldt switch where and how it makes the most sense.
Broadcom is playing the long game with its co-packaged optics ambitions. The company’s roadmap includes a 51.2-Tb/s switch that will pair its new Tomahawk 5 ASIC with eight optical engines embedded in a package.
Broadcom is not alone in investing in the co-packaged optics era. Marvell is working with Cisco, a leader in networking gear, to build a 51.2-Tb/s co-packaged optical switch, which is slated to launch in 2024.
Humboldt is set for deployment in Tencent's data centers sometime next year. It has already started shipping samples to potential early customers.
This article is part of the TechXchange: Silicon Photonics.