There’s seemingly never enough compute power, memory, or bandwidth available, so vendors keep pushing the envelope. Mellanox’s latest Spectrum-3 Ethernet switch chip does just that. Not only does the chip have bandwidth of 12.8 Tb/s, it includes a host of features that are very data-center-friendly. It can handle up to 128 ports of 100-Gb Ethernet (GbE), 64 ports of 200-GbE, and 32 ports of 400-GbE. The system manages up to 200,000 NAT entries and over a million on-chip routes.
The Spectrum-3 is at the heart of SN4000 platform family (Fig. 1). The rack-mount systems can come installed with Mellanox Onyx, Cumulus Linux, Open Network Install Environment (ONIE), Microsoft’s open-source switch Operating System for Open Networking in the Cloud (SONiC), or Linux DENT.
The chip supports NRZ and PAM4 signaling. The latter is used for the highest bandwidth connections.
Spectrum-3 works equally well as a leaf or spine switch. Its adaptive routing support (Fig. 2) operates in a mixed switch environment. However, the chip performs better in a homogeneous Spectrum-3 network because it will automatically distribute traffic information, allowing for more effective routing. The layer-3 system support is “flowlet-aware.” Flowlets are switched packet bursts that can be split across paths between a source and destination. Mellanox’s FlexFlow systems enable traffic to be distributed across many connections so that data can bypass congestion. It also helps with load leveling.
The other advantage of using a homogeneous environment is to take advantage of the adaptive flow prioritization (Fig. 3). The chip can adjust the queues so that smaller packets aren’t delayed when many large packets are in the queue. Large packets, often called “elephant flows” are more efficient, but lock up queues for a longer time if not handled properly. All of this is designed to provide fair bandwidth sharing.
The list of features and standards supported by Spectrum-3 is extensive. The platform also supports a range of overlay protocols including EVPN, VXLAN-GPE, MPLS-over-GRE/UDP, NSH, NVGRE, and MPLS/IPv6 based Segment routing.
RDMA over Converged Ethernet (RoCE) is a feature of the Spectrum-3. It’s “RoCE-Ready,” including one-click configuration with hardware-accelerated and end-to-end congestion management.
One unique feature is Mellanox’s What Just Happened (WJH) diagnostic support. This granular telemetry system is designed to simplify network operations and debugging by having the firmware keep track of what’s going on and reporting not only that an error occurred, but the current context and prior state. This allows for faster identification of the root cause of an error. WJH support is integrated with open-source tools like Wireshare and Grafana as well as tools like Kibana and NetQ.
