Electronic Design

Ethernet Channel Adapter Sets Throughput And Latency Records

NetEffect has teamed up with Fulcrum Microsystems to demonstrate what's probably the fastest 10-Gbit/s Ethernet system available. Using NetEffect's NE010x Ethernet channel adapter (ECA) and Fulcrum's 24-port FocalPoint switch chip, benchmarking tests at the Ohio Supercomputing Center demonstrated a combined end-to-end latency of less than 10 µs—a new performance record. The NetEffect ECA uses the iWARP Ethernet standard.

Today, virtually all companies using high-speed networking are experiencing ever-growing network traffic. But network throughput seems to be getting ahead of server CPU clock speeds, as well as memory capacity and speed. Furthermore, the networks themselves are becoming increasingly more diverse and segregated.

The local-area network (LAN) part of the system stays with standard Ethernet. But storage systems typically use a separate Fibre Channel network, while server clusters use some other networking technology such as InfiniBand. Many IT managers wish Ethernet could be leveraged to handle all three of these applications. That wish has come true, thanks to NetEffect's ECA.

iWARP refers to a high-speed Internet method known as Internet Warp. This memory data transfer technology involves remote direct memory access (RDMA), remote direct data placement (RDDP), and marker PDU alignment (MPA) protocol stacks. It's an extension to Ethernet developed by the RDMA Consortium, an organization of more than 50 companies that includes all the big computer and networking players.

According to the RDMA Consortium, RDMA "is the ability of one computer to directly place information in another computer's memory with minimal demands on memory bus bandwidth and CPU processing overhead while preserving memory protection semantics. RDMA over TCP/IP defines the interoperable protocols to support RDMA operations over standard TCP/IP networks."

The RDMA consortium submitted its draft design to the Internet Engineering Task Force (IETF), and the draft became the standard called iWARP. The standards are ratified, and new requests for comment (RFCs) will be be issued shortly.

iWarp alleviates the greater stress that 10-Gbit systems are putting on server processing power and memory bandwidth. The RDMA Consortium says RDMA over TCP addresses these issues in two very important ways. First, much of the protocol processing overhead can be moved to the Ethernet adapter. Second, each incoming network packet has enough information to allow its data payload to be placed directly into the correct destination memory location, even when packets arrive out of order.

RDMA's direct data placement (DDP) eliminates intermediate memory buffering and copying (and the associated demands on the memory and processor resources of the computer nodes), without requiring the addition of expensive buffer memory on the Ethernet adapter. Additionally, RDMA over TCP/IP uses the existing IP/Ethernet based network infrastructure.

About 40% of a server's overhead is processing the TCP/IP stack. It's usually done in software. As network speeds increase, the need to process packets goes up linearly. Moving from 1GE to 10GE means a tenfold increase in CPU overhead. As a result, performance grinds down to some lower speed. NetEffect's implementation of iWARP offloads this TCP/IP processing to special hardware in the ECA, virtually eliminating this problem.

But that's not all. TCP/IP processing also requires the packet to be stored and copied multiple times during processing. Each copy means that a read and a write must be implemented, greatly slowing the overall operation. Using iWARP's RDMA and DDP eliminates this whole process.

Finally, context switching also adds an overhead in the server that slows down the whole packet processing sequence. Context switching, which occurs during an I/O operation, requires the server to store registers, counters, stack pointers, status words, and other information before switching gears. With iWARP, the ECA performs an operating-system bypass that deals with all of those operations. The bottom line is that iWARP blows away all that overhead, reducing latency to its lowest possible level.

NetEffect's NE010 ECA handles all of the iWARP functions—most directly inside NetEffect's chip (see the figure). The NE0010 is available now for designers who want to participate in NetEffect's Beta testing program. The University of New Hampshire's InterOperability Laboratory is providing testing and certification for iWARP equipment to ensure product interoperability.


Fulcrum Microsystems
Internet Engineering Task Force

RDMA Consortium
University of New Hampshire Interoperability Laboratory

Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.