Just as you can never be too rich or too thin, you can never have too much storage capacity on your computer. At one time, a 5- or 10-Mbyte hard drive on a PC was a big deal. Today, 30- to 60-Gbyte drives are common on even the low-end PCs, with that capacity surging to many hundreds of gigabytes in network servers.
Large companies and government agencies routinely have storage capacities of many terabytes (Tbytes) or 1,000,000,000,000 (1012) bytes (or the binary equivalent 240 = 1,099,511,627,776). Storage-area networks (SANs) now make it possible to implement and manage the access to those huge storage facilities.
In fact, storage-area networking is the fastest growing computer networking technology today. A SAN is an interconnection of PCs, servers, and disk-drive subsystems that communicate over a fiber-optic network. SANs are used for massive storage of data and files, as well as system backup. The servers in the network communicate with the storage boxes via a special switch that allows any PC full, speedy access to any of many disk arrays. These SANs result from the huge need for extra storage capacity. Ballooning storage requirements can be traced to the Internet and e-mail, along with the need for backup, disaster recovery, and government regulations.
Many applications also require huge storage files to hold graphics such as video, digital photographs, and medical imaging. Research firm iSuppli (www.isuppli.com) estimates that use of Fibre Channel SAN ports has grown 30% annually for the past several years. The company also projects continued SAN growth at a compound annual growth rate (CAGR) of 31% through 2008. Let's look at the recent developments in SANs.
SAN TOPOLOGY AND ARCHITECTURES Disk drives in servers are still primarily connected directly via the Small Computer Systems Interface (SCSI) bus in the smaller networks (Direct Attached Storage). However, two storage networking solutions have emerged over the years: Network Attached Storage (NAS) and SANs.NAS systems use hard-disk arrays that are usually connected to the Ethernet LAN used by the server and PCs. They are given an IP address, and they access data in file format rather than block format, which is common for SANs. In this way, the files become accessible by virtually anyone on the local-area network (LAN) with authorization. NAS is a file-based (rather than a block-based) access method that uses file protocols like the Network File System (NFS) in UNIX/Linux systems and the Common Interface File System (CIFS) in Microsoft Windows systems.
For larger storage needs, a block-based SAN access system is preferred because it's easier to scale and provides access through any attached server. Figure 1 illustrates the generic SAN architecture. PCs and workstations are connected to the various servers by way of the enterprise LAN. Typically, the LAN is 100-Mbyte/s Fast Ethernet on CAT5 cables, although One Gigabit Ethernet (1GE) is becoming more common.
Then the servers are joined to the storage systems in either Redundant Arrays of Independent Disks (RAID) or Just a Bunch of Disks (JBOD) through switches. The transmission medium is almost universally fiber optics. In most SANs, the link is Fibre Channel (FC), the well known fiber-optic system designed specifically for block transfers of data between disk drives and servers. The SAN provides the most flexible method that lets any PC access anything, use tape backup, or store less frequently used files.
An alternative to FC interconnections is Ethernet connectivity using a newer protocol called Internet Small Computer Systems Interface (iSCSI). It encapsulates standard SCSI disk commands into TCP/IP packets for transmission over Ethernet or the Internet. While FC still dominates in most SANs, iSCSI systems are growing in number because the network connections usually cost less and in some cases use existing LAN connections rather than new runs of fiber. In a nutshell, iSCSI is making progress, but FC is keeping pace.
WHAT'S NEW WITH FIBRE CHANNEL? More than 90% of SANs now use FC connectivity because it is robust and has kept up with the speed needs of large organizations. This ANSI standard came about in the late 1980s and defines a protocol and a fiber-optic physical layer (PHY) for transmitting data at 1-, 2-, 4-, and 10-Gbyte/s rates. Yet it is a bit expensive and complex, requiring special knowledge and talent to manage and maintain. It is perfect for large enterprise SANs.The FC physical link is a pair of fiber-optic cables, one for transmit and one for receive. Usually, the cable is 50- or 62.5-µm multimode fiber for the shorter, lower-speed links. But single-mode fiber is used for the higher-speed, longer-reach connections. Most FC transceivers implement the short-wavelength (850 nm) lasers, while longer-wavelength (1310 and 1550 nm) lasers are used in the faster systems over longer distances.
The FC rate of 1062.5 Mbits/s or 1 Gigabit FC (1 GFC) is very common today, although most newer systems have migrated to 2 Gbits/s or 2 GFC. Both 4-Gbyte/s and 10-Gbit/s rates have been defined but are not yet widely used. The table shows the line rate and the total throughput for duplex operation.
One of the most recent developments in FC is the availability of the 4-Gbit/s controller chips and transceivers at reasonable prices. Currently, there isn't much interest in the defined 10 GFC standard because of its much higher cost.
As for network connections in FC, they may be point-to-point, a loop, or a switch fabric. The fabric topology offers the most flexibility and can accommodate up to 224 devices in a cross-point switching arrangement. FC switches allow simplified connections to servers and devices and usually permit multiple devices to communicate with one another at the same time. In FC SANs, the servers, called initiators, use a host bus adapter (HBA) that plugs into a PCI slot and provides a connection to the fiber-optic cable. This cable links to an FC switch that in turn links to the box of disk drives known as the target.
FC is also inherently secure. Because FC uses its own fiber wiring, which is separate from the corporate LAN and other connections, it's nearly impossible to hack into it. No special software or other precautions are usually needed to ensure security. On the downside, FC is expensive and complex, and typically it has a range of only about 40 km. But despite the overhead required to buy, use, and maintain FC, it's still the interface of choice in most SANs.
In addition to its higher speeds, FC can communicate over longer distances via the Internet. Two new protocols aid in this quest. FCIP or Fibre-Channel-over-Internet Protocol is an Internet Engineering Task Force (IETF) development that encapsulates FC frames into packets and transmits them via TCP/IP. This protocol allows multiple FC SANs to be interconnected and managed over any IP network. This technique is known as a tunneling protocol.
Another option is the Internet Fibre Channel Protocol or iFCP. This approach permits FC SANs to be interconnected using TCP/IP via off-the-shelf Ethernet switches or routers. The switch or router forms a gateway that replaces the usual FC switch. The iFCP protocol initiates and terminates each FC session.
THE EMERGENCE OF A CHALLENGER: iSCSI As the SAN market pie gets bigger, FC is growing. But now a newer technology, iSCSI or Internet SCSI, is taking market share. It overcomes most disadvantages of FC, so the newer competing iSCSI systems look attractive to many designers contemplating a SAN for the first time. The iSCSI systems use standard affordable off-the-shelf Ethernet components and TCP/IP software that are familiar to so many.Also, iSCSI or I-skuzzy is an Internet Engineering Task Force (IETF) standard that was approved last year. Although it lacks an RFP number to signify that it's a fully ratified IETF standard, that should come this year. The iSCSI protocol is essentially a serial version of SCSI using Ethernet. It permits the SCSI command and data structure so widely used in DAS to be packaged and sent over standard lower-cost Ethernet connections.
To access remote storage-over-SAN, an application request to the operating system produces the appropriate SCSI commands and data requests. The iSCSI protocol in the NIC encapsulates the SCSI commands into TCP, then into IP packets. These packets are next transmitted over existing Ethernet connections (Fig. 2). The connection to the RAID or JBOD is by way of a standard Ethernet switch. Separate Ethernet wiring can be used, but usually it's possible to use existing CAT5/6 LAN wiring if it handles the higher speed of typically 1 Gbit/s. Using existing LAN wiring cuts cost enormously compared to an independent fiber connection as with FC. The neat thing about iSCSI is that all storage operations are entirely transparent. The server simply believes it is dealing with an SCSI controller while the LAN sees only TCP/IP transactions.
In iSCSI systems, the interface cards in the servers are Ethernet network interface cards (NICs) referred to as initiators. Many include a TCP/IP offload engine (TOE), a separate dedicated processor that handles all TCP/IP processing to minimize the processing overhead on the main server CPU. This greatly improves performance. The interface in the storage device is also a form of Ethernet NIC called a target. The system otherwise uses standard Ethernet switches and routers.
Another neat benefit of iSCSI is that distance no longer is an issue. The storage devices may be miles apart while a standard Internet connection via a metropolitan-area network (MAN) or wide-area network (WAN) provides the link to the server. The server actually thinks it is speaking to a locally connected disk where in reality it may be on the other side of the continent.
Of course, there are downsides to iSCSI as well. First, there is the security issue. Since you are using the LAN and possibly MAN or WAN connections, some form of encryption and/or authentication is warranted. Segmenting the LAN with switches can help but is not a complete solution. Standard IETF IPsec security, using either 3DES or AES encryption, is designed for use with iSCSI. This extra security slows things down. Second, the SCSI commands and TCP/IP headers all add considerable transmission overhead to storing or accessing data. For a given raw data rate, this actually makes iSCSI slower than FC. With FC products now at the 2-Gbit/s and 4-Gbit/s levels, FC tends to have a higher throughput. However, TCP/IP offload engines (TOE), special high-speed processors built into the server NICs, relieve the server CPU of this chore and thereby correct for this difference. A throughput dilemma can be solved by 10 GE hardware, but the cost is still high.
SANs using iSCSI are growing in usage simply because they are easy to implement and cost considerably less than an FC system. You can actually implement a small, simple SAN in software with only existing Ethernet hardware. But this is rarely done because such SANs are very slow. Microsoft operating systems support it now. SANs with iSCSI are ideal for smaller- and medium-size businesses, but FC will remain strong in the larger enterprise systems.
NEW PRODUCTS FOR SANs HARDWARE Each day, the semiconductor and laser transceiver manufacturers have been addressing the FC and iSCSI SAN sector more and more, making it easier for box manufacturers to create affordable products. Here is just a sampling of the many recent products as an example.Intel recently introduced its TXN31015 and TXN3115 laser transceivers targeted at new 4-Gbit/s FC HBAs, switches, and RAID interfaces (Fig. 3). The TXN31015 is based on the compact Small Form Factor (SFF) Multi-Source Agreement (MSA), while the TXN31115 is based on the Small Form Factor Pluggable (SFP) MSA. Both operate at 850 nm. These units are sampling today and run $47 in sample quantities of 1000.
Picolight, a manufacturer of SFP optical transceivers for Ethernet and FC, recently announced the company's 40- and 80-km Ethernet transceivers. The PL-XPL-LC-L13 and PL-XPL-XC-E13 devices use 1310- or 1550-nm distributed feedback (DFB) lasers, respectively, to achieve the 40- and 80-km distances with 1GE over single-mode fiber. Both devices are available now and support the SFF-8472 Digital Diagnostics Monitoring Interface for optical transceivers. This enables manufacturers and systems administrators to monitor the performance of an installed network's physical layer. Picolight also has new 850- and 1310-nm vertical-cavity surface-emitting lasers (VCSEL) for 4-Gbit/s FC.
One leader in the FC chip business is Agilent. With an estimated market share of 50%, Agilent has a full line of FC controllers and related chips, as well as the SFP laser transceiver modules used in 1-, 2-, and 4-Gbit/s systems. A very new product from Agilent is the HPFC-5700A Tachyon DX4+ dual-channel FC controller for 1/2/4-Gbit/s systems. It is designed for use in FC HBAs as well as in the storage units themselves. It implements a state machine architecture rather than a processor, but it is fully programmable down to the register level to fit user needs. The chip also includes the new Data Integrity Field (DIF) feature for advanced error detection. This 8-byte field helps to identify data corruption during transmission. Sample quantities of the Tachyon will be available in May with production in the fourth quarter.
Another new Agilent chip is the HDMP-0540 22-port 4-Gbit/s loop switch. This device emulates a port bypass circuit by creating a loop between host, drives, or cascaded ports. It allows simultaneous transactions to occur between two non-busy nodes. This chip is expected to eventually replace conventional port bypass bus controllers because the loop switches scale better. The device can be cascaded to control up to 126 disk drives. Each port has an integrated serializer/deserializer (SERDES), input equalization, and on-chip 100- or 150-(omega) terminations. A two-wire interface similar to I2C manages, programs, and monitors status/failure. The HDMP-0540 is expected to cost less than $100 in volume with samples available in June and production in the third quarter.
Lots of chip manufacturers are paying attention to the 4-Gbit/s FC option. Broadcom recently introduced three new products that target 4.25-Gbit/s FC HBA, switches, and storage units. The BCM8426 is a 12-port FC arbitrated loop hub device that can be configured to connect FC to hard-disk drives in an intelligent bunch of disks (iBOD) configuration. The chip is designed for low latency. Each port has retiming or repeating capability to support the various configurations of back-end arbitrated loops.
The BCM8422 is a four-port FC arbitrated loop hub device that can also be configured as a dual bidirectional retimer/repeater. The BCM8421 is a single-channel, bidirectional retimer/repeater chip designed for FC switches and storage devices. All of these chips feature Broadcom's EyeOpener technology, which helps guarantee jitter performance. These chips also include Broadcom's advanced diagnostics, called Active Signal Integrity and Active Link Integrity, to help detect and recover from signal impairments at both the physical and data link layers. The Broadcom chips are made with 0.13-µm CMOS and feature small BGA packaging. The chips are sampling now.
LSI Logic recently introduced its LSIFC929XL and 949XL FC Protocol Controller chips. This full-duplex, dual-channel controller is designed for both initiator (server) HBAs and target (storage box) applications. The 929XL is a 2-Gbit/s FC device that adds a Data Integrity Field (DIF) to the protocol. The 949XL provides 4-Gbit/s performance with DIF. Just like a CRC and other error-detection schemes, the DIF provides additional end-to-end data protection in the store and access processes. The DIF algorithms are sometimes implemented in software, but these are hardware implementations in the chip that speed up the process. All DIF algorithms blessed by the FC T10 standards group are accommodated.
The 929XL features three on-chip ARM controllers, one for each channel and a third for management purposes. The chip interfaces via a 64-bit 133-MHz PCI-X bus. Each channel can achieve performance levels of 130,000 I/O operations per second in initiator mode and 85,000 I/O ops per second in target mode.
The device comes in a 456-pin PBGA housing and is pin-compatible with the previously available LSIFC929X non-DIF version. A JTAG interface is included. In addition to these chips, LSI Logic offers a complete line of FC HBA cards and related products.
PMC-Sierra is another recent contributor to the 4-Gbit/s FC progress. The PM8377 PBC 4x4G and PM8369 PBX 18x4G chips are four- and 18-port intelligent FC port bypass controllers, respectively. They are designed for enterprise-class disk enclosure applications and are fully compatible with PMC-Sierra's 2-Gbit/s PBC devices. PMC-Sierra also has a new four-channel SERDES device, the PM8386 QuadPHY 4 GFC, for FC fabric switches and HBAs. All of these parts are available now.
While there are few 10-Gbit/s FC or iSCSI systems in use or even available, they are on their way. More and more products are becoming available to make them affordable.
One example is Quake Technologies' new QT-2022 10-Gbit/s serial-to-XAUI physical-layer device. Designed for 10GE or FC systems, it's fully bidirectional and handles four 3.125-Gbit/s XAUI channels and a single 10.3- to 10.57-Gbit/s serial stream. The QT-2022 produces less than 0.7 ps of rms jitter and has a receive sensitivity of 10 mV p-p. The built-in receive equalizer recovers data over as much as 12 in. of FR4 PC board trace. The unit operates from a 1.2-V supply and has a power consumption of only 900 mW. The chip fully complies with the IEEE 802.3ae 10GE standard and the INCITS/T11 10-Gbit/s FC standards, as well as the standards for the XENPAK/XPAK/X2 and XPF MSA transceiver packaging.
One of the hottest products to emerge for new iSCSI systems is the Silverback Systems iSNAP2110 IP storage network access processor (Fig. 4). This device is an upgraded version of the earlier iSNAP 2100 introduced last year. It provides a fast and easy way to make iSCSI initiator NICs and target interfaces for storage units. The 2110 is designed as an iSCSI TCP/IP offload engine. It fully terminates all TCP/IP connections and implements upper-layer Protocol Data Unit (PDU) awareness. The result is performance of up to 220,000 I/O operations per second and a throughput of 440 Mbytes/s.
The iSNAP2110 is designed for the PCI-X bus and features two Gigabit Ethernet interfaces. It also allows implementation of upper-layer protocols such as RDMA. A key feature of this device is that it is firmware programmable and may be reprogrammed and updated as required. The chip comes in a 748-pin EPBGA package and dissipates 4 W. Samples and production quantities are available now.
As for the future, look for that anticipated growth and lots of new products. SANs users with Fibre Channel will no doubt settle in at the new 4-Gbit/s rate for the time being, but great progress is being made in both 10-Gbit/s FC and iSCSI. Lots more will come, such as the new Content Addressable Storage (CAS) systems on the horizon and, who knows, maybe even wireless SANs.
| Need More Information? | |
|
Agilent Technologies Inc. www.agilent.com/view/storage Broadcom Corp. www.broadcom.com Intel Corp. www.intel.com LSI Logic Corp. www.lsilogic.com/storage |
Picolight Inc. www.picolight.com PMC-Sierra ww.pmc-sierra.com Quake Technologies Inc. www.quaketech.com Silverback Systems Inc. www.silverbacksystems.com |
| USEFUL SOURCES OF INFORMATION ON SANs |
|
Books Clark, Tom, Designing Storage Area Networks, Addison-Wesley, Pearson Education, 2003 Poelker, C. and Nikitin, A., Storage Area Networks for Dummies, Wiley Publishing, 2003 Web Sites Fibre Channel Industry Association (FCIA) www.fibrechannel.org InterNational Committee for Information Technology Standards (INCITS) www.incits.org Internet Engineering Task Force (IETF) www.ietf.org SCSI Trade Association www.scsita.org Storage Networking Industry Association (SNIA) www.snia.org Technical Committee T11 (INCITS committee handling FC) www.t11.org |
