Wireless-local-area-network (WLAN) technology first became prevalent in the small-office/home-office (SOHO) market. In that forgiving environment, convenience often outweighs performance concerns. But as 802.11 technology extends to the enterprise, wireless-system designers and developers must face the enterprise market's performance-based realities. Enterprise WLAN networks are evolving to support business-critical data and voice applications, large numbers of network users, and diverse network elements. All of these elements have historically been part of the wired-Ethernet LAN. Because network managers and end users have long been accustomed to the high performance of the Ethernet, they increasingly expect uncompromised service quality and data throughput.
Despite the fast adoption rate of WLAN technology, comprehensive test metrics and methodologies for benchmarking wireless equipment are only now beginning to emerge. The traditional testing methods that were developed for Ethernet-based hardware and networks aren't applicable. They can't handle the increased complexity and differences in WLAN devices and end-user requirements.
Currently, there are no generally accepted test methods or metrics for evaluating the performance of wireless networks. This issue has led wireless-network designers and vendors to rely primarily on custom-built testing solutions. Such solutions attempt to mimic the conditions of a wireless network.
However, recent developments in WLAN testing promise to hasten the acceptance of 802.11 technology in the enterprise. In a series of three articles, we will delve into metrics and measurement techniques that have been specifically developed and defined for wireless networks. We'll also present a practical example of an enterprise application—voice over WLAN—and the metrics related to it. This first article reviews the end-user and network requirements that are unique to wireless networks. It also examines common types of WLAN tests while describing traditional WLAN-testing solutions.
Ethernet-device performance is essentially a measure of the packet-forwarding rate. In a wireless network, however, the performance measurement has to take into account several other factors. These factors include automatic data-rate adaptation, roaming, security, and quality of service (QoS). Most of the differences between wired and wireless protocols spring from the innate mobility of wireless users and the associated physical-layer dynamics.
Unlike wired LANs, which have a stable physical layer, wireless networks rely on an inherently erratic physical layer: air. Air is subject to random interference from the radio frequency (RF) that's emitted by 2.4-GHz phones, microwave ovens, radar, adjacent channels, and objects in motion. In addition, protocols that deal with mobility and open-air transmission simply don't exist in the wired world.
Compared to wired networks, WLANs are subject to a completely different set of end-user and network needs. What conditions define a real-world wireless environment in an enterprise? Network managers must guarantee network stability and performance for the end users' business-critical, bandwidth-intensive applications. Those applications require:
- Mobility and roaming: The mobility of the WLAN end user has a major impact on WLAN networks and the complexity of the 802.11 protocols. Mobility can result in throughput loss and service disruptions. For example, the position of a client device with respect to an access point (AP) is critical for maintaining throughput. As users move between access points (roam), they create random demand and location spikes. The roaming process is complicated. It requires the completion of multiple steps while the end user is in motion. At the same time, the network connection must be kept transparently available to the user.
- Scalability: Unpredictable user and traffic loads are the result of users joining and leaving the network randomly. For example, wireless airport networks carry unpredictable traffic loads. Three hours before flight departure, an AP in an airport waiting area might have one user. Three hours later, 1 to 200 users might all be trying to use the same AP.
In addition to the end-user characteristics, certain network aspects become increasingly important to measure in the wireless paradigm:
- Interoperability: Large enterprise networks often consist of thousands of access points and tens of thousands of clients. The diversity and widespread nature of wireless technology requires multiple vendors to work in a single network. Single users must work in multiple public and private networks. The need to test interoperability is therefore critical to ensuring the successful deployment of Wi-Fi equipment.
- Performance: Mission-critical applications must deliver reliable performance and QoS. Although 802.11 technology is progressing in terms of capacity, it still relies on a shared medium to provide connectivity. Wireless solutions must perform well. They also must implement and deliver sound QoS features, which support applications that require guaranteed service levels.
- * Security: Security requirements are more stringent for WLANs than for their wired counterparts. After all, wireless networks are susceptible to intruders, who can easily tap into the network with an antenna. If they're improperly implemented, 802.11i encryption and authentication protocols can be a significant drain on the performance of wireless networks. A client that has been authenticated in one place in the network must maintain authentication while roaming. The authentication process is lengthy and can disrupt a session if it isn't conducted properly. As a result, the security process can affect network performance, roaming times, and association times. Testing must characterize the effectiveness and efficiency of security implementations.
A variety of tests must be performed to make sure that wireless equipment meets the complicated requirements of wireless networks and end users. The complexity of the 802.11 protocol requires testing to verify network and device functionality, capacity, interoperability, and conformance prior to deployment. As vendors strive to deliver WLAN products that meet the diverse needs of both users and wireless networks, they usually perform the following test categories:
- Certification testing ensures that wireless equipment meets industry- or government-established minimum requirements like interoperability, safety, and emissions. In 802.11, this testing is driven by the Wi-Fi Alliance (WFA). The WFA is a vendor-based industry forum that is focused on testing and certifying product interoperability for the good of the consumers. The WFA has defined industry-accepted interoperability test suites. It also has contracted labs to perform certification worldwide. Once certified, a product is allowed to bear the Wi-Fi label.
- Performance testing ensures that the product meets expected network results. Performance characteristics are universal. But the mobility, roaming, and scalability factors are unique to the wireless world. Performance testing must measure the range of the wireless device and its throughput capability. In addition, it must account for unique wireless phenomena like the number and mobility of users, security schemes, and QoS. Devices must perform appropriately under fault conditions, such as a sudden surge in throughput or other unexpected event.
- Co-existence testing verifies that the product is versatile and robust in diverse environments. Performance is tested in the presence of interference to prove that a system can intelligently avoid interfering with other wireless systems. Examples of such other systems include analog wireless phones and wireless technologies like Bluetooth, Ultra Wideband (UWB), WiMAX, and ZigBee.
- Conformance testing validates that the product completely adheres to the specifications of the 802.11 standard. Complete conformance testing and protocol analysis is needed to verify that a device is compliant with the IEEE 802.11 specification. This development-level test is usually performed using sophisticated and expensive signal-generation and analysis tools.
Traditionally, there have been a variety of testing options for wireless-systems designers. Many can be described as home-grown or custom-built solutions. These solutions include isolated screen rooms for controlling RF interference, large open spaces for testing mobility, and off-the-shelf testing solutions that focus on point-to-point tests of the physical layer. The limitations in traditional wireless-testing platforms and solutions render them unable to evaluate the real-world performance of wireless networks in a laboratory setting. In addition, wireless protocols do not currently dictate testing methods or metrics. As a result, vendors have had to develop their own benchmarks. Because of the differences in each vendor's methodology and metrics, the traditional WLAN-testing solutions don't result in consistent outcomes.
Some of the typical testing methods used in the design of wireless equipment include:
- Isolated screen rooms for controlling RF interference: Effective wireless testing requires that the devices under test be isolated from random RF emissions. As shown in a comparison of test results performed in environments with and without RF interference, it's nearly impossible to acquire repeatable results in situations where RF interference can't be restricted (FIG. 1). When tests are performed in environments where devices under test (DUTs) cannot be shielded from RF interference, they lack accuracy and repeatability.
Many WLAN system developers balance the effects of RF interference by conducting tests in the wireless equivalent of a "clean room." This large screen room isolates DUTs from extraneous RF interference. Yet such screen rooms are incapable of testing real-world network conditions like mobility and roaming. Such conditions can't be tested in an isolated room. In addition, screen rooms can be expensive to construct and maintain.
- Large open spaces for testing mobility: 802.11 protocols require performance metrics and methodologies that can replicate the real conditions of wireless-LAN environments, such as mobility and roaming. Often, wireless design engineers try to simulate these conditions by renting or buying empty office buildings, homes, or even outdoor open spaces like football fields. Mobility and roaming are often tested by piling numerous mobile carts with Wi-Fi devices and pushing them down the hall of an empty office building.
In addition to the expense of such solutions, techniques that use large open spaces to test real-world wireless conditions are often unable to scale to multiple devices and users. Both scalability and automation are required to maintain a cost-effective testing solution. Finally, testing in large open environments is problematic because RF interference can't be controlled. Interference can be eliminated in a shielded test chamber. But the physical constraints of a chamber are directly at odds with testing network conditions like mobility and roaming.
- Off-the-shelf testing solutions: A variety of off-the-shelf test equipment is available that focuses on point-to-point tests at the physical layer. For example, signal analyzers and generators, which emit and analyze realistic 802.11 signals for Layer 1 conformance, are generally used in RF-isolated conditions. They are designed to operate in the 802.11 frequency range. Open-air black-box devices emulate WLAN clients or access points. They effectively "converse" with the DUT, performing MAC-layer conformance validation. Finally, network test devices can be placed in the network to generate "load" or capture and analyze the network. They evaluate the network in stress or exception conditions.
The issues with such testing solutions are mainly device-focused rather than network- or system-focused. WLANs should not be evaluated in specific components, but rather as a complete solution. Effective system-level test solutions must scale beyond single devices like clients and APs to the network itself. They must provide the ability to test the WLAN as a system and precisely analyze Layer 2 performance under various mobility and load conditions. If off-the-shelf solutions test only specific aspects of the physical layer, they can't accurately and efficiently characterize the merits of wireless networks as systems. In addition, the variability and control of the environment is unstable. This instability renders the test results invalid.
A newer test methodology is gaining momentum. This methodology addresses the shortcomings of traditional wireless testing methods. It replicates the WLAN network in a controlled, cabled environment by stabilizing the RF connection (FIG. 2). The DUTs are connected via a network of RF attenuators, combiners, and switches. This approach isolates them while eliminating open-air interference (FIG. 3).
Solutions that use a controlled, cabled RF environment eliminate the need to design, build, and maintain home-grown test beds and costly RF screen rooms. Such platforms have programmable RF test beds that allow the user to configure an entire WLAN network on a benchtop chassis. Software automates configuration and analysis. In addition, the user can thoroughly evaluate WLAN equipment and networks under varying conditions and traffic patterns. He or she can then precisely analyze the results. The clients and APs that comprise the network are actual clients and APs of the user's choice.
There are numerous benefits to this type of WLAN testing methodology, such as repeatability. Repeatable, consistent test outcomes are possible only when the system under test is isolated from RF interference. This allows the repeatable emulation of real-world wireless-networking conditions, such as mobility and roaming. Such conditions are difficult to test in screen rooms.
Another benefit is scalability. To create stringent performance test conditions, traffic from hundreds of devices and users can be effectively emulated. Using a variety of wireless traffic, multiple devices also can be simultaneously tested for performance, quality of service, and security under stressful data traffic. The new methodology also provides automation. Programmable solutions reduce cost and enable quicker time-to-market by automating the testing process.
Test devices that use a controlled, cabled RF environment have dramatically shifted the way that wireless-systems designers develop and test networks. These devices give designers the flexibility to test real-world wireless-networking conditions, such as mobility and roaming, in a repeatable manner. The new platforms automate the testing process, providing system designers with repeatable test outcomes and a test process that cost effectively scales to multiple clients and access points. These off-the-shelf solutions are more advanced than point-to-point testing systems that test only single devices. The new solutions allow wireless networks to be tested as dynamic, scalable systems. In addition, they have been adopted by many of the major chip manufacturers and equipment vendors.
The test methodologies that control RF interference in a cabled environment have gained industry momentum through their ability to hasten the adoption of WLAN in the enterprise. The wireless industry has recognized the need for the creation of standardized, industry-wide test metrics and methods. It also sees the need to develop advanced test methods and solutions that consider the unique needs of wireless networks and users.
An important step toward achieving this goal was the formation of the IEEE's 802.11T task group. Its mission is to define performance metrics and test methodologies for evaluating wireless-LAN equipment and systems. This group's efforts will create a foundation that enables accurate performance comparisons of WLAN devices. 802.11T will support enterprises by helping them predict WLAN performance in large deployments without extensive measurement.
Initially, the group will focus on developing appropriate test metrics, such as device throughput, roaming time, and rate versus range. It also will provide enterprise network managers with a uniform set of planning benchmarks based on device characteristics, network layout, and usage parameters. Benchmarks created by the group will help system designers identify and understand the impact of variables like system noise and interference on network devices.
The scope of the task group does not include a rating system. In other words, the 802.11T task group will not attempt to define what performance level is acceptable. Instead, the goal is to develop a set of standardized methods and tests that help in measuring and predicting performance.