Certification Testing of Wi-Fi Mobile Devices

Figure 1. Growth Forecast for Wi-Fi-Enabled ASDs

Unfortunately, the unique requirements of these devices were not considered when the original IEEE 802.11 standard was specified in 1997. ASDs are not networked devices in the traditional sense and lack a common interface for control, operation, or testing.

Each device has a unique user interface such as a keypad or touch screen and may use different security mechanisms. These platforms typically do not support off-the-shelf traffic generators and performance measurement tools.

Additionally, the application processors used by ASDs typically run at much lower speeds, limiting the overall performance capabilities when compared to a PC. As a result, new testing methodologies are required in some cases to verify specific features.

For companies, test labs, and industry groups like the Wi-Fi Alliance that are testing the interoperability of these new products, this becomes a challenge of scope that clearly needs to be addressed. The continued lack of a cost-effective standard testing methodology can potentially restrict the growth of this device category. Without a new way to streamline the process, current methods of manual testing will result in very high testing expenses followed by higher product costs and lower product quality.

Wi-Fi Alliance Certification
The Wi-Fi Alliance is a global, nonprofit industry association of more than 300 member companies devoted to promoting the growth of wireless local area networks (WLANs). The alliance provides its members with testing and certification programs to ensure the interoperability of wireless products based on 802.11. Since the certification program began in 2000, more than 3,300 products have been designated as WI-FI CERTIFIED .1

By validating the interoperability of Wi-Fi devices from different manufacturers, certification ensures a positive user experience. It is a critical testing step in bringing Wi-Fi products to market.

When the Wi-Fi Alliance certification process and programs were developed, the vast majority of 802.11 clients were PC-centric network equipment or network clients, and certification testing adequately addressed them. In subsequent years, the number of non-PC-centric Wi-Fi devices has grown significantly along with a dramatic increase in nonstandard 802.11 devices.

Embedding Wi-Fi in Mobile Devices
Vendors in these emerging market segments are quickly discovering the complexity involved in incorporating Wi-Fi into a product design and the difficulties of testing connectivity. For example, mobile phones recently consisted of one radio supporting one to four radio bands. Today's advanced mobile devices are becoming more complex, with dual-mode phones containing up to six radios operating with up to 12 radio bands.

Besides the challenges in developing the hardware, software complexity to support new features such as Wi-Fi, 3-D graphics accelerators, GPS, FM, multimedia processors, Bluetooth, mobile digital TV, and near-field communications is increasing exponentially. Ensuring that all of the features of mobile devices function correctly and are interoperable is a significant challenge, one that requires a disciplined and well-defined testing methodology.

Testing ASDs in a PC/AP World
Since ASDs have many forms and unique characteristics, they require a different testing approach than PC-based client devices. The increasing number of ASDs as broad and diverse as televisions, phones, and video gaming products required the creation of a manual, customized test plan for each new ASD.

As a result of these challenges, the Wi-Fi Alliance recently introduced a new test engine methodology that allows station tests to be streamlined for any type of product. The key component is a framework for client testing that is extensible to Wi-Fi clients of any form without compromising the integrity of the comprehensive series of Wi-Fi tests developed over the years that ensure device interoperability.

The Test Engine Methodology
In defining the new test engine methodology, the Wi-Fi Alliance ASD technical task group proposed a framework referred to as the Wi-Fi Alliance Test Engine, or simply the test engine, which includes three key areas:

Station Configuration and Control
The methodology defines a standard interface that allows vendors to design stations that can be configured for the tests and then stimulated to perform the actions required. Because the devices tested lack standard user interfaces such as a keyboard or a mouse, the interface is defined independent of the vendor's implementation.

Test Traffic Generation
A large part of interoperability testing requires the transmission and reception of traffic by the DUT as a way to measure throughput, connectivity, and functionality. The DUT station initiates or terminates this traffic, and the test engine methodology provides a traffic-generation element for integration into the DUT as well as guidelines to create this capability on the station. The traffic generator is designed specifically for devices with limited memory and processing power.

Test Results Measurement
A methodology is required to measure and report results. The test engine uses the Azimuth ADEPT-WFA Tool and test automation to enable traffic analysis and reporting independent of the capabilities of the station DUT. The methodology includes intelligent capture and analysis techniques to validate connectivity, performance, and functional testing.

In addressing these areas of station testing, the test engine provides a scalable approach suitable for the growing variety of Wi-Fi-enabled products. The resulting test solution required to certify products using the test engine methodology includes five key components (Figure 2):

Figure 2. Wi-Fi Alliance Test Engine Architecture

1. DUT Software: The Wi-Fi traffic generator (WTG) software and DUT agent reside on the DUT. The WTG software produces the Wi-Fi traffic used in the tests, and the DUT agent software enables communications to configure and control the DUT.

2. Control PC Software: The control software implements the parser for the control API to convert commands to a USB or other proprietary interface, enabling vendors to control devices through their native control protocol or command set.

3. Capture Engine: The capture engine provides the mechanism to capture accurate, time-stamped, promiscuous wireless and wired traffic for analysis.

4. Test Engine Management Tool: The management tool controls and configures the DUT via the control PC and executes the test plan. In addition, the management tool controls the test-bed equipment, including access points (APs), to ensure a stable environment.

5. Alliance Test Bed: The test bed selects the clients and APs used as reference devices for interoperability, helping vendors verify their products in a multi-vendor marketplace.

The Wi-Fi Alliance provides sample software in source-code form to its members. This software implements both the DUT software and the control agent on the control PC, including an implementation of the DUT control API. It is designed to prepare DUTs for Wi-Fi certification.

Software to run the tests (certification scripts) is provided by Azimuth Systems and orchestrates the DUT, control PC, capture engine, and test-bed devices and reports the results of the tests.

Implementing the Test Engine in an ASD
Assumptions made within the Wi-Fi Alliance test engine code regarding capabilities, peripherals, and user interface can create challenges in implementing the DUT test engine on non-PC-based devices. The capabilities of embedded devices such as mobile phones can vary greatly from product to product.

The reference target device for the sample software from the Wi-Fi Alliance is a Linux single-board computer that connects to the control PC via a sockets interface. Many ASD devices will not have a standard sockets interface because they lack a networking stack.

Additionally, some devices may have a sockets interface, but it might not be compatible with the Linux operating system's implementation of sockets. As a result, the out-of-band communications mechanism must be adapted to accommodate the interface mechanism available for the DUT.

Implementing the Test Engine
Once the difficulties associated with implementing the test engine methodology have been overcome, the first step is porting the Wi-Fi Alliance software for configuration and traffic generation. The alliance provides the reference implementation source code of these components to assist in installing the software on the device.

Because of their unique user interfaces, ASDs typically are more difficult to configure for testing than PC-centric devices. With limited input options, manually configuring most ASDs for the required test cases would be tedious, if not impossible. The test engine configuration software standardizes programmatic control of the DUT, including the WPA-PSK and WPA2-PSK security modes and different Extensible Authentication Protocol (EAP) types if the DUT supports enterprise authentication.

Each device also may have a unique method for sending configuration commands such as USB, COM, IrDA, and Ethernet. The test engine methodology uses the concept of a DUT control agent (DUT-CA) to abstract the control interface, allowing the user to implement configuration commands over interfaces supported by the DUT. The DUT-CA is the user's intellectual property, securing the control interface and enabling commands to be kept confidential.

The DUT-CA receives each command in the standard format specified by the test engine essentially ASCII characters sent over a TCP socket. Each command then is translated into proprietary DUT-specific commands and executed, after which the DUT-CA may return a result. The user must test each command in the specification with different parameter settings to ensure that the DUT is configured properly after the completion of each command and that it returns correctly formatted results.

To cause the DUT to perform an action, the test manager issues an appropriate command. This command passes through the control network and arrives at the PC that runs the ASD control agent, which ultimately controls the DUT. The DUT then executes the command and performs the action required for the test.

While this architecture originally was designed for certification testing of ASDs, vendors also can use these tools for performance and certification testing of both ASDs and traditional Wi-Fi devices.

Certification Testing
The test engine architecture automates test plans through scripts and libraries. The steps accomplished by the test script, shown in Figure 3, allow vendors to:
Operate the DUT through the API defined in the test engine methodology. Generate traffic from the test manager PC or DUT through the ASD WTG. Capture the resultant activity. Analyze the results and determine whether the device has passed or failed the test.

Figure 3. Certification Test Script Flow Chart

Many of the certification test cases require traffic capture and analysis of packets to and from the DUT to determine test results. By capturing packets on the 802.11 and Ethernet interfaces and performing state analysis on those packets, the capture engine can: Verify the number of test packets successfully sent and received by the DUT during the test. Measure the throughput performance of the data transfers. Validate that the content of the packets is in accordance with test plan requirements.

The capture engine used in running the test is the Azimuth ADEPT-WFA; the company also provides software automation to control and run the test. The AzCert• Test Engine WPA2 Test Script performs the certification on the test engine DUT. It also automates configuration of the DUT and traffic generation from both the DUT and the control PC and captures test packets using the ADEPT-WFA. The script automatically determines the throughput achieved for all the traffic tests and if those test cases have passed or failed.

The AzCert test can run as a semi-automated or fully automated script. While the same test script and license are used in both cases, not all features are available using the semi-automated script.

In the semi-automated approach run on the user's PC, the script prompts the user to configure any test-bed APs, stations, and security servers required for the test. The automated approach is run on a dedicated server, and all test steps and analysis can be batched and performed without operator assistance.

Summary
As the number of ASDs continues to grow, it is critical for companies, test labs, and industry groups to have scalable test methodologies and tools to support interoperability. The Wi-Fi Alliance test engine methodology provides such an approach for station testing of ASDs, which dramatically improves the flexibility and scalability of the certification program to support both non-PC and traditional PC devices.

The benefits of this new methodology include the following: A standard approach that simplifies device implementation and can be used by different vendor platforms. A standard platform to run the certification test, which can reduce the cost of certification at a Wi-Fi Alliance authorized test lab. The ability to run the same test that will be performed at the lab, enabling reliable precertification. The ability to test multiple types of devices simultaneously in unattended mode via a fully automated certification test.

Despite the challenges associated with implementing this new methodology due to the breadth of devices and domain knowledge required, the new test engine is expected to be cost-effective and efficient for vendors and test labs to implement. The revolutionary approach standardizes testing of the increasing number of Wi-Fi-enabled devices, propelling the continued proliferation of interoperable, Wi-Fi-certified products.

Reference
1. “Wi-Fi Certified Makes It Wi-Fi,” Wi-Fi Alliance, September 2006.

About the Authors
Graham Celine is the senior director of marketing at Azimuth Systems. He has been in the high-tech industry for more than 15 years, joining Azimuth after a 13-year career focused on data networking. His previous experience includes positions with LANNET (later acquired by Lucent) and Avaya's Multi-Service Networking Group. Mr. Celine received a BSc in electrical engineering from University of the Witwatersrand. Azimuth Systems, 31 Nagog Park, Acton, MA 01720, 978-263-6610, e-mail: [email protected]

Justin Helmig, director of product management at TapRoot Systems, has more than 10 years of experience in the wireless industry including business, marketing, sales, and technical roles. Before joining TapRoot, Mr. Helmig was employed by Texas Instruments Strategy and Corporate Development and Sales, General Instrument, and Qualcomm. He earned a B.S. in computer engineering from Clemson University and an M.B.A. from Kenan-Flagler Business School at the University of North Carolina. e-mail: [email protected]

Mark Ranta is director of WLAN development at TapRoot Systems and a graduate of the Georgia Institute of Technology with a B.S. and an M.S. in electrical engineering. He is a 15-year veteran of the wireless industry including more than six years of experience with 802.11 technologies. Prior to joining TapRoot, Mr. Ranta worked at Texas Instruments, Ericsson, and TRW Space and Electronics Group. e-mail: [email protected]

TapRoot Systems, 3000 Aerial Center Parkway, Suite 103, Morrisville, NC 27560, 919-465-2266

FOR MORE INFORMATION on the Wi-Fi Alliance certificationwww.rsleads.com/706ee-184