Expectations for devices in the rapidly expanding smartphone market segment are high. In addition to reliable voice calling, they must support data services over a range of cellular technologies (GPRS/EDGE, WCDMA/HSPA, CDMA) across multiple bands to enable global roaming as well as Wi-Fi and Bluetooth.
Users expect support for location-based services, which increasingly will mean hybrid location technology solutions, such as GPS and GLONASS, used in conjunction with other techniques such as Wi-Fi positioning. Devices that include these capabilities and sometimes more we refer to as multimode.
To complicate things further, in the near future, devices with LTE capability will need to support multiple additional bands to enable global roaming, including 700 MHz and 2.6 GHz. This requirement is in addition to the existing multiband support needed for legacy cellular technologies.
While multiband, multimode devices are what the consumer wants, the complexities of these devices present huge challenges for those who must design and manufacture them. Since the industry-standard tests for device certification are intended to ensure only basic conformance with the specifications, additional testing becomes ever more critical to guarantee that the full range of device functionality works well across all bands and modes. The most cost-effective approach to this testing need combines a lab-based simulation environment with selective field testing.
Challenges of Multiband, Multimode Testing
Device testing decisions are all about trade-offs. More testing equates to better user experience, but it also increases cost and time to market. As devices are expected to support more bands and modes, the problem gets worse. Devices are more likely to experience issues when multiple bands and modes unexpectedly interact in a negative way while testing for this takes longer and becomes more expensive.
The testing strategy becomes an optimization problem where three objectives need to be carefully balanced: good user experience, time to market, and capital and labor costs. The formula simplifies to something like this: We are willing to invest $X in testing to ensure our customers don’t return their devices, but we must keep our testing cycle below Y weeks. What test strategies can address this?
Testing Strategies and Analysis
The four most common testing strategies, which often are combined, are field test, test network, lab simulation, and doing nothing. Figure 1 summarizes the pros and cons of the various testing strategies.
Figure 1. Test Strategies and Trade-Offs
Field Testing
Devices are taken out into the real world and tested on live networks. Field testing is the most realistic form of testing and very good at verifying interoperability between a device and the network. However, since the real world is continuously changing, there is no way to reproduce the exact scenario at two different points in time.
This is an issue when it comes to reliably identifying problems and their resolution, leading to increased testing cycle time and delayed time to market of new devices. Furthermore, testing the capability of devices to roam on other networks and support all possible bands and technologies adds to costs.
• Multimode Testing: Must pick appropriate drive routes or field test points to ensure all device modes are tested.
• Multiband Testing: Must send devices and people all over the world to ensure that all supported bands are tested.
Test Networks
Testing devices with real infrastructure set up in labs is good for interoperability testing, somewhat reducing the testing cycle time compared with field testing alone. However, when radio signals are piped around a lab, the resulting test environment is far from representing the real world.
Making changes to the network configuration can be difficult and time-consuming, and channel impairments such as noise, multipath, fading, and interference typically are nonexistent. Although it is possible to use channel emulation test equipment that enhances the realism of the signals, this is very difficult to do accurately. As a result, test networks tend to be expensive and are used mainly for interoperability testing.
• Multimode Testing: A test network requires real infrastructure for all supported technologies. GNSS positioning testing requires the satellite signals to be piped in from an outside receiver, which can be very problematic. A wide variety of Wi-Fi access points needs to be set up to cover all the frequencies and channels supported by the various 802.11 standards.
• Multiband Testing: The cellular infrastructure must be set up for all supported bands. While this is possible, it usually is extremely expensive.
Lab Simulation
Lab-based test equipment, typically integrated into automated solutions, can be used to simulate the real environment. Lab-based simulation has the advantages of test automation and repeatability. Device problems found using lab simulation solutions can be debugged and fixed more easily since test scenarios can be replicated exactly.
Although lab simulation solutions widely vary, the best ones can accurately simulate cellular networks, channel impairments, and additional modes such as GNSS satellites and Wi-Fi hotspots. However, no matter how well the test equipment simulates the real network, it never is going to actually be the real network. Consequently, lab simulation solutions will always have some limitations when it comes to interoperability testing.
• Multimode Testing: Lab simulation solutions must be complete enough to support simulation of all relevant modes.
• Multiband Testing: Testing multiple bands with lab simulation solutions usually is as easy as setting a parameter in the test executive software.
Doing Nothing
There is an option to simply not test certain functionality, and this approach is quite common. While this approach may be appropriate for some less important functions or features, it generally carries significant risk.
It may be inexpensive and quick, but what happens when the device performance does not measure up to customer expectations? It may work one, two, or more times. But the complexity of today’s devices makes the likelihood of issues extremely high, and this risk will only increase going forward.
Cost-Effective Solution
The optimal solution often is a combination of lab simulation and small-scale field testing. These two approaches compliment each other well (Figure 2).
Figure 2. The Benefits of Lab-Based Testing(To view larger image click here.)
Lab simulation also can provide a race track on which device performance can be easily benchmarked and analyzed. Multiband, multimode testing is particularly well suited to lab simulation because the increased test burden from band and mode proliferation can be mitigated by automation.
Since lab simulation is not the real network, some testing in the field still is needed to ensure interoperability with the live network. Using lab simulation to quickly identify and fix as many issues as possible and perform device benchmarking before embarking on field testing keeps overall cost down, compresses time to market, and helps to ensure a good user experience.
Lab simulation also can be used to find problems earlier in the device development cycle. For example, some leading operators require device manufacturers to conduct specific tests using lab-based simulation prior to submitting devices for acceptance. While this may increase the overall capital investment in test equipment, the costs can be more than recouped by getting devices to market faster with fewer problems.
Four Key Characteristics of Lab Simulation
We have established that cost-effective testing of multiband, multimode devices requires lab simulation, but lab simulation solutions in the industry vary widely. So what are the most important characteristics of lab simulation? Here are some key characteristics to consider:
Automation
Test scenarios should be easily configured to closely mimic the real-world environment. In many cases, this allows for known issues to be targeted and reproduced. One key characteristic to consider is the level of effort needed to customize a scenario. Most commercial implementations fall into one of three areas:
• Parameter Manipulation: Test automation is customized by adjusting test parameters. Although this type of automation generally is very easy to configure, the test scenarios are not very flexible.
• Coding, or Scripting, Required to Customize a Scenario: Test scenario automation can be customized, but it requires a programmer to create or modify test scripts. This type of automation is only appropriate when knowledgeable programming resources are readily available.
• No-Code Customization Using a Point-and-Click Graphical User Interface: This type of automation minimizes the time needed to customize scenarios. No specialized programming knowledge is required; users simply interact with a GUI to set up the test automation and customize test scenarios.
High Cell Density per Network Emulator
To simulate the real network, many cells must be simulated at the same time since devices are continuously reselecting and handing over between multiple cells. For that reason, it is important to consider the cell density of the network emulator used in a lab simulation solution and the number of radio access technologies (RATs) it can support. Solutions that require one network emulator per cell and RAT tend to be very expensive, and overall costs will decrease as more cells are included in a single instrument.
Simulation of the RF Channel
Key elements of real-world simulation are the physical-layer impairments such as noise, multipath, and fading. Many problems observed in the real world can never be replicated by lab simulation of the network infrastructure alone. Accordingly, it is important that all signals—whether they come from the cellular network, GNSS satellite, or Wi-Fi access point—have the option of being impaired with realistic channel models.
Simulation of GNSS, Wi-Fi, and Other Modes
The key to testing multimode devices is multimode test environment simulation. Testing every mode supported by a handset may require more than one lab simulation solution, but some solutions on the market can support all the common cellular technologies across multiple bands, together with GNSS (GPS, GLONASS) and Wi-Fi.
The following case studies illustrate how lab simulation plus small-scale field testing can be used to prevent some common user-experience issues with multiband, multimode devices.
Case Study 1: Mobility
Users typically have high mobility within both home and foreign networks. It is critical for devices to operate and perform seamlessly across these networks.
Example Problems
• Frequent voice call drops when handing over across cells.
• Unable to set up voice/data call in urban environments due to quickly changing cell power levels.
• Poor data performance while roaming across foreign networks.
• Device locks up when powered on after airplane travel to another country due to a system selection issue.
What Is Going on Here?
Mobility performance is critical when many cells are visible to the mobile device and multipath conditions cause signal power levels to fluctuate quickly such as in urban environments, making multipath mitigation and cell power measurement algorithms very important. Cell power measurements are complicated by multiband environments because devices must use compressed mode, allowing them to measure multiple frequencies with one receiver. Getting these algorithms to work optimally is very challenging due to all the inherent trade-offs, so today’s commercial devices have widely differing performance under these scenarios.
The Radio Resource Management algorithms are intended to ensure that the device chooses the best available cell at all times. However, in attempting to optimize for variables such as battery life, the implementation of these algorithms can compromise the performance of cell selection and reselection. The algorithms also are highly reliant on the device making accurate cell power measurements; if power measurements are inaccurate in certain bands/frequencies, mobility performance will suffer.
When a device is powered on in a foreign network, it must correctly select the preferred network with which the home operator has a roaming agreement, often requiring the device to cycle through several different networks. If this type of mobility has not been fully tested, a device may lock up in the worst case, rendering it unusable until the battery is removed for a hard reboot.
How Can These Problems Be Prevented?
Stage 1: Utilize an automated lab simulation solution to run the device through a variety of mobility scenarios. For voice, key performance indicators (KPIs) such as call setup failure and call drop rate should be benchmarked and monitored. For data, KPIs such as throughput and latency are very important. For maximum effectiveness, lab simulation testing should be pushed as far back in the development cycle as possible; regression testing at various integration points also is important.
Stage 2: Field-test devices in drive test routes with challenging environments such as cities, in representative target markets, and in foreign networks where customers are likely to roam.
While the testing in Stage 1 should dramatically reduce the number of issues found, more may still appear. When issues have been identified and characterized, a representative scenario that showcases an issue should be created for the lab simulation solution in an attempt to catch future problems earlier in the device life cycle.
Case Study 2: Service Interaction
Even if all the modes work well in isolation, there is no guarantee they will work well when used at the same time. The goal is to prevent the interaction of multiple services from impacting user experience.
Example Problems
• Audio blanks out or clicks when handing over between cells.
• Data throughput performance decreases dramatically when GPS is used to get position fixes.
• E911 position fix requirements cannot be met because A-GPS position fix fails when a voice call is active and a device is handing over between cells.
• Incoming e-mail causes application download to stop.
What Is Going on Here?
When multiple modes are used at the same time, unexpected issues tend to occur for three reasons: RF, CPU, and protocol interaction.
Services often share the same RF antennas and receivers so dynamic sharing of these resources during simultaneous usage is critical. Another potential problem is RF leakage from one subsystem interfering with other subsystems.
When users are on a voice call, streaming videos, receiving e-mail, and making position fixes at the same time, CPU utilization can become a bottleneck.
The most common source is flaws in the protocol stack that can occur at multiple layers. In a scenario where a user calls 911 while running away from an attacker, that call will automatically trigger an A-GPS positioning session in the United States to enable the emergency operator to determine the caller’s location. However, as the user is running, their device hands over between a WCDMA (3G) and GSM (2G) cell. Now voice call, positioning session, and Inter-RAT handover collide at Layer 3 in the protocol stack, and the positioning session never goes through.
As a result, the phone falls back to stand-alone GPS and Cell-ID location techniques which are much slower and less accurate. The degraded position location capability can be a life-threatening problem in this case. Although most protocol service interaction issues are not this serious, they nonetheless have the potential to make users very unhappy.
Conclusion
Consumer demand is driving ever-greater complexity in the multimode, multiband smartphones of today and tomorrow and the services they are expected to support. To ensure that increasing complexity does not result in decreasing quality of the end-user experience, a comprehensive test plan is required. Using an optimal mix of lab simulation and field testing can help control the time and costs associated with the additional testing, minimizing its impact on device time to market.
About the Author
Brock Butler is a product segment manager at Spirent Communications and responsible for UMTS location test solutions. He holds a bachelor’s degree in electrical engineering from Villanova University. Spirent Communications, Wireless Division, 541 Industrial Way West, Eatontown, NJ 07724, 732-578-2372, e-mail: [email protected]
February 2010