The importance and pervasiveness of software are driving all types of networking and computing systems. Systems developers are finding that with the right software they can leverage Moore's law to deliver more functionality than a dedicated ASIC with comparable performance.
Witness, for instance, the evolution of the PBX from a hardware-based system to a software application that can be installed on a general-purpose computer. This trend is driving a greater need to automate the software test lab with an eye on the most time-consuming and manual processes in the test process: the management and configuration of the growing number of computers used in the testing.
Consequently, highly trained employees and a significant amount of hardware and software are underutilized for hours or days depending on the complexity of the system. Add to this the difficulty of managing this process in collaborative test labs. These labs may depend on off-site and offshore colleagues, partners, or customers who must duplicate this testing infrastructure or wait for onsite personnel to configure the systems.
At the same time, technology and economic pressures have amplified the ever-present pressures to do more with less and shorten test cycles, spawning an increased interest in a solution to the configuration challenge.
But new capabilities now are available to target automating the configuration process, dramatically reducing the amount of time required and opening up the test lab in a secure way for offsite developers and customers. The potential impact of these systems can be dramatic. To help explain the functionality and the results, a large maker of IP multimedia subsystem (IMS) software that automated its performance and interoperability test lab will serve as an example.
The Test Environment
The company's IMS software is designed to deliver multimedia and voice services across wireline and wireless networks. The software is deployed on Sun Netra servers, and the test equipment and ancillary devices include a mix of electronic equipment and computer-based applications from multiple vendors.
The company faced the same challenges as would any software company developing an application: support for a large number of computers, frequent new releases, heterogeneous test environments, and complex computer configurations.
With each new product release, the configuration problem grows exponentially. Figure 1 shows the series of sequential steps, ordered from the bottom to the top, that are required. Each step is assigned an average percentage of the effort required for the entire process.
The absolute time required for the complete process to be performed will vary from platform to platform, operating system to operating system, and application to application. Typically, it requires anywhere from four to eight hours although it's not unheard of for a configuration of a complex system to take several days. Any lab that must test computer-based products faces the accelerating problem of testing new releases against a growing number of existing releases and possible deployed environments, each requiring a new server configuration.
In the example IMS test lab, the engineers must verify interoperability and performance levels for a collection of integrated products, requiring a computer setup process that is repeated as many as 224 times per release test cycle. The process includes the following elements:
1. Approximately 14 computers, along with various switches and traffic generators and analyzers, make up a specific test topology, and each is reset to a known state just prior to running required tests. The computers are configured with standard product releases for a given deployment scenario. Different traffic patterns are generated to emulate real-world conditions.
2. The process is repeated for each of approximately 16 deployment scenarios. These variations include all the possible product release permutations the new release will be required to interact with once released into production.
3. Whenever a new release of a product has completed development, about six to eight times per year, steps one and two are repeated.
In this lab, each test must be run on a specific test topology with known configured elements. Testers use traditional automated test scripts to start the test and initiate traffic patterns with test suites executing overnight or over an entire weekend in some cases.
However, the testers must first set up the test topologies and manually restore complete computer configurations, including operating systems, patch levels, applications, product provisioning parameters, and test data, that are required for each automated test script. It takes a tester an average of three hours to perform the configuration steps for each computer in the topology.
Once the tests are completed, the results are queried to determine if the test has produced the expected results. While the test-results investigation is performed, the topology is maintained, and no other tests are performed. This process can consume two days of a tester's time for each topology.
Key Challenges
In a lab environment where products and systems are growing in complexity, it becomes exceedingly difficult to perform all mandatory and desired testing within reasonable cost and in the required amount of time. As a result, the cost of testing increases, and the testing capacity and quality decrease.
There are several key challenges in our example lab:
• Manual and time-consuming computer configurations are bottlenecks to automated test scripts. Each test topology setup requires two testers working manually for 2.5 workdays before any automated test can be run.
• The lab is significantly underutilized due to lengthy, manual reconfiguration periods.
• There are no efficiencies of scale. There is a linear relationship between the number of new product releases and the time and cost of testing each required topology.
• Remote collaborators in an offshore facility depend on their onsite counterparts to set up test topologies for them.
Solution Alternatives
To complete test cycles within given time constraints, the options are to add test beds and testers, reduce the quality of testing, or improve the test process so current resources can handle increased test throughput.
A new technology is becoming a popular choice for quickly creating computer configurations in production environments: virtualization. However, in a test environment where lab-configured systems must exactly replicate deployed systems, virtualization is not an option. The behavior and performance of virtual machines are not identical to those of physical machines, which eliminates virtual machines as a viable option for improving the process of configuring the DUT.
For the test lab, the required characteristics of a solution are the following:
• Configure computer equipment with operating system, applications, and test data.
• Provide a programmatic interface to support test automation scripts that select the configuration target and provisioning options and perform the configuration process.
• Store and set up any number of saved configurations on demand.
• Not require any deviation from the installation and upgrade procedures when creating a new configuration.
Complete transparency to the product under test. Do not incorporate any additional software in the environment under test.
• Able to scale as new computers are added to the test environment.
• Not require the use of the test topology for test-results analysis, allowing other tests to be performed while analysis takes place.
• Be cost-justified by measurable savings in current test resource costs.
Automated Computer Configurations
A new class of solutions available today automates the computer setup process. It has a flexible architecture that accommodates high-end UNIX, Linux, and Windows platforms. The solution incorporates a software application managing storage infrastructure made up of a storage device for the computer image archive and a storage area network (SAN) or LAN switch that connect to each of the managed computers and the storage device. Computers to be managed are connected to the solution using a host bus adaptor (HBA) or preboot execution environment (PXE) card.
Figure 2 represents the deployed solution architecture for the example test lab.
The testers in the example lab initially configure each of the 14 computers as usual. Each computer is connected to the Fibre Channel network through an HBA, which provides standard disk I/O capability over a serial fiber bus. Using the configuration management graphical user interface (GUI) or application programming interface (API) commands, the testers save the computer configurations to the storage array that, in this case, provides 7 TB of disk storage.
Important features of the solution for this lab include the following:
• Restores the mirrored disk drives that are in the servers hosting the product under test. The solution reconstructs the mirrored contents of all the disk drives on each server with the specified images and then reboots all the servers using the new images. The solution restores mirrored drives simultaneously.
• Configures multiple computers at the same time. In an environment where a test topology includes multiple computers, the capability to perform concurrent configurations exponentially increases the benefits of the solution.
• Restores a computer image to a different computer than that used to create the image. This feature provides several important process advantages, including the capability for one computer to be used to create master images for other computers of the same type and new computers that are added to the solution.
Additionally, computers dedicated to test-results analysis can be used to access mounted (used virtually from the storage array) images. This aspect frees the computers in the primary test environment to be used for continued testing.
In summary, the key benefits of this automated solution for the IMS lab include the following:
• The manual process of configuring computers is replaced by a completely automated process that eliminates the previous two-day bottleneck.
• Off-site and offshore personnel can remotely manage and control computer configurations.
• New release configurations are easier to create by recalling an existing configuration, performing required changes, and saving the new variant.
• Test results can be saved and recalled on any computer, allowing analysis to be delayed or performed on alternative systems while automated tests continue uninterrupted.
• Existing computer resources are optimized and leveraged, allowing additional tests to be performed using the existing test investment.
Quantitative Results
The automated configuration solution has achieved two key process improvements for the IMS lab. First, the amount of time needed to perform each test has been greatly reduced by decreasing the amount of time required to reconfigure the computers and set up the test topology. Secondly, the capability to perform off-line analysis for each test further shortens the total test cycle as shown in Figure 3.
Quantitative benefits for the lab can be summarized as follows:
• The average setup time for each computer was reduced from three hours to 12 minutes or just 7% of the time.
• The capacity of the lab to support release test cycles increased from three to eight or more releases per year.
• The amount of equipment required to support the company's release development schedule decreased from two sets of equipment to one, eliminating a $700,000 test bed and a $150,000-per-year tester to configure the equipment.
Conclusion
As the reliance on sophisticated software applications increases, more manufacturers will find their product development schedules increasingly at the mercy of the software test lab and its manual configuration process. As the IMS example shows, test lab automation offers a way out of this scenario without compromising test performance as a virtualization-based solution would. Automation is customized to the needs of software test labs and increases productivity and product quality while reducing costs.
About the Authors
Roberta Gonzalez co-founded EdenTree Technologies in 2002. During 20-plus years of experience in the communications networking industry, she has held vice president and director level roles in marketing and business development at Spirent Communications, Hewlett-Packard (Agilent), and Network General. Ms. Gonzalez also has delivered technical seminars throughout the United States, Europe, Asia, Australia, and Latin America and holds a B.S. in engineering from Duke University. EdenTree Technologies, 810 Lawrence Dr., Suite 210, Newbury Park, CA 91320, 805-499-4555, e-mail: [email protected]
Rob Spade, the founder and principal consultant for SASA Solutions, has more than 30 years of experience in network test engineering. He has held CTO, vice president, director, and principal development positions at Optimems, Motorola, Idea, OMEX, and NCR. Mr. Spade also has been affiliated with the IEEE Network Management Committee, the TIA Committee on Satellite Transmission of IP and ATM, and the Corporation of Open Systems and co-founded the Communication Test Automation Group. SASA Solutions, P.O. Box 51807, Phoenix, AZ 85076, 480-361-8482, e-mail: [email protected]