Seattle, WA. Think your software is buggy? Relief may be at hand thanks to recent advances in automated software testing, as described in a Thursday morning keynote address at the International Test Conference by Patrice Godefroid, a principal researcher at Microsoft Research.
Recent advances in automated test, he said, can help test programs like powerpnt.exe, which has millions of lines of code and involves complex input formats as well as issues related to memory allocation, data structures, loops, libraries, and system calls.
Automatic code-driven test generation using program analysis—not model-based testing—he said, has been studied for 30 years, but now practical tools are emerging.
He reviewed some approaches to software test. Static test generation, he said, is ineffective when symbolic reasoning is not possible, which is frequently the case.
More promising is dynamic test generation, which runs a program (with random inputs), gathers constraints, and uses a constraint solver to generate new inputs. He cited DART (directed automated random testing), which covers all program paths—although the number of program paths can be infinite. Nevertheless, he said, DART works for small programs and significantly improves code coverage vs. random testing.
He then described SAGE (Scalable Automated Guided Execution) and its use in hunting for million-dollar bugs. Software security bugs, he said, can be very expensive, and it’s important to find these bugs as early as possible.
He noted that black-box bug inspection is a simple yet effective way of finding many bugs, adding that fuzzing is mandated by the Security Development Lifecycle.
He then introducing white-box fuzzing, which mixes fuzz testing with dynamic test generation.
SAGE results since 2007, he said, indicate that SAGE has found many new security bugs missed by black-box fuzzers and static analysis.
He cited several initiatives toward improving software quality. For example, with Windows Error Reporting (WER), if you encounter a problem a box pops up asking if you want to send a report to Microsoft. And the process works not just for Windows but for third-party programs that run on Windows. The reporting helps prioritize fixes and improves customer satisfaction, he said.
The proliferation of smartphones and the cloud present additional challenges. Nevertheless, he said, In the future, research will focus on better depth (with a move toward formal verification) and better breadth (more applications). “We live in a world of remarkable innovation, diversity, and opportunity, he concluded, “and the same is true for testing.”