The latest spate of ransomware highlights the need for security testing. One technique for doing this is called fuzz testing, or fuzzing. It is an automated test methodology that uses large amounts of invalid or random data for the input to the application. The application is tested in the usual fashion for errors such as memory leaks and invalid pointers. This approach works best with applications that have structured inputs, such as packets for a protocol or file formats.
Automatically generating test vectors is not a new approach, and has been used in other test environments—including unit testing. The trick is to generate examples that are semi-valid so as to expose corner cases, parsing errors, and errors in general. It can also be used to test security to see if trust boundaries are breached.
There are toolchains available to generate the test cases. Some provide automated input minimization support that is designed to reduce the number of test cases and help isolate a failure. This is useful because a malformed, totally random input may cause a difficult-to-identify bug. A minimization tool will minimize the input stream to the point where the error still occurs, making the task of identifying the problem easier. Delta debugging is an approach that can reduce a test case to a minimal failure-inducing case.
Fuzzing can also be used to detect differential bugs, or bugs detected by comparing results using the same inputs with different application implementations. This might be an incrementally improved application or two different applications that perform the same function, such as an encryption algorithm or a communication protocol stack. Different results using the same input indicate a case that should be investigated.
Google’s OSS-Fuzz program found more than 1,000 bugs in 47 open-source projects. Of course, this type of testing is not limited to open-source projects, but these do provide easy access to large amounts of code. Top errors included heap buffer overflows, global buffer overflows, and stack overflows. Many of these were caught as part of regression testing of new versions.
Fuzzing is only one part of a testing regimen, but it may not be one that you are already using. While it’s not applicable to all applications, it can be quite useful and automated for many applications.
