Finding and fixing bugs is critical to delivering quality software. One-time new bugs are often introduced into a system when changes are uploaded to a software repository. Changes may be due to adding new features or possibly correcting existing bugs.
Mozilla is planning on taking advantage of research being done by Ubisoft using artificial intelligence (AI) and machine learning (ML) to automatically catch software bugs when source code is committed to a software repository. The software is called CLEVER for Combining Levels of Bug Prevention and Resolution techniques. CLEVER was outlined in the paper “CLEVER: Combining Code Metrics with Clone Detection for Just-In-Time Fault Prevention and Resolution in Large Industrial Projects.” The paper was done at Ubisoft’s La Forge Research Lab and the ECE Department at Concordia University by Mathieu Nayrolles and Abdelwahab Hamou-Lhadj.
CLEVER is designed to flag risky commits to a source-code repository. It was able to detect risky commits 79% of the time with 65% recall based on a dozen Ubisoft projects, which is an improvement over a similar application—Commit Guru.
Defeating the False-Positive Problem
The challenge with these and other static-analysis applications is reducing the number of false positives. Missing a bug is a problem as well, but since the process is automatic, a false negative is no worse than not checking the code in terms of the amount of work a programmer needs to perform. A false positive causes additional work to be done to verify that a bug doesn’t exist. Of course, the systems are useful because they do identify real bugs.
CLEVER uses code matching versus code metrics to reduce the number of false positives. As with similar systems, analysis of a large body of prior material is used to train the system. In addition, the training takes into account commits that were used to fix a fault to provide recommendations when similar errors are found. CLEVER is tied into the bug tracking system, too, in order to match bugs that have been fixed using a commit as well as finding fixes that caused bugs that were fixed later.
The system determines if a new commit swings to the risky or non-risky side. It does this by comparing the changed blocks with historical defects. In a sense, it’s doing a type of code review.
Using machine learning to help identify bugs isn’t new. Microsoft’s IntelliCode feature in the Visual Studio IDE was developed by canning popular projects in GitHub code repositories so that it could make suggestions based on best coding practices. A number of commercial static-analysis tools have also been enhanced by using big-data and ML techniques to improve identification of errors in the code.
Flexible Approach
The CLEVER approach can be applied to any programming language. It does need to be customized for a particular language to help identify code blocks.
CLEVER takes a much different path to detecting problems when compared to programming languages like Rust, Ada, and SPARK. The latter try to eliminate certain types of problems through stricter programming requirements. In the long run, this approach is much more effective, but it doesn’t address existing applications written in other programming languages.
Likewise, more conventional static-analysis programs that examine program syntax and semantics may use ML techniques, but in a much different fashion than CLEVER.
Mozilla is testing CLEVER with its Firefox web browser development process. It will be incorporated into the mainline development if this proves successful. It will be useful if the false-positive rate can be significantly reduced, otherwise it could generate more work than might be saved.
