Agena Company Logo

Agena Company Logo

Bayesian nets provide radical improvements in software defect prediction
January 2007

The developers of any new complex software system will confirm that, no matter how much testing they perform, there will still be plenty of defects (or ‘bugs’) yet to be found. The hope is that, when the software is released, any defects found by end-users will have minimal impact. Hence, the decision about when to stop testing and release the software must always be balanced by the likely number (and criticality) of remaining defects. It follows that the ability to produce accurate predictions of ‘residual’ defects in software systems is one of the most important and challenging tasks confronting software engineers. It is especially relevant for safety critical software (such as in transport and medical systems where software that is released with too many defects can have life-threatening impact); but the business of any commercial software producer can be devastated if they get their release decision wrong.

In 1999 Prof Norman Fenton and Dr Martin Neil wrote a paper [1] about software defect prediction that was published in the IEEE Transactions on Software Engineering. The paper was a critique of the commonly used approaches to the problem and they proposed a novel approach based on causal models. Although the 1999 paper provided no validation of the new approach, it became one of the most cited papers in computer science. According to Essential Science Indicators the paper is in the top 1% most influential papers.

In 2006 the editors of the journal Information and Software Technology invited the authors of the 20 most cited papers in software engineering since 2000 to submit a new paper for a special edition of the journal. The intention was that the papers (which would still be subject to the journal’s rigorous reviewing process) should reflect on the work that was done and report on new developments. Fenton and Neil were keen to respond to this invitation since their work on defect prediction had recently taken a new and exciting turn.

Ever since the 1999 paper Fenton and Neil had been working with Phillips Consumer Electronics to evolve and validate the causal models. Phillips saw the work as especially important because they developed complex software that was embedded in electronic devices like TVs and DVDs. Being able to improve their decision-making about when to release the software was critical from a business perspective because faulty software could lead to the recall of entire batches of TVs or DVDs. Using models that were built and executed in the AgenaRisk toolkit Phillips were eventually able to perform a comprehensive validation on 36 major projects based in Bangalore and Eindhoven. The ‘headline’ result was that Fenton and Neil’s 1999 conjecture that causal models could outperform traditional approaches was well and truly proven. The predictive accuracy of the causal models was 93% (comparing predicted with actual numbers of defects) compared with 80% that could be achieved with the best of the traditional approaches using the same project data. But even more important than the predictive accuracy was that the causal models and tools enabled project managers to do genuine risk assessment and ‘what-if’ analysis that simply was not possible before.

In addition to the exciting results with Phillips the new paper also describes how the approach can be easily tailored to handle arbitrary software development processes. Hence the approach can be used by organisations whose development processes are very different to those of Phillips.

The new paper [2] appears in the January 2007 issue of Information and Software Technology (Volume 49, pages 32-43). A preprint version can be found here:

http://www.dcs.qmw.ac.uk/~norman/papers/ist_fenton.pdf

A simplified version of the defects prediction model in action can be seen here.

References

[1] Fenton NE and Neil M, ''A Critique of Software Defect Prediction Models'', 25(5) IEEE Transactions on Software Engineering, 675-689,1999.

[2] Fenton NE, Neil M, Hearty P, Marsh W, Marquez D, Krause P, Mishra R, "Predicting Software Defects in Varying Development Lifecycles using Bayesian Nets", Information & Software Technology 2006