Quantitative analysis of faults shows that…

Not to worry, we are not going to get overly scientific here. I happened across this extremely interesting paper called “Quantitative analysis of faults and failures in a complex software system” published by Norman Fenton and Niclas Ohlsson in ye god old year 2000. The paper is very much worth a read, so if you have the patience I recommend you read it and make your own conclusions. For the impatient I present my own conclusions that I draw from reading the paper.

The gentlemen have done a pretty interesting piece of research that coincides well with my own observations of software development in various companies and countries. They worked with a large software base of a large company to investigate a couple of pretty simple theorems that most people take for granted. The research is about general software faults but the security faults are also software faults so this is all relevant anyway.

First, their object of investigation concerned the relationship between the number of faults in the modules of the software system and the size of the modules. It turns out that the software faults are concentrated in a few modules and not scattered uniformly throughout the system as one may have expected. That coincides very well with the idea that the developers are of different quality and experience and the modules written by different people will feature different levels of code quality.

Then, the finding that confirms my experience but contradicts what I hear quite often from managers and coders alike at all levels: the complexity of the code does not have any relation to the number of faults in that module. The more complex (and larger) code does not automatically beget more faults. It is again down to the people who wrote the code whether the code is going to be higher or lower in quality.

And then we come to a very interesting investigation. Apparently, there is strong evidence that (a) software written in similar environments will have similar quality and (b) the software quality does not improve with the time. You see, the developers do not become better at it. If they sucked at the beginning, they still suck ten years later. If they were brilliant to start with, you will get great code from day one. I am exaggerating but basically that is how it works. Great stuff, right?

So, the summary of the story is that if you want to have good code – get good developers. There is simply no other way. Good developers will handle high complexity and keep the good work, bad (and cheap) developers will not and will not learn. And no amount of tools will rectify that. End of the story.