Schneier originally pointed me to this story about the rate at which security-related bugs were found in Open Source projects by a static code scanner:

Open source code, much like its commercial counterpart, tends to contain one security exposure for every 1,000 lines of code, according to a program launched by the Department of Homeland Security to review and tighten up open source code’s security.

Popular open source projects, such as Samba, the PHP, Perl, and Tcl dynamic languages used to bind together elements of Web sites, and Amanda, the popular open source backup and recovery software running on half a million servers, were all found to have dozens or hundreds of security exposures and quality defects.

A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006, according to David Maxwell, open source strategist for Coverity, maker of the source code checking system, the Prevent Software Quality System, that’s being used in the review.

In the comments to the story, people felt that the story was just Open Source bashing because it did not publish statistics for closed-source products such as those from Microsoft, Oracle, or IBM’s closed-source software (in general, IBM is a HUGE contributor to the Open Source code base).

So is this FUD or not? I think not, although I do believe that it falls into the category of “Information that most people lack the baseline level of knowledge to utilize.”

It tells us a lot about both relative code quality and project quality. It allows us to compare both how many security-related bugs (of the types the product can detect!) were written as well as how quickly and thoroughly the projects respond to those reports and correct them.

So rather than writing off data, especially data with limitations, as FUD simply because we may not like the role it plays in an external argument, we must consider what it does tell us, whether or not it improves our greater understanding of the problem it measures, and utilize it as best we can.

So in this case, what do we know and what don’t we know about vulnerabilities? First, we know that all non-trivial software has bugs, some of which fail in a manner which impacts the security (confidentiality, availability, and/or integrity of the application). Some of that software makes its source code freely available to anyone to download, run, review, or modify; some does not.

Historically, code quality and (especially) project team responsiveness of Open Source software has been attacked by commercial (i.e. Closed Source) software vendors as being inferior to commercial software.

Thus, when we look at the findings from the Coverity report, the questions to ask are, “Is Project X’s codebase significantly buggier than average?” and “How responsive is project X on fixing reported (security) bugs?”

The first question allows us to gauge if there is a higher level of inherent risk from using a particular piece of Open Source software. The second allows us to determine if we will be placed at a higher-than-acceptable level of risk during the vulnerability latency window (time from vulnerability announcement to patch availability). The organizations actual level of risk tolerance is, by definition, unique to the organization.

With commercial vendors, if we want to attempt to measure the risk of using a piece of software, we are forced to attempt to reverse-engineer this data by looking at patch count & severity, then wonder how many unpublished vulnerabilities or bugs were also fixed in that patch. In the Open Source world, Coverity has pulled the covers off a portion of that process.

If anything, I would argue that Open Source gains an advantage here, because Coverity is actually providing hard data to work with, which is more than the commercial vendors have been willing to provide.

Despite knowingly perpetrating FUD by writing it, I still have to wonder how commercial Coverity customers are performing versus Open Source on these metrics, and why they wouldn’t publish their results if they were actually better.

And, lest we forget, the reality is that most organizations don’t aggressively or consistently patch security vulnerabilities in anything but operating systems, anyway (which is a post for another day), which weights the decision toward average initial quality and away from project responsiveness–balance your analysis accordingly.

We strive toward metrics, but the reality is that to effectively manage risk, we must exercise good judgement in the face of incomplete knowledge and uncertainty–after all, what is risk but an attempt to quantify future uncertainty?

Our own experience has been that, typically, commercial software rates quite a bit lower than open source in terms of quality metrics that can be determined using static analysis. Or, let me qualify that by saying “lower than the best known, most popular open source projects.” At least, it does until it starts to get measured, then it tends to catch up.

In our experience, the open source community is at the forefront of the adoption of automated quality improvement tools and techniques. You can speculate about why that might be. Perhaps it has something to do with the fact that most open source projects aren’t constrained by the same notion of deadlines as commercial projects, so there is less pressure to cut corners.