Open Source Code No Less Buggy Than Commercial Apps

By Charles Babcock (Saturday, January 12, 2008)

Open source code, like its commercial counterpart, generally contains one quality defect or security exposure per 1,000 lines. So says a program launched by the U.S. Department of Homeland Security to tighten open source code security. These findings challenge the common assertion that the many eyes inspecting open source projects eliminate most bugs.
 
Commercial code is subject to the same failings, of course, but its producers tend to keep their work under wraps and not air the outcome of inspections. The Homeland Security program, on the other hand, publishes results at the Web site of Coverity, which produces the Prevent SQS code scanner being used in the review and has a $300,000 contract to conduct scans. Anyone may see results for about 180 open source projects.

Bug Tracks (Data: Coverity)

11 - Number of open source projects that resolved all defects found in first stage: Amanda, NTP, OpenPAM, OpenVPN, Overdose, Perl, PHP, Postfix, Python, Samba, and TCL

7,826 - Total defects fixed since March 6, 2006

0.127 - Flaws per 1,000 lines of code in Linux kernel 2.6

Prevent SQS is a product of the latest Stanford University research on code analysis and is designed to minimize the false hits returned by scanners as they throw every possible variable into a program's logic to see how it knots up the system. Prevent SQS relies on "Boolean satisfaction," a fancy way of saying that only those variables that are possible within the context of the program's logic need be tested; if it's impossible for the program to originate a variable, the software won't be dinged for not being able to handle it. This eliminates about 30% of the false positives normally produced by code inspection systems, says David Maxwell, Coverity's chief open source strategist.

So what's Prevent SQS finding? A total of 7,826 defects have been identified and fixed through the Homeland Security review, or one every two hours since it was launched in 2006. Bugs and vulnerabilities have been found in most open source projects, which isn't surprising. What is surprising is the speed with which some projects resolve the issues as Coverity airs them on its Web site, versus other projects that lag behind.

The 2.6 Linux kernel, for example, came through its automated scan with 913 problems identified. At press time, 452 had been fixed, 48 were verified, and plans were in place to work on the remaining 413. With its 3,639,322 lines of code, Linux's quality is far above average, with just 0.127 defects per thousand lines. Developers of Samba also have been adept at correcting vulnerabilities, Maxwell says.

The Apache Web server, which powers most active sites on the Web, has 135,916 lines of code, with a defect rate of 0.14 bugs per 1,000 lines. Three have been fixed, seven have been verified, and 12 remain. The PostgreSQL database system contains 909,148 lines of code, with a 0.041 defect rate. A few projects, including the Free Software Foundation's glibc or Gnu C Library, even have gotten the defect count down to zero.

Meanwhile, FreeBSD, sometimes posed as an alternative to Linux, appears slower to respond to the scans; it has 605 unfixed bugs. In fact, FreeBSD took the work on its defects off the Coverity site and now runs Prevent in-house. "It's fixed about 200 bugs ... and marked many more as false positives," says Colin Percival, FreeBSD's security officer.

Bruce Momjian, lead integrator for the PostgreSQL open source database project, says the Coverity test of PostgreSQL occurred in 2006, early in the project. "When I first saw the report I said, 'Wow, this is really a great tool that's able to see a lot of stuff.'" Volunteers were assigned to tackle defects, and the PostgreSQL development team was surprised at some of the items exposed. Even so, says Momjian, Prevent "doesn't see everything." The most recent flaw was spotted by Google.

To know the number of security exposures within popular software is unusual--commercial vendors rarely acknowledge security defects unless an exploit has been produced. "Our commercial customers wouldn't like it too much if we aired the number of defects found in their code," says Maxwell, when asked about the results from scans on 400 product lines of its private customers.

Originally appeared in Information Week and Yahoo.