One of the more heated debates in security circles over the past few years got a kickstart today as a company that last month claimed Microsoft Windows server is more secure than a comparable Linux offering from Red Hat released its study methodology.
Security Innovations (SI) created a small stir last month at the RSA Security Conference, when it released some initial findings in a comparison of Windows 2003 Server and Red Hat Enterprise Server ES3.
SI reported that the Red Hat offering had more than twice the number of reported vulnerabilities and more than twice the total "days of risk." The study also found that even a minimally configured Red Hat offering had a higher number of severe vulnerabilities and a higher days-of-risk average than the Windows product.
The study's findings, released in full yesterday, stung Linux advocates, who typically cite security as an area where Linux trumps Windows, and it earned a mild but firm response from the head of Red Hat's Security Response Team, who disagreed with the study's conclusions. Contributing to the controversy was the fact that Microsoft funded the study.
The core function of SI's study was a "role comparison" between the two platforms acting as a Web server. As such, the components involved for each platform were limited to the Web server software (Apache 2 for Red Hat, IIS 6.0 for Windows), database software (MySQL for Red Hat, SQL Server 2000 for Windows), and a scripting engine (PHP for Red Hat, ASP.NET for Windows).
The study's authors used both platforms in their default configurations, arguing in their report that "experience in the security space has shown time and time again that the default configuration is often the configuration used in the real world – for better or for worse."
The researchers ran through installations of each platform, running port scans (define) on each at the end of the installation process and considering factors such as the total number of components installed by default. The study also considered reported vulnerabilities using the National Institute of Standards' ICAT Metabase, which assigns a severity of high, medium, or low based on a number of factors including whether a given vulnerability is remotely exploitable.
Based on the study, the researchers found the Red Hat server to have 174 total vulnerabilities, vs. 52 for the Windows Server system.
Severity Windows Server 2003 RHEL ES 3 Default
High 33 77
Medium 17 69
Low 0 8
Not Known 2 20
Total 52 174
In a "days of risk" comparison, in which the study compared the total amount of time between a vulnerability's public disclosure and the availability of a patch, Red Hat fared little better, with the study finding Microsoft's server to have an average of 31.3 days of risk compared to Red Hat's 71.4.
In a study based on the same methodology but using Red Hat's minimal installation, which is meant to provide a smaller "attack surface," Red Hat was found to have 152 vulnerabilities to Microsoft's 52, and its "days of risk" average was found to be 69.6 to Microsoft's 31.3.
Anticipating some blowback based on both the study's underwriter and the traditional opacity of methodology comparable security studies have demonstrated, the researchers also released a whitepaper providing their methodology in full, which Mark Cox, the head of Red Hat's Security Response Team addressed both in his personal blog and on on his Web space on Red Hat's site.
"Despite the report's claim to incorporate a qualitative assessment of vendor reactions to serious vulnerabilities, the headline metrics treats all vulnerabilities as equal, regardless of their risk to users," Cox wrote. He went on to note that he's published data and scripts that paint a starkly different picture from the study's findings.
Considering, for instance, "critical" vulnerabilities, which would allow a remote attacker to gain control of a system, Cox's numbers found 8 vulnerabilities with an average of eight days of risk, 75 percent of which were fixed within one day. Counting all vulnerabilities, regardless of severity, Cox found an average of 40 days of risk with 25 percent fixed within one day.
Both the study's findings and the methodology whitepaper are freely available at SI's Website.