To print this page, select "Print" from the File menu of your browser salon.com > Technology April 27, 1999 URL: http://www.salon.com/tech/feature/1999/04/27/mindcraft Microsoft's flawed Linux vs. NT shootout The test lab looked for help online. But did it really want answers? - - - - - - - - - - - - By Andrew Leonard On April 13, a test lab named Mindcraft released a report commissioned by Microsoft that declared Windows NT to be far superior in performance to Linux. NT, trumpeted the study from Mindcraft, an "independent testing lab," is "2.5 times faster than Linux as a file server and 3.7 times faster as a Web server." Numbers like that can get Linux lovers hopping mad. But what really has the Linux community incensed is the way Mindcraft apparently went about seeking information and help from Linux experts. Sure, Mindcraft -- either itself or through intermediaries -- did ask for help in an Internet-based discussion forum. But it did so under false pretences: It didn't tell its would-be helpers that it needed the information because it was comparing the performance of Windows NT and Linux at the commercial request of Microsoft. And its request for advice originated from a computer with a Microsoft address. It refused to answer any requests for more information about its test. Finally, it apparently promised to redo the test with the input of Linus Torvalds himself -- but isn't giving him the info he needs to do the job right. The original news of Mindcraft's test sent the Linux world into immediate convulsions -- the sound of open-source geeks gnashing their teeth could be heard all over the Internet. Were all those claims for Linux's technical superiority just so much overheated hooey? Not so fast, said the Net. As demonstrated in a recent report by Linux front man Eric Raymond, if there is one thing that the Internet is really, really good at, it is marshaling the combined forces of thousands and thousands of watchful eyes. In combination with some aggressive trade press reporting, numerous details undermining the Mindcraft report soon emerged. First, the study had been commissioned by Microsoft -- a fact not mentioned in the original press release, and buried deep within the Mindcraft Web site. Second, the test had been able to take advantage of extensive "tuning" information from Microsoft -- details on how to make NT work especially well on the particular hardware configuration used by Mindcraft. And third, Mindcraft's attempts to obtain similar tuning information from the Linux community were half-hearted, at best -- and here is where the story gets interesting. The Linux community swears by its accessibility, by its willingness to help answer questions in a variety of forums. But it isn't perfect. The convoluted story of Mindcraft's attempt to get information from one of the most popular such forums -- Usenet newsgroups -- illustrates a key Linux weakness, the lack of centralized sources of information on the newest and most advanced technical issues. But at the same time, the story underlines the essential worthlessness of commercially sponsored comparison tests. The purpose of these tests is to please the customer who commissions them. Why expend too much energy attempting to find information that your customer probably won't appreciate? The trail starts with a post to Usenet on March 11 by a correspondent known only as "Will." The post -- which was cross-posted to the newsgroups comp.os.linux.setup, comp.os.linux.networking and comp.infosystems.www.servers.unix -- asked for information on how to maximize performance from a Linux-Apache system on an extremely high-performance, multiprocessor hardware setup. And it received one lengthy post offering technical advice and asking for more details -- which were not forthcoming. Nearly a month later, a perceptive Usenetter noticed that the hardware of that setup matched exactly what Mindcraft had been using for its tests. And, most intriguingly to the conspiracy minded, the "message header" for the Usenet post from "Will" included information indicating that the message had been originally sent from a computer on Microsoft's internal network. I sent "Will" an e-mail asking if the situation he had described in his Usenet post indeed referred to the Mindcraft test configuration. I also asked him why he had chosen not to identify himself. The correspondent acknowledged that he was acting as an "intermediary" between Mindcraft and Usenet. As to his reluctance to come clean, he wrote: "Any form of post comparing NT vs Linux, or by @mindcraft.com or @microsoft.com would have yielded nothing but flaming insults (I've seen it all too often). I've personally been running Linux for some time as well as NT, and regardless of my personal opinion, it is appallingly clear that Linux fans are deeply attached to their OS on an emotional level akin to Mac users. So much so that communications often degrade to mindless sarcasm and insults." "Will" then noted that Mindcraft was in the process of "rerunning these tests (and others) with the direct aid and assistance (albeit remotely) of the top minds in the Linux community." By consensus, the two "top minds" of the Linux community are Linus Torvalds and Alan Cox. Cox and Torvalds acknowledged that they had been in communication with Bruce Weiner, the president of Mindcraft, but both expressed dismay at the extent to which they were being allowed to participate. "I've traded a couple of e-mails with Mindcraft people about this," says Alan Cox. "They seem solely intent on trying to re-create their existing pro-Microsoft results and hoping, by attaching some kind of 'Linux top mind' credibility to it, they can do more damage." "The whole thing has been fairly painful," says Torvalds. "Mostly because these people don't actually let us know what the hell they are doing. We've been offering to be on site to see what the hell is going on, but so far they've refused." "There isn't that much we can do -- they don't actually allow anybody access to the dang thing, so while they have e-mailed us about their problems, we can only tell them that the results they are seeing are extremely low, and we can only guess at the reason why," continues Torvalds. "Experiments on similar (but not the same) hardware has shown much better performance in other places." "Will" refused to provide his full name or employer, although he denied that he is a Microsoft employee. He explained the issue of his mail originating from a Microsoft mail server by saying that he has "friends at Microsoft" who "owe me favors." One such favor, he said, was to host his machine -- which, he noted, runs Red Hat Linux. Getting up-to-date information about Linux configuration issues for cutting-edge hardware is a serious issue for the Linux community. But even if Mindcraft had identified itself, right off the bat, to Usenet, or Red Hat, or Torvalds himself, it's not entirely clear that the community could have come up with answers that would have significantly improved Linux's performance. Mindcraft did not return repeated phone calls and e-mail queries, but at least one source indicates that the company is tiny, perhaps no more than two or three employees. Mindcraft's business depends on satisfying customers like Microsoft who request specific numbers on specific hardware and software combinations. As one engineer familiar with the testing business notes, "The choice of what to test and which results to publish are always driven by the customer's interests ... There's plenty of money in performance testing, as long as it's possible to produce credible numbers that are favorable to the customer." Credible or incredible? The numbers certainly are favorable. One wonders, though, whether the fallout from this episode will be exactly the opposite of what Microsoft might have wished. The Linux world is suddenly highly motivated to accomplish two new challenges -- centralizing access to "tuning information" and upgrading Linux performance on the kind of advanced hardware that hasn't been Linux's priority. Stay tuned. salon.com | April 27, 1999