SALON

Andrew Leonard

Tuesday, Apr 27, 1999 4:00 PM UTC

Microsoft's flawed Linux vs. NT shootout

The test lab looked for help online. But did it really want answers?

By

Topics:

On April 13, a test lab named Mindcraft released a report commissioned by Microsoft that declared Windows NT to be far superior in performance to Linux. NT, trumpeted the study from Mindcraft, an “independent testing lab,” is “2.5 times
faster than Linux as a file server and 3.7 times faster as a Web server.”

Numbers like that can get Linux lovers hopping mad. But what really has the Linux community incensed is the way Mindcraft apparently went about seeking information and help from Linux experts.

Sure, Mindcraft — either itself or through intermediaries — did ask for help
in an Internet-based discussion forum. But it did so under false pretences:
It didn’t tell its would-be helpers that it needed the information because it was comparing the performance of Windows NT and Linux at the commercial request of Microsoft. And its request for advice originated from a computer with a
Microsoft address. It refused to
answer any requests for more information about its test. Finally, it apparently promised to redo the test with the input of Linus Torvalds himself — but isn’t giving him the info he needs to do the job right.

The original news of Mindcraft’s test sent the Linux world into immediate convulsions — the sound of
open-source geeks gnashing their teeth could be heard all over the Internet. Were all those claims for Linux’s
technical superiority just so much overheated hooey?

Not so fast, said the Net. As demonstrated in a recent href="
http://slashdot.org/article.pl?sid=99/04/23/1316228&mode=nested&threshold=-1
">report by Linux front man Eric Raymond, if there is one thing that the
Internet is really, really good at, it is marshaling the combined forces of
thousands and thousands of watchful eyes.

In combination with some
aggressive trade press reporting, > numerous details undermining the Mindcraft report soon emerged. First, the study had been commissioned by
Microsoft — a fact not mentioned in the original press release, and buried deep within the Mindcraft Web site. Second, the test had been able to take advantage of extensive
“tuning” information from Microsoft — details on how to make NT work
especially well on the particular hardware configuration used by
Mindcraft. And third, Mindcraft’s attempts to obtain similar tuning
information from the Linux community were half-hearted, at best — and here is where the story gets interesting.

The Linux community swears
by its accessibility, by its willingness to help answer questions in a
variety of forums. But it isn’t perfect. The convoluted story of Mindcraft’s
attempt to get information from one of the most popular such forums –
Usenet newsgroups — illustrates a key Linux weakness, the lack of
centralized sources of information on the newest and most advanced technical issues.

But at the same time, the story
underlines the essential worthlessness of commercially sponsored
comparison tests. The purpose of these tests is to please the customer who
commissions them. Why expend too much energy attempting to find information
that your customer probably won’t appreciate?

The trail starts with a post to Usenet on March 11 by a correspondent
known only as “Will.” The post — which was cross-posted to
the newsgroups comp.os.linux.setup, comp.os.linux.networking and
comp.infosystems.www.servers.unix — asked for information on how to maximize
performance from a Linux-Apache system on an extremely high-performance, multiprocessor hardware setup. And it received one lengthy post offering technical advice and asking for more details — which were not forthcoming.

Nearly a month later, a perceptive Usenetter noticed that the hardware of
that setup matched exactly what Mindcraft had been using for its tests.
And, most intriguingly to the conspiracy minded, the
“message header” for the Usenet post from “Will” included information
indicating that the message had been originally sent from a computer on
Microsoft’s internal network.

I sent “Will” an e-mail asking if the situation he had described in
his Usenet post indeed referred to the Mindcraft test configuration. I
also asked him why he had chosen not to identify himself.

The correspondent acknowledged that he was acting as an “intermediary” between Mindcraft and Usenet. As to his reluctance to come clean, he wrote:

“Any form of post comparing NT vs Linux, or by @mindcraft.com or
@microsoft.com would have yielded nothing but flaming insults (I’ve seen it
all too often). I’ve personally been running Linux for some time as well as
NT, and regardless of my personal opinion, it is appallingly clear that
Linux fans are deeply attached to their OS on an emotional level akin to Mac
users. So much so that communications often degrade to mindless sarcasm and
insults.”

“Will” then noted that Mindcraft was in the process of “rerunning these
tests (and others) with the direct aid and assistance (albeit remotely) of
the top minds in the Linux community.”

By consensus, the two “top minds” of the Linux community are Linus Torvalds and Alan Cox. Cox and Torvalds acknowledged that
they had been in communication with Bruce Weiner, the president of
Mindcraft, but both expressed dismay at the extent to which they were being
allowed to participate.

“I’ve traded a couple of e-mails with Mindcraft people about this,” says Alan
Cox. “They seem solely intent on trying to re-create their existing
pro-Microsoft results and hoping, by attaching some kind of ‘Linux top mind’
credibility to it, they can do more damage.”

“The whole thing has been fairly painful,” says Torvalds. “Mostly because
these people don’t actually let us know what the hell they are doing. We’ve
been offering to be on site to see what the hell is going on, but so far
they’ve refused.”

“There isn’t that much we can do — they don’t actually allow anybody access
to the dang thing, so while they have e-mailed us about their problems, we
can only tell them that the results they are seeing are extremely low, and
we can only guess at the reason why,” continues Torvalds. “Experiments on
similar (but not the same) hardware has shown much better performance in
other places.”

“Will” refused to provide his full name or employer, although he denied that he
is a Microsoft employee. He explained the issue of his mail originating
from a Microsoft mail server by saying that he has “friends at Microsoft”
who “owe me favors.” One such favor, he said, was to host his machine –
which, he noted, runs Red Hat Linux.

Getting up-to-date information about Linux
configuration issues for cutting-edge hardware is a serious issue for the
Linux community. But even if Mindcraft had identified itself, right off the
bat, to Usenet, or Red Hat, or Torvalds himself, it’s not entirely clear
that the community could have come up with answers that would have
significantly improved Linux’s performance.

Mindcraft did not return repeated phone calls and e-mail queries,
but at least one source indicates that the company is tiny, perhaps no more
than two or three employees. Mindcraft’s business depends on satisfying
customers like Microsoft who request specific numbers on specific hardware and software combinations. As one engineer familiar with the testing business notes,
“The choice of what to test and which results to publish are always
driven by the customer’s interests … There’s plenty of money in performance
testing, as long as it’s possible to produce credible numbers that are
favorable to the customer.”

Credible or incredible? The numbers certainly are favorable. One wonders,
though, whether the fallout from this episode will be exactly the
opposite of what Microsoft might have wished. The Linux world is suddenly
highly motivated to accomplish two new challenges — centralizing access to
“tuning information” and upgrading Linux performance on the kind of advanced
hardware that hasn’t been Linux’s priority. Stay tuned.

Next Article

Related Stories

More Related Stories

Featured Slide Shows

The week in 10 pics

close X
  • Share on Twitter
  • Share on Facebook
  • Thumbnails
  • Fullscreen
  • 1 of 11
  • Lisa Montgomery embraces her nephew Thursday after a tornado tore apart her home in Cleburne, Texas. The twister killed six people and destroyed entire swaths of the North Texas town.
    Credit: AP/LM Otero

  • Jack McMahon, the defense attorney for abortion doctor Kermit Gosnell, speaks outside the Criminal Justice Center in Philadelphia Tuesday. His client was convicted of killing three babies in his clinic, and will serve multiple life sentences.
    Credit: AP/Matt Rourke

  • A photo taken Monday captures Vice President Joe Biden's response to a Milwaukee second-grader's innovative proposal to end America's epidemic of gun violence. This guy!
    Credit: AP/Jenny Aicher

  • Sen. Rand Paul, R-Ky., flanked by a grouper-eyed Michele Bachmann, addresses the IRS' admission that it targeted Tea Party groups in advance of the 2012 election. In an op-ed for CNN Thursday, the Kentucky senator slammed the president for his faux outrage.
    Credit: AP/Molly Riley

  • Ousted IRS chief Steven Miller is sworn in on Capitol Hill Friday. Miller testified before the House Ways and Means Committee on the extra scrutiny the agency gave conservative groups applying for tax-exempt status.
    Credit: AP/J. Scott Applewhite

  • Attorney General Eric Holder pauses as he testifies on Capitol Hill before the House Judiciary Committee Wednesday. Holder is under fire, among other things, for the Justice Department's gathering of phone records at the Associated Press.
    Credit: AP/Carolyn Kaster

  • O.J. Simpson sits during an evidentiary hearing at Clark County District Court in Las Vegas, Nev., Thursday. Simpson, who is currently serving a nine-to-33-year sentence in state prison for armed robbery and kidnapping, is using a writ of habeas corpus to seek a new trial.
    Credit: AP/Las Vegas Review-Journal/Jeff Scheid

  • Major Tom to ground control: On Sunday astronaut Chris Hadfield recorded the first music video from space, a cover of David Bowie's "Space Oddity."
    Credit: AP/NASA/Chris Hadfield

  • When it rains it pours. President Barack Obama speaks during a news conference Thursday with Turkish Prime Minister Recep Tayyip Erdogan, inexplicably inspiring an #umbrellagate Twitter meme.
    Credit: AP/Jacquelyn Martin

  • A smoke plume rises high above a road block at the intersection of County A and Ross Road east of Solon Springs, Wis., Tuesday. No injuries were reported, but the the wildfire caused evacuations across northwestern Wisconsin.
    Credit: AP/The Duluth News-Tribune/Clint Austin

  • Recent Slide Shows

  • Share on Twitter
  • Share on Facebook
  • Thumbnails
  • Fullscreen
  • 1 of 11

Related Videos

Comments

 0 Comments

Comment Preview

Your name will appear as username

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>

Andrew Leonard has been working at Salon as a technology reporter, editor, blogger and staff writer for quite a bit longer than he ever anticipated being employed by an online magazine -- 15 years. He's enjoyed the luck of becoming obsessed with the Internet just before it broke into mainstream

consciousness and the housing bust just before it precipitated a global economic collapse. Prior to becoming a Salon lifer he freelanced for a wide variety of publications, from Newsweek to Rolling Stone to Wired, and wrote a book, "Bots: The Origin of New Species". He lives in Berkeley, Calif., with his two children and he likes to ride his bicycle up hill.

Continue Reading

follow salon

Most Read

Popular on Reddit

links from salon.com

vote
Oklahoma senator: Tornado aid “totally different” from Sandy aid
3084 points3085 points | 2461 comments
vote
Anyone Regret Slashing National Weather Service Budget Now?: With GOP-backed cuts to forecasting agency, experts warn future storms will go undetected and more lives lost
1059 points1060 points | 567 comments
vote
Oklahoma senator wants to offset tornado aid with other cuts
154 points155 points | 61 comments
vote
New campaign compares gun control to anti-LGBT discrimination
34 points35 points | 11 comments
vote
Horrifying new trend: Posting rapes to Facebook " In Chicago this weekend, prosecutors announced three teenaged boys will tried as adults for aggravated criminal sexual assault after allegedly raping a 12-year-old girl — and posting a video of the attack on Facebook."
31 points32 points | 15 comments

From Around the Web

Presented by Scribol
listener_new_rect The Listener: So long, Sookie Stackhouse

Latest Tweets

Recommendations