Let's Get This Straight: Yes, there is a better search engine

Yes, there is a better search engine. While the portal sites fiddle, Google catches fire.

Published December 21, 1998 8:00PM (EST)

I don't know about you, but I simply cannot get excited about the
much-ballyhooed arrival of Go.com -- Disney's new (and still "beta-testing") entry in the portal wars. Just what the world needs: another Web site that unites directory listings, news, weather, stock quotes, movie reviews, free e-mail, shopping and other online functions on one ugly-as-sin Web page -- one that looks and behaves remarkably like its popular competitors.

Go.com isn't breaking any new ground -- unless you value the prominent horoscope function on its home page, a device intended chiefly to collect visitors' birth dates for marketing purposes. As a Web portal, a site designed to be users' first daily stop for all their online information needs, Go.com retreads ground that has been thoroughly trampled by the existing portals such as Yahoo, Excite and Lycos, Netscape's Netcenter, AOL.com and MSN.com. (In the interest of full disclosure, I should mention that Salon's book reviews will be featured content in Go.com's book area.)

In 1998 we saw such portals elevated by Wall Street traders into the standard-bearers of the new digital economy. These companies' future prospects for audience share and e-commerce revenue were expected to be so vast that investors blithely ignored their meager or nonexistent profits and bought their stocks at prices sane observers considered ridiculous.

I'm not an Internet investor myself, and I don't care much whether Wall Street's love affair with portals is consummated in a bath of riches or sours in some imminent bubble-bursting market correction. But I am an Internet user. And I resent that today's portals are so obsessed with fine-tuning their demographics and matching every dubious feature their competitors offer that they are doing virtually nothing to improve the service at the heart of all their businesses: helping us all find stuff on the Web.

Most of the portals have the eyeballs -- the site traffic -- that make them potentially successful businesses because they started as search engines. But in the three years or so of the commercial Web's evolution, during which the number of indexable Web pages has mushroomed, these search engines have made only the smallest improvements to their technology.

When you conduct a general search on a broad term like, say, "President Clinton," you never know whether you'll actually find the White House Web site -- or some homely page chronicling an eighth-grade class trip to D.C. (Infoseek does a decent job returning the Oval Office site at the top of the list, but Excite sends you to an impeachment poll on Tripod and the Paula Jones Legal Defense Fund -- the president's page doesn't even make it into the first 10 results. Hotbot's top result is a site called Tempting Teens -- "All the Kinky Things that make our Government what it is.")

This is an everyday problem familiar to anyone who uses search engines regularly. So here's some good news for us -- and bad news for the big portals: There is a better way to build a search engine. And a Silicon Valley start-up company with the unlikely name of Google.com is showing the way.

Google.com started as a research project by a couple of Stanford grad students -- which, of course, is just how Yahoo, the directory site that has become the Web's most popular service, began. Yahoo tends to be more valuable than other search sites because its index is created by human beings rather than computer programs. But for the same reason, Yahoo has a hard time keeping up with the Web's explosive growth.

Google gets remarkably smart search results by using a mathematical algorithm that rates your site based on who links to you. The ranking depends not simply on the number of sites that link to you, but on the linking sites' own importance rating. The result is a kind of automated peer review that sifts sites based on the collective wisdom of the Web itself.

The program is complex, but the proof is in the results. Since discovering Google a few weeks ago, I've been so impressed with its usefulness and accuracy that I've made it my first search stop.

Google isn't a finished product yet. Its creators, Sergey Brin and Larry Page, started their company only three months ago, and the Google.com home page calls itself an "alpha test."

Page says the current version of Google, which has indexed about 60 million pages, will continue to be improved as the company expands. He adds that the search tool, which is running on Linux systems, ought to "scale up" well as the Web keeps growing. And according to Page, its site-ranking approach is nearly impossible for devious webmasters to trick or "spam," since it's based on links and judgments made by other respected sites: "You have to actually convince someone who's important that you're important."

In my book, Google itself is important -- as a sign, amid the profusion of look-alike portals, that there's still plenty of room for improvement in the basic technologies we use on the Web every day. If the portals themselves don't generate innovation, smart people elsewhere will. Commerce is a big driving force in how the Web evolves, but creativity is another. Just
as imaginative marketers will keep finding ways to sell us more stuff, inventive programmers will keep finding ways to reduce noise and confusion online and help us all find what we're looking for.

The irony here is that the big portal sites are the ones, increasingly, making it harder to use the Web: They're under such pressure to turn a profit to justify their market valuations that their pages have become crowded, blinking arrays of commercial distractions. Meanwhile, they're failing to drive forward the technology at the root of their business. That a couple of grad students could build a better search engine than a whole raft of media and technology companies with stock-market valuations in the billions does not speak well of how these firms are spending their budgets.

Which is one more reason to distrust the conventional view that the portals have the future of the Web sewn up. There's something ultimately dumb about these all-things-to-all-people sites in a medium whose greatest strength is the ability to be specific things to specific people. If the portals can't even build a better search engine, I am not betting on their
ability to control an industry as fast-moving, innovative and metamorphic as the Internet -- next year or any year.

By Scott Rosenberg

Salon co-founder Scott Rosenberg is director of MediaBugs.org. He is the author of "Say Everything" and Dreaming in Code and blogs at Wordyard.com.

MORE FROM Scott Rosenberg

Related Topics ------------------------------------------