Google co-founders Larry Page and Sergey Brin, both in their mid-20s, started the company in 1998 after three years of graduate research at Stanford University. Page and Brin quickly pulled in Sun Microsystems co-founder Andy Bechtolsheim as an investor. Stanford has also put money into Google, and last year’s $25 million round of equity funding, led by venture capital powerhouses Sequoia Capital and Kleiner Perkins Caufield & Byers, has deepened the company’s rosy glow.

One reason for this torrent of cash is that Google works. It’s fast and accurate, with an uncanny ability to put the thing you most wanted to find directly under your nose. The technology that makes this happen is equal parts rocket science and peer review. Google’s hypertext-based system for ranking search results uses a mathematical algorithm to rate Web sites based on the number of other sites linking to them, then factors in how heavily linked those sites are. The result is a form of objectivity that springs directly from the Internet community, translating its distributed judgments into a quick, precise read of what matters and what you can do without.

Google’s latest initiative is the integration of its own search technology with the Open Directory Project, the Web’s largest human-edited directory. Melding such disparate tools into a single search service is hardly a no-brainer — particularly when, like Google, you enjoy the kind of devoted following that recoils at the slightest interface tweak. Add to that the sometimes uneven editorial quality of the all-volunteer ODP, toss in the chance that both search results and user experience might suffer in the wake of the integration, and you could anticipate an upset stomach or two at the company’s Mountain View offices.

But while Georges Harik, the software engineer (and “director of directories”) who led the integration project, had been up until 7 a.m. the night before the March 16 launch, he was all smiles — and with good reason. As aficionados of Google’s no-nonsense interface were relieved to discover, the company has managed to roll in the new functionality with hardly a ripple. It’s still as fast as ever. The difference is that there are now targeted directory entries among the search results, providing both intelligible context and lateral, topic-based browsing, with your results as point of departure. If you search on “Eric Raymond,” for example, you get links to sites associated with the open-source advocate, plus a selection of relevant directory categories, including “Computers > Open Source > Advocacy.”

As Harik explains it, the decision to go with Open Directory — just one player in a space that includes Yahoo, LookSmart and Excite, among others — had two main drivers. One was the licensing — it’s pretty hard to argue with “free,” whether you’re talking software, beer or content. The ODP’s evolving editorial culture was also a plus, said Harik. “We like the way submissions are made to the Open Directory, and we think it has the potential to be more accurate and more timely than other directories. The people who contribute to it care about what they’re doing.” The truly decisive point, though, was the Open Directory’s potential to scale in parallel to the Web’s hypercharged expansion.

That potential scalability derives from the “open content” aspect of the ODP. The premise is familiar to anyone who has followed the phenomenal rise of the Linux operating system: A highly motivated, globally distributed community of contributors, each with particular talents and expertise, can out-code and out-debug any corporate engineering team on Earth. The trick, though, is to evaluate and integrate those parallel, semi-anarchic efforts.

Different “open” projects take different approaches, from the benign despotism of Linus Torvalds (who, as he puts it, personally “sprinkles holy penguin pee” over each new Linux release) to the barely moderated laissez faire of the discussion site Slashdot. While it’s clear that “to many eyes, all bugs are shallow,” it’s not so obvious how to keep new bugs out of each new mix. And if that’s true of software, where “Does it work?” provides an unbending benchmark, it’s doubly true of the Open Directory’s attempt to map and rank the online universe.

But if Harik was concerned that adding the Open Directory to the mix might disperse Google’s mojo somehow, he didn’t show it. He neatly elided any unspoken reservations about the ODP’s editorial limitations, focusing on synergy instead: “We have confidence in the combination of their editorial process and things we can do with PageRank and automated methods over the long term to make the directory better.”

In fact, the biggest obstacle Harik sees for the company has nothing to do with the ODP at all. One of Google’s most powerful advantages — and a pillar of its claim to objectivity — is its link-based page ranking system, PageRank. But a user study demonstrated how poorly some users understand how Google’s search engine works — and gave an inkling of how entrenched online cynicism has become. “We asked people, ‘Why do you think we’re giving you this site first?’ And they said, ‘Well, they probably pay you more than the other sites do.’ This was really shocking to us, that people would assume we were getting paid to display certain search results.”

Shocking or not, the popular assumption that search engines are agents of the marketing devil is now canonical. The ability to act instantly on online information has inspired many search portals to treat their results pages like slot machines, with every search button clicked presenting a new opportunity to dangle “buying decisions” in front of their short-fused, impulse-driven users. And why not? The Internet Gold Rush is upon us, and if a search engine is at least as honest as a Las Vegas casino, we consider ourselves lucky. If the roulette wheel turns out to be rigged, well, what did we expect?

For a company like Google, this world view presents a problem. The company is well funded, but for the most part that money goes into research and engineering, not marketing, so opportunities to counter e-commerce FUD are few and far between. Besides, the reflexive belief, (first formulated by Freud) that “denial is avowal” makes “I am not a crook” (first formulated by Nixon) a bad marketing tack. What’s an honest search engine to do?

The answer: Emphasize reality, and hope people notice. The reality is that Google, while clearly looking to make a buck, or even several, has its roots in the world of academic research. Besides its genealogical link to Stanford Research, the company boasts a research group of its own, despite numbering less than a hundred employees. “The research group develops the core technologies that we’ll be using a year from now,” says Google CEO Brin. “It’s three people, going on four. Aside from that, we have about 15 Ph.D.s on our engineering staff.”

The research group is all the more vital because of the company’s focus on search technology, as distinct from the marketing-driven portal plays that now pepper the Web. “It’s critical to start developing the next generation technologies. AltaVista would never have existed if it weren’t for DEC having SRC [Systems Research Center] and WRL [Western Research Laboratory], the two research centers in Palo Alto. And IBM has certainly been doing well based on their long-term research. So for us, providing such a core technology-based service, research is critical.”

How does this research and technology focus square with the Open Directory initiative? Brin sees Google’s technology and the ODP’s directory as complementary approaches that allow people to leverage each system’s respective strengths. He describes a scenario in which the user targets the initial search as much as possible, then uses the directory links included in the results to navigate outward from there, like a paratrooper dropped behind enemy lines to run reconnaissance.

For the same reason, Brin isn’t overly concerned that the Open Directory — begun as a grass-roots project staffed entirely by volunteers with a passion for some corner of the world’s knowledge — be globally comprehensive: The point isn’t to navigate down through the hierarchy from the top. “If you look locally on one of these maps, you’ll find it’s usefully accurate. That’s the important thing for a directory: how things work locally. Because there are other ways” (read: search) “to get close to where you want to go.”

ODP: The power of people

Chris Tolles may be a marketer, but he’s no hired flack. As one of the co-founders of the Open Directory Project, the ODP’s marketing director has been around long enough to remember the June 1998 Slashdot posting that described “GnuHoo,” the directory’s original moniker, as “an interesting experiment which might work.” Unfortunately, the name didn’t work for Slashdotters, who found the allusion to the Free Software Foundation’s GNU Public License galling, especially since GnuHoo’s code and content were then proprietary. The “GnuHoo Booboo” — and the ensuing flamefest — led to a homophonous rechristening as “NewHoo.” The new name lasted until the directory’s December 1998 acquisition by Netscape, which dubbed the non-commercial, all-volunteer project (which had since shed its proprietary shackles) the Netscape Open Directory, soon to be known to open content enthusiasts by the initials ODP.

The ODP has come a long way since the flame wars of yore. The project’s Web site proudly proclaims it is “the largest human-edited directory of the Web” — not too shabby, given that the competition includes semi-namesake Yahoo. The ODP’s 20,000-plus editors have covered more than 1.5 million sites in a quarter-million distinct categories, and show no sign of slowing down.

That rapid pace — and the enormous team’s potential to scale as the Web scales — is one of the main reasons Google chose to work with the Open Directory. But when you talk to Tolles about the directory’s future, he speaks less about the imperative to ramp up and more about the need to preserve the character of the editorial community that has grown up around the ODP. He’s glad to talk about the technology the team uses: editing tools have recently been localized for Italian, French, German and Japanese as well as English, and other infrastructure improvements have made the editors’ job much easier. But his voice shows real passion when he speaks of the need to defend his editors’ autonomy and the community they’ve created.

Tolles takes the project’s self-description as a “self-regulating republic” very seriously. “We’ve turned over control of the community to the editors themselves,” he says, including everything from editorial decisions to the admission (or exclusion) of new editors. They’ll be “more selective going forward,” says Tolles, partly to preserve the working relationships that now exist. “When you sign up to be an editor, you’re applying for citizenship. You may get it, you may not.” He cites the role of precedent in guiding the group’s self-governance, describing the directory’s administrative model as “more open yet controlled than any other.”

Underscoring the need for control, Tolles points to “the signal-to-noise ratio on Usenet” as an example of how bad ungated mass participation can be. “We believe that keeping the voice of our editors … is one of our most important responsibilities. We have a goose that lays golden eggs here. The goose eggs are great, but the goose is the real miracle.”

But as the ODP becomes more influential — it’s used by such major players as America Online (duh — AOL owns Netscape), Netscape (duh redux), Lycos and now Google — it’s hard not to wonder whether this concentration of influence is altogether consistent with the notion of “open content.” How does the ODP solicit broader input as to what will appear in what’s fast becoming the Web’s default directory? Nonplussed by the question, Tolles notes the small, closed teams at competitors like Yahoo and LookSmart, then points to his 20,000-plus editors: “How open is the ODP? Roughly 220 times more open than anyone else.”

The Open Directory’s influence isn’t lost on Tolles, however — including its power to ignore. “Implicit in our directory is that we exclude sites as well. You’re also creating a list of things that aren’t on your list.” And there are two groups other than the editors whose views do command attention: big licensees (“We take their input seriously”) and the press. In fact, he considers those licensees a benchmark of the project’s editorial success: “As long as we’re getting substantial players who choose us, I think we’re headed in the right direction.”

Despite the more selective approach, Tolles views the ODP as a model of inclusion. “We’ve made this a more participatory project than any other open source effort. How many people participate in the actual building of Linux? At most, it’s a thousand people. We have 5,000 people who regularly participate, and 22,000 who are signed up and eligible. Look at Linux, look at Apache, and I will guarantee you that less people worked on the major distributions — the thing that you can easily download — than on the Open Directory. Can anybody come and demand to be an editor? No. But it’s more open than anything else out there.”

While understanding why people think search sites are all about filthy lucre, Tolles sees directories and relevance technologies as an antidote. “All the major players in the search space are going to a similar model: You have paid listings; then there’s a directory, either LookSmart, Yahoo or ourselves; below that, you have Web crawler listings, usually from Inktomi, Google or Direct Hit.

“The move now is toward things like Google and Direct Hit, to carve away that top layer of crust. A whole industry has grown up doing search engine placement, tweaking sites to get a better ranking. Google is able to route around that based on their technology. Likewise, a directory is a way of avoiding those play-for-pay things — at least if you don’t play for pay in your directory. Using relevance technology is very important, and will be increasingly important down the line. With the rise of directories, and with things like Google, you’re going to have much better search results.”

Tolles is clearly used to taking the long view: Phrases like “50 years from now” roll off his tongue without hesitation. “We’re conscious that we’re building something that will outlast us,” he says. Eventually, he envisions the ODP taxonomy “as a platform for other products and services” — including technologies like Google’s. For now, though, the key goals are simple: to scale with the Web, to be useful and to be as impartial as possible. “Will we create something that’s of substantial use and scalable?” he asks. “If so, we’ll have done the right thing.”

So what’s new at “launch?” The site doesn’t look any different — other than a slightly doctored logo, which includes some fancy drop-shadows and eliminates the word “beta.” Google has also unveiled a new feature, called the GoogleScout, which lets you expand a search to include related information about your topic, such as a company’s competitors. Otherwise, it’s the same, streamlined search engine we’ve learned to love, in all of its spare, speedy glory.

Perhaps the most significant changes promised by this “launch” can be glimpsed in the new “About Google” and “Jobs@Google” buttons. Google, which started its beta life as a project by a few Stanford students, has in the last four months managed to pull in $25 million in investments, from such heavyweight venture capital firms as Sequoia Capital and Kleiner Perkins Caufield & Byers. And according to Google’s jobs page, the company is now frantically hiring engineers and marketing managers.

Expect to see Google start a new marketing push, in hopes of competing with the big-daddy portals — Yahoo, Lycos, Excite, and the like — while the company strives toward fat returns for its high-profile investors. But let’s hope, now that it’s graduated from beta to bonafide, that the search engine won’t start to clutter up its gorgeous, streamlined look with “buy” buttons, sponsored links and the other useless features that so many search engines rely on for profits.

It’s always a little risky to look to an industry conference, with its predictable eruptions of spin-doctoring PR and self-promotional hot air, for any kind of accurate assessment of a business. But these gatherings — which are lucrative enterprises in their own right — can provide a fascinating temperature gauge for the mood and mind-set of an industry’s leadership.

As executives and financiers like Yahoo’s Tim Koogle, America Online’s Bob Pittman, Softbank’s Masayoshi Son, Amazon’s Jeff Bezos and John Doerr of Kleiner Perkins stepped forward to detail strategies and rate competitors, the room became suffused with a strange mixture of giddiness and paranoia, self-confidence and insecurity — the heady psychic brew that fuels most Internet companies today. We’re unstoppable — but we never stop looking over our shoulders! This is as big as the industrial revolution — but if we’re not careful we’ll wind up as yesterday’s fad! Stock prices are unreal — but we’re buying as fast as we can!

The contradictions in these leaders’ messages were echoed in the sentiments of the larger crowd, which registered opinions via a wireless real-time polling system, punching “yes” and “no” buttons like focus-group participants (68 percent, for example, said they felt Yahoo’s stock price is too high).

Softbank’s Son, whose early investment in Yahoo has become the cornerstone of a loose-knit online empire, teased out the biggest paradox with two questions: First, do Net stock prices represent a “bubble”? Forty-nine percent said “yes” — not quite a majority, but an extraordinarily high number, considering how Net-besotted a crowd this was. (Son said that other audiences tend to vote 70 percent “yes” on the same question.) Second: In 10 years, will the market capitalization of Internet stocks exceed that of personal-computer stocks? Ninety-nine percent said “yes.” Yet, Son declared, today’s Net stocks are valued at only 10 percent of PC stocks. His conclusion: “I say Internet stocks are too cheap.”

That, of course, assumes that Net stocks will surpass PC stocks by growing 10 times — and not because of some collapse of PC company valuations. Even the executives with the most dour outlooks — oddly, the gloomiest panel was a lineup of venture capitalists, all shaking their heads at the stratospheric market’s “over-hyped deals” — share a deep long-term optimism. When they talk about the “need for a correction,” they’re imagining a six-month pause in the Internet’s victory march — not the kind of general, sustained retreat that would follow if this really were a “bubble” and it went pop.

Mostly, it seems, the poobahs of the Net are not worrying about the stock market a whole lot. And it’s not as if the stock market is giving them reason to do so. In what more than one speaker referred to as “an era of permissive capital,” investors seem to have thrown the market’s traditional yardsticks, like price-to-earnings ratios, out the window — at least for now. The future’s so bright, who wants to bother wearing green eye-shades?

“Revenues? We don’t need no stinkin’ revenues!” read one entry on a top 10 list of “Reasons to Go Public” that one of the conference hosts, Bill Gurley of Benchmark Capital, read to the amusement of the crowd. Bezos offered a similar quip: When book publishers miffed at Amazon.com for allowing customers to post negative reviews would tell him, “You don’t understand our business — we sell books to make money,” he’d reply, “No, you don’t understand our business — we don’t make money.”

Ha, ha. In the context of such humor, a reminder from Charles Schwab president David Pottruck that this is an “unusual time” — and ultimately “you need to have a clear sense of how you’re going to make money” — came off as the nag of a party-pooping fogey.

If these executives aren’t worrying about making money, what do they worry about? Two things, it seems: talent and users. By talent they mean executive talent: With all this money flooding the industry and its fast-sprouting start-up companies, often led by entrepreneurs in their 20s, there’s a shortage of experienced adults to manage Internet companies as they balloon. There are more than 400 CEO searches under way today in Silicon Valley, a recruiting expert told the conference.

At a poolside auction led by eBay’s Meg Whitman that raised $351,000 for a nonprofit called Schools Online, the bidding was sprightly for such “unique items of unclear value” (eBay argot for “tchotchke”) as John Doerr’s tie ($12,000) or the dress CEO Katrina Garnett wore in her ads for Crossworlds Software ($10,000) — both of which wound up in the hands of CyberCash founder Dan Lynch. But the item that sparked the fiercest bidding — and that eventually sold for a cool $89,000 — was an executive search by conference sponsor Korn/Ferry International.

When not struggling to recruit executive talent, the Net’s moguls are struggling to attract users. One speaker after another bragged about how obsessed they were with customers — from Bezos’ claim that Amazon is “earth’s most customer-centric company” to EarthLink founder Sky Dayton’s plea for Internet companies to make their customers more “Net-savvy” instead of “dumbing down” their services. But figures presented by Media Metrix president Mary Ann Packo suggest that sites are doing a better job of capitalizing on their existing users than attracting new ones: According to Packo, in the past year, Web page views have increased by 48 percent and hours of usage by 44 percent, yet the overall number of Web users has increased by only 15 percent.

Some Internet leaders have decided to worry about amassing users and figure that the money will then take care of itself; others have chosen to worry about fattening a war chest that they can then use to attract users. Either way, the ultimate goal is, in industry parlance, to “monetize” these users — somehow or other.

When making money is at war with making users happy, though, Net companies may run into big problems. For instance, a panel of venture capitalists expressed doubts about a recent deal in which the search engine Google received $25 million in funding from traditional rivals Sequoia Capital and Kleiner Perkins — with the deal rumored to value the company “pre-money” (i.e., before adding in the additional funding to the company’s total worth) at $70 million.

Afterward, Kleiner’s Doerr explained that Google is already getting 4 million page views a day: “We’ll figure out how to monetize that.” He also pointed out something that the other VCs didn’t seem to grasp — that Google searches actually work a lot better than those on most other search sites. Of course, one of the things users love about Google is that its home page isn’t cluttered with ads and promotions; once the “monetizing” begins, who knows how long that will last.

One company that used to top Net entrepreneurs’ list of worries stayed strangely off the radar at the Industry Standard conference. While participants pondered whether AOL or Yahoo would triumph in the portal wars, or whether AOL or Excite@Home would prevail in the effort to control home Internet access, nobody seemed terribly concerned about Microsoft, the industry’s former No. 1 nemesis. Microsoft’s Brad Chase, who recently assumed the head of the company’s consumer and commerce group, provided a warm and fuzzy, low-key view of Microsoft’s Net presence: “There are a lot of urban legends and myths about working with Microsoft — we’re very interested in setting up win-win partnerships.” Then he handed out his e-mail address.

Other speakers barely mentioned the Redmond giant. Asked to assess the likelihood of Microsoft leading the portal market a year from now, the crowd voted 71 percent for “0-25 percent chance.” Chase showed up on crutches (after a basketball injury), and while the symbolism was coincidental, the company’s concurrent announcement of its retreat from the Net content business with the sale of its Sidewalk city guides added to the image of a hobbled giant.

Of course, at the same time, Microsoft announced the latest in a long string of profit-laden quarters. The company is rumored to be thinking about issuing a new “tracking stock” to represent all of its Net businesses, and perhaps its unloading of Sidewalk was an effort to improve its bottom line in those areas before such a move. But it might also be that Microsoft is no longer obsessed with dominating the Internet media, and is happy to leave that war for eyeballs to others while it continues to rake in high-margin software profits. Either way, it no longer seems to loom quite so large in the dreams and nightmares of the Net’s business leaders.

In the end, all the executives, analysts, bankers and investors didn’t seem to have any better clue than the rest of the world about the biggest question facing the Net economy: Is the ’90s boom an aberration or a fundamental change? A speculative mania or a “paradigm shift”? EarthLink’s Dayton said that his company’s customer service reps spend a lot of time helping new users who call to say, “I’m on the Internet now — what do I do?” A surprising number of businesspeople seem to be asking the very same question.

So at first the headlines last week about AltaVista — that it was going to sell search engine results to advertisers — sounded dire. Shouldn’t searches be neutral? Was AltaVista going on the take? Horrors!

As it turns out, the initial reports — which made it sound like AltaVista wasn’t going to come clean about the plan — were wrong. If you look at the company’s discussion of what it calls “relevant paid links,” you’ll see that the paid results will be set off by boxes and labeled “paid.” The example, a search for “weddings,” brings up a plug for the Wedding Channel at the head of the results list — before you get to the 900,190 unpaid results.

Over on Yahoo, if you search for “weddings,” you get a plug for the Wedding Channel, too — only it’s in the form of a banner ad. Search sites have been “selling keywords” in this fashion for ages now. On Lycos, a search on “weddings” gets you that Wedding Channel banner, too — along with a “First and Fast” listing with links (to sites like Honeymoons.com and Marthastewart.com) that feel like paid links but aren’t labeled as such.

What’s new about AltaVista’s move is that the ads are beginning to invade — or at least bump up against — the search results list. Even this isn’t entirely novel; over at Goto.com, selling search results is the whole idea — and the site will tell you exactly how much each advertiser paid for the placement. (I don’t find Goto very useful as a search tool, but it does turn the machinery of Internet marketing into a fascinating spectator sport: Every time you click here, for instance, the Wedding Channel coughs up 35 cents to Goto.com.)

That AltaVista should be trying to squeeze more bucks from its site right now will come as no surprise to anyone who’s been paying attention to the tech industry’s corporate battlefield. AltaVista’s parent, Digital, is now owned by Compaq — and last week Compaq reported a disappointing quarter and fired its CEO, amid complaints that the company lacked a good “Internet strategy.” AltaVista is Compaq’s highest-profile Net holding. The pressure is on.

What’s interesting about “relevant paid links” isn’t their greed; it’s their admission that people still aren’t getting what they want from search engines. Aren’t all the results of a search supposed to be “relevant”?

Of course, AltaVista’s bumper crop of 900,190 results on a search for “weddings” exposes the flaws of the whole search-engine model. Search engines are good at pointing the way to specific tidbits of information on the Web — particularly if you’re skilled at picking keywords and using Boolean connectors (“and,” “or,” “not”) and you know the specific quirks of the search engine you’re using. Search engines have never been very good at up-to-the-minute information or at helping you find, say, the best overall sites on a particular general topic, and human-built directories like Yahoo are only a little better.

So the “relevant paid link” to Wedding Channel may well be more useful to the hapless AltaVista searcher looking up “weddings” than, say, most of the first 10 unpaid results that turn up. Those seem to offer a lot of geographically specific information that may not be of much value — unless you happen to be in the Albany, San Diego, Hawaii, Pennsylvania or Aruba areas.

The companies buying AltaVista’s links are aiming at the search-engine users who haven’t yet figured this out, and who insist on using the search engines as tools to find their way to general information or top-level sites. If, for instance, you want to find the home page for the New York Times, don’t expect much help from AltaVista: Search on the phrases “New York Times,” “The New York Times” and “new york times” and you will get three entirely different sets of top 10 results — none of which include links to the New York Times’ own site (perhaps because it requires users to register, which blocks out the “spider” or automated program the search engine uses to index Web pages).

AltaVista does offer a link to the Times’ actual site via a service called Realnames that pops up at the top of the search list. And guess what? If you want to be listed by Realnames, you have to pay them.

In other words, AltaVista needs to sell search results and employ work-arounds like Realnames — because that’s the only way it can shortcut its own overgrown index and get relevant answers to less skilled users. Like all the search engines from its generation that have grown up to become “portals” — including Excite, Infoseek, Yahoo and Lycos — AltaVista today has tons of Web traffic and vast hordes of users. Too much of the time, though, it doesn’t have good answers.

For those, increasingly, you have to turn to a newer generation of search tools that use smarter schemes — and that (so far) aren’t selling off their results. My favorite new search site, Google (I wrote about it last December), ranks its results based on how many other sites link to each listing — and how highly ranked those sites are. Its complex algorithm generates the best search results I’ve seen; one additional advantage is that the site, still technically in beta testing, hasn’t yet ramped up commercially, so the pages are uncluttered with promotions.

Type “New York Times” into Google and it sends you right to the newspaper’s site — without any distractions and without that company having to pay to get there.
Even Google doesn’t do that well with the “weddings” query, though; it’s still a search engine indexing a vast number of pages, and it still doesn’t provide great results from searches on general topics.

With AskJeeves, another highly touted Web information service, you phrase your search in common English — like, “Where can I find information about weddings?” — and the site sends you to a series of related questions that it “knows” the answers to: in this case, “How should I provide music at my wedding?” or “Where can I set up a personal Web site for my wedding?” or “What are some suggestions for wedding music?” That might be helpful, but it’s also cumbersome and narrow. It doesn’t simply provide me with what I’m looking for — a list of the Web’s best wedding-information resources.

For that, I found my best results with DirectHit. DirectHit isn’t a search engine itself but a technology that’s licensed to other companies — you’ll find it in action on HotBot. With DirectHit, search results are ranked based on the choices other users have previously made; if lots of previous searchers for wedding information actually clicked through to a particular site when it turned up in the results, and spent time on that site, it gets ranked higher. With DirectHit on HotBot, nearly all of the top 10 search results for “weddings” point to sites that are deep, relevant and useful.

If AltaVista were able to provide those kinds of results, I doubt its “relevant paid links” program would be much of an issue. As it is, AltaVista and its cohorts in the first generation of search engines are stuck: They’ve got a mass of users that other Web sites would kill for — but they’re not delivering what they promise. That means sooner or later, users will drift away to more effective services. And then who will want to buy AltaVista’s links?

Go.com isn’t breaking any new ground — unless you value the prominent horoscope function on its home page, a device intended chiefly to collect visitors’ birth dates for marketing purposes. As a Web portal, a site designed to be users’ first daily stop for all their online information needs, Go.com retreads ground that has been thoroughly trampled by the existing portals such as Yahoo, Excite and Lycos, Netscape’s Netcenter, AOL.com and MSN.com. (In the interest of full disclosure, I should mention that Salon’s book reviews will be featured content in Go.com’s book area.)

In 1998 we saw such portals elevated by Wall Street traders into the standard-bearers of the new digital economy. These companies’ future prospects for audience share and e-commerce revenue were expected to be so vast that investors blithely ignored their meager or nonexistent profits and bought their stocks at prices sane observers considered ridiculous.

I’m not an Internet investor myself, and I don’t care much whether Wall Street’s love affair with portals is consummated in a bath of riches or sours in some imminent bubble-bursting market correction. But I am an Internet user. And I resent that today’s portals are so obsessed with fine-tuning their demographics and matching every dubious feature their competitors offer that they are doing virtually nothing to improve the service at the heart of all their businesses: helping us all find stuff on the Web.

Most of the portals have the eyeballs — the site traffic — that make them potentially successful businesses because they started as search engines. But in the three years or so of the commercial Web’s evolution, during which the number of indexable Web pages has mushroomed, these search engines have made only the smallest improvements to their technology.

When you conduct a general search on a broad term like, say, “President Clinton,” you never know whether you’ll actually find the White House Web site — or some homely page chronicling an eighth-grade class trip to D.C. (Infoseek does a decent job returning the Oval Office site at the top of the list, but Excite sends you to an impeachment poll on Tripod and the Paula Jones Legal Defense Fund — the president’s page doesn’t even make it into the first 10 results. Hotbot’s top result is a site called Tempting Teens — “All the Kinky Things that make our Government what it is.”)

This is an everyday problem familiar to anyone who uses search engines regularly. So here’s some good news for us — and bad news for the big portals: There is a better way to build a search engine. And a Silicon Valley start-up company with the unlikely name of Google.com is showing the way.

Google.com started as
a research project by a couple of Stanford grad students — which, of
course, is just how Yahoo, the directory site that has become the Web’s
most popular service, began. Yahoo tends to be more valuable than other
search sites because its index is created by human beings rather than
computer programs. But for the same reason, Yahoo has a hard time keeping
up with the Web’s explosive growth.

Google gets remarkably smart search results by using a mathematical
algorithm that rates your site based on who links to you. The ranking
depends not simply on the number of sites that link to you, but on the
linking sites’ own importance rating. The result is a kind of automated
peer review that sifts sites based on the collective wisdom of the Web
itself.

The program is complex, but the proof is in the results. Since
discovering Google a few weeks ago, I’ve been so impressed with its
usefulness and accuracy that I’ve made it my first search stop.

Google isn’t a finished product yet. Its creators, Sergey Brin and Larry
Page, started their company only three months ago, and the Google.com home
page calls itself an “alpha test.”

Page says the current version of Google, which has indexed about 60 million
pages, will continue to be improved as the company expands. He adds
that the search tool, which is running on
Linux systems, ought to “scale up” well as the Web keeps growing. And
according to Page, its site-ranking approach is nearly impossible for
devious webmasters to trick or “spam,” since it’s based on links and
judgments made by other respected sites: “You have to actually convince
someone who’s important that you’re important.”

In my book, Google itself is important — as a sign, amid the profusion
of look-alike portals, that there’s still plenty of room for improvement in
the basic technologies we use on the Web every day. If the portals
themselves don’t generate innovation, smart people elsewhere will. Commerce is a
big driving force in how the Web evolves, but creativity is another. Just
as imaginative marketers will keep finding ways to sell us more stuff,
inventive programmers will keep finding ways to reduce noise and confusion
online and help us all find what we’re looking for.

The irony here is that the big portal sites are the ones, increasingly,
making it harder to use the Web: They’re under such pressure to turn
a profit to justify their market valuations that their pages have become
crowded, blinking arrays of commercial distractions. Meanwhile, they’re
failing to drive forward the technology at the root of their business.
That a couple of grad students could build a better search engine than a
whole raft of media and technology companies with stock-market valuations
in the billions does not speak well of how these firms are spending their
budgets.

Which is one more reason to distrust the conventional view that the
portals have the future of the Web sewn up. There’s something ultimately
dumb about these all-things-to-all-people sites in a medium whose greatest
strength is the ability to be specific things to specific people. If the
portals can’t even build a better search engine, I am not betting on their
ability to control an industry as fast-moving, innovative and metamorphic
as the Internet — next year or any year.