Farhad Manjoo
Wikipedia’s founder builds an open-source search engine
Jimmy Wales invests in a "distributed" effort to index the Web, part of his plan to build an open alternative to Google.
Note: This post has been updated below.
Jimmy Wales, the co-founder of Wikipedia, announced today that his for-profit community-hosting site Wikia has deepened its investment in developing an open-source Web search engine. Wikia purchased Grub, a company that makes a distributed Web-crawling program. Instead of having a single set of computers index the Web — as Google and other search engines do — Grub passes out the indexing work to computers across the globe.
You can download the Grub client to make your own computer pitch in on the indexing work. While you’re not using it, the machine will scan the Web and send back its index to a central server; your scan, combined with input from others running the Grub client, will form the index that will power Wikia’s open-source search engine.
Wales, who was speaking at the O’Reilly Open Source Convention in Portland, Ore., announced that Wikia has turned Grub into an open-source program; the company hopes for input from developers all over the world.
Unlike Wikipedia, Wikia’s search engine will run as a for-profit venture. Gil Penchina, the CEO of Wikia, has said that the company hopes to one day reach 5 percent of the search market — a number that sounds small but that could be quite lucrative. But because the project is open-source, anyone else could build a competing search engine, whether for profit or not, based on the same index, Wales pointed out to me this afternoon in the briefest of phone conversations (we had some kind of cellphone issue).
Wikia sets out several guidelines for its open engine: It will be transparent — the algorithms determining how results are ranked will be visible to all. Google and other engines invest huge sums to develop these algorithms, and they guard them extremely closely. But that’s precisely why Wales believes we need an open search engine — the world, he says, must have an alternative to a Web that’s ranked by “invisible rules inside an algorithmic black box.”
But Wales isn’t looking for transparency for transparency’s sake: The project rests on the idea that community involvement will improve on today’s search results. Whether that’s possible seems a gamble; Wikia has not announced a timeline for the project’s debut. A search engine is a huge undertaking, and there’s something nearly crazy about the idea of doing it with volunteers. But then, so too does Wikipedia and every open-source project seem somewhat impossible; that all those people could make something together doesn’t seem likely. Miraculously, though, these projects work — and the same thing could happen for search.
Update: I just got back in touch with Wales. He clarified, first, that not only will the open search engine take contributions for its source code but community members will also be actively involved in the editorial process governing search engine results.
“The idea would be a wiki-like process where the community can whitelist URLS, blacklist URLs, control for spam, block users who are being bad, that kind of thing,” Wales says.
Wales says that Wikia will have a simple front end for the search engine built by the end of this year — a place where people can “enter a search term and get some results.” He adds, “We expect that it probably won’t be very good at that point, and we’ll probably have to put a big disclaimer on the site: ‘We know this isn’t very good; please help us to make it better.’”
I asked Wales if it’s possible he’s too late in starting this — is Google too entrenched to beat? “Sure,” he says. “I could fail. I have no idea. But I’m going to have fun trying.”
The thinking man’s action hero
Using paper clips, chewing gum, chocolate and down-home ingenuity, MacGyver always saved the day. Let's bring him back -- and give him a girl!
It isn’t necessary to explain how, in the pilot episode of “MacGyver,” our mulleted, Midwestern hero gets himself trapped inside a top-secret research bunker overflowing with sulfuric acid. Suffice it to say, he needs to find a way out, and probably soon (because government agents are fixing to fire a missile at the bunker to prevent the acid from spilling into a nearby aquifer). Plus, he has to save the people he has found inside (among them a gun-wielding climate scientist who wants destroy the bunker in an effort to set back research into an ozone-layer-ruining weapon of mass destruction). Fortunately, MacGyver has a few chocolate bars, a scrap of sodium metal, a cold capsule, a pair of binoculars and cigarettes.
Continue Reading CloseGoodbye to Machinist
Yo, I'm out.

Today much of the tech world is sad that the iPhone 3G’s launch is going so miserably. But I’m sad that it’s my last day at Salon.
I’ve accepted a job at Slate, where, starting next week, I’ll be writing a twice-weekly technology column. Machinist will go on a break for a week, after which a guest blogger will bring you the latest tech dish.
Continue Reading Close“True Enough” at Google, and in San Francisco
A YouTubey presentation of my book.
As I mentioned in the comments yesterday, I’m getting ready to depart this space; I’ll have a fuller explanation tomorrow, sometime before or after I get in line to buy the new iPhone.
In the meantime, I thought I’d add a note about one of the more fun events related to my book’s release — the opportunity I had, in May, to speak at Google’s headquarters in Mountain View.
Continue Reading CloseThe iPhone 3G reviews are in: It’s pretty good
But battery life suffers, and the GPS isn't as great as you hoped.
Walt Mossberg (WSJ), David Pogue (NYT) and Edward Baig (USA Today) have been using the new iPhone 3G for a couple of weeks now, and today they all dish on their experiences.
Continue Reading CloseScary! YouTube ordered to hand your viewing history to Viacom
But there's a silver lining to one of the most bone-headed legal decisions in recent times.
Update: This post has been updated with comments from Viacom.
In the fall of 1987, a freelance reporter named Michael Dolan learned that judge Robert Bork kept an account at Potomac Video, a D.C. rental shop. This was at the height of the contentious and ultimately failed Senate confirmation hearings for Bork’s nomination to the Supreme Court — so naturally, Dolan thought there was a story here, and he went to work on getting a peek at Bork’s video rental history.
Continue Reading ClosePage 1 of 143 in Farhad Manjoo
