Going down in the data mine

It was Professor Plum in the billiard room with a lead pipe, and he gave money to ImpeachPAC and attended a conference with John Yoo.

Published February 9, 2006 6:04PM (EST)

When we stopped using our supermarket discount card -- the one that got us two-for-one 12-packs of Diet Coke in exchange for handing corporate America a weekly update on our food tastes and hygiene habits -- we felt a little, you know, paranoid.

After reading today's Christian Science Monitor, we're feeling a little vindicated. In a piece that should add an extra layer of "Big Brother" gloom to the debate over warrantless spying, Monitor staff writer Mark Clayton describes the government's efforts to develop a "massive computer system that can collect huge amounts of data and, by linking far-flung information from blogs and e-mail to government records and intelligence reports, search for patterns of terrorist activity."

The "core" of the effort, Clayton says, is "a little-known system" called Analysis, Dissemination, Visualization, Insight, and Semantic Enhancement. Supporters call it "ADVISE"; you might know it as data mining. "It means sifting through data to look for patterns," Clayton writes. "If a supermarket finds that customers who buy cider also tend to buy fresh-baked bread, it might group the two together. To prevent fraud, credit-card issuers use data-mining to look for patterns of suspicious activity."

So far, so good, so long as it doesn't bother you that a supermarket sales team somewhere is noting that you prefer organic strawberries and ribbed condoms and wondering how to use that information to sell you more deodorant. ADVISE is different. It's the government, and its reach is a lot longer than the conveyor belt at the checkout stand. As Clayton writes, ADVISE would "collect a vast array of corporate and public online information -- from financial records to CNN news stories -- and cross-reference it against U.S. intelligence and law-enforcement records. The system would then store it as 'entities' -- linked data about people, places, things, organizations, and events."

It was Professor Plum, in the billiard room with a lead pipe, and he gave money to ImpeachPAC and attended a conference with John Yoo.

What would the government do with all this information once it had it? In a report presented last fall, the manager of the Department of Homeland Security's "Threat and Vulnerability, Testing and Assessment" said the goal is to identify patterns in the data that may reveal the motives and plans of potential terrorists.

Some parts of the program are operational now, Clayton says, and he quotes Jim Thomas, the director of the government's National Visualization Analytics Center, who says that it's helping to stop terrorist plots already. "There's no question that the technology we've invented here at the lab has been used to protect our freedoms," Thomas says, "and that's pretty cool."

That's not the word we had in mind.

By Tim Grieve

Tim Grieve is a senior writer and the author of Salon's War Room blog.

MORE FROM Tim Grieve

Related Topics ------------------------------------------

Espionage War Room