Salon.com > Big Data

Netflix, Facebook — and the NSA: They’re all in it together

Fri, 14 Jun 2013 11:28:00 +0000

On June 9, the Wall Street Journal reported that for the last few years the National Security Agency has been relying on a software program "with the quirky name Hadoop" to help it make sense of its enormous collections of data. Named after a toy elephant that belonged to the child of one of the original developers of the program, "Hadoop," reported the Journal, is a crucial part of "a computing and software revolution ... a piece of free software that lets users distribute big-data projects across hundreds or thousands of computers."

"Revolution" is probably the most overused word in the chronicle of Internet history, but if anything, the Wall Street Journal undersold the real story. Hadoop's importance to how we live our lives today is hard to overstate. By making it economically feasible to extract meaning from the massive streams of data that increasingly define our online existence, Hadoop effectively enabled the surveillance state.

Big Data arms Big Brother

Fri, 07 Jun 2013 12:50:00 +0000

SAN FRANCISCO (AP) — With every phone call they make and every Web excursion they take, people are leaving a digital trail of revealing data that can be tracked by profit-seeking companies and terrorist-hunting government officials.

The revelations that the National Security Agency is perusing millions of U.S. customer phone records at Verizon Communications and snooping on the digital communications stored by nine major Internet services illustrate how aggressively personal data is being collected and analyzed.

Verizon is handing over so-called metadata, excerpts from millions of U.S. customer records, to the NSA under an order issued by the secretive Foreign Intelligence Surveillance Court, according to a report in the British newspaper The Guardian. The report was confirmed Thursday by Sen. Dianne Feinstein, D-Calif., who chairs the Senate Intelligence Committee.

Former NSA employee William Binney told the Associated Press that he estimates the agency collects records on 3 billion phone calls each day.

Elsevier: All your data belongs to us

Tue, 09 Apr 2013 22:02:00 +0000

A distinct sense of déjà vu kicked in Tuesday afternoon as I read a series of outraged tweets decrying the news that Elsevier, the giant publisher of scientific journals, was buying Mendeley, a cloud-based social media research platform popular with academics for such tasks as organizing citations and managing access to articles.

The aggrieved tone was almost exactly the same as the backlash that erupted a few weeks ago when Amazon bought GoodReads -- at the time, the largest independent reader community on the Web.

The issues at hand aren't identical. In the Elsevier case, many of the academics who are most upset are strong believers in the principle of "open access" to scientific information, and they see Elsevier as their blood enemy. An excellent summary of the argument against Elsevier can be found at the Cost of Knowledge but the gist is simple: Elsevier extracts huge profits from a business in which the vast majority of creative work is contributed on a voluntary basis by academics. The company then turns around and uses a portion of those profits to lobby for even tighter intellectual property control.

How Netflix is turning viewers into puppets

Fri, 01 Feb 2013 12:45:00 +0000

I hit the pause button roughly one-third of the way through the first episode of "House of Cards," the political drama premiering on Netflix Feb. 1. By doing so, I created what is known in the world of Big Data as an "event" -- a discrete action that could be logged, recorded and analyzed. Every single day, Netflix, by far the largest provider of commercial streaming video programming in the United States, registers hundreds of millions of such events. As a consequence, the company knows more about our viewing habits than many of us realize. Netflix doesn't know merely what we're watching, but when, where and with what kind of device we're watching. It keeps a record of every time we pause the action -- or rewind, or fast-forward -- and how many of us abandon a show entirely after watching for a few minutes.

Netflix might not know exactly why I personally hit the pause button -- I was checking on my sick son, home from school with the flu -- but if enough people pause or rewind or fast-forward at the same place during the same show, the data crunchers can start to make some inferences. Perhaps the action slowed down too much to hold viewer interest -- bored now! -- or maybe the plot became too convoluted. Or maybe that sex scene was just so hot it had to be watched again. If enough of us never end up restarting the show after taking a break, the inference could be even stronger: maybe the show just sucked.