SALON

Friday, Jun 28, 2013 11:44 AM UTC

The NSA’s early years: Exposed!

A newly declassified work of history shows how U.S. intelligence agencies helped launch the digital age

By

Topics: NSA, surveillance state, intelligence agencies, World War II, Cold War, IBM, It Wasn't All Magic, Colin Burke,

The NSA's early years: Exposed! The Dayton "Bombe" (Credit: Wikimedia)

On Oct. 29, 1948, the Soviet Union suddenly changed all its ciphers and codes. What later became known as “Black Friday” delivered a huge shock to the two U.S. intelligence agencies that had conducted the bulk of American code-breaking efforts during World War II and its immediate aftermath. Before Black Friday, the Army’s SIS and the Navy’s OP-20-G complacently assumed that they had acquired the keys to most of the world’s encrypted communications. But with a flip of the switch the U.S. was once again in the dark — just as the Cold War was heating up.

“One of the gravest crises in the history of American cryptanalysis,” writes historian Colin Burke, led directly to the 1949 mergingof the SIS and OP-20-G into the Armed Forces Security Agency. Three years later, another bureaucratic shuffle transformed the AFSA into the National Security Agency. A sense of panic induced by the “Soviets’ A-Bomb, the Berlin Blockade, the forming of the satellite bloc in Eastern Europe, the fall of China, and the Korean War” — all of which “were not predicted” by the intelligence agencies — encouraged the U.S. government to authorize the NSA to spend tens of millions of dollars on computer research, in the hope that technological advances would help crack the new Soviet codes.

Colin Burke is the author of “It Wasn’t All Magic: The Early Struggle to Automate Cryptanalysis, 1930s-1960s.” Burke completed his history in 1994, but until last week, his volume of crypto-geekery had only a handful of readers. Part of a series produced by the NSA’s Center for Cryptological History, “It Wasn’t All Magic” was considered classified material until May 2013, and was only made available online on June 24.

Nice timing! With the NSA currently occupying its highest public profile in living memory, a look back at its early history is quite instructive. It is useful to be reminded that the mandate to spy and surveil and break codes was absolutely critical to the early growth and evolution of computer technology. Some things never change: The immense effort required to crack German and Japanese codes during World War II are an early example of the intimidating challenges posed by what we now call “big data.”

“Incredible amounts of calculation were needed for pure cryptanalysis,” writes Burke. Meeting the need for that calculation led to the NSA and its predecessors becoming key players in unleashing the digital age; and, ironically, creating the conditions that have made so much of our modern lives surveillable by the NSA. A direct line connects our Twitter-obsessed, Snapchat-crazed lives with the exploits of the codebreakers 60 and 70 years ago.

Few people in 1948 might have been able to imagine a future when the NSA would be employing its massive computing capabilities to gather and process information on all Americans, but that is exactly what the NSA was born to do: make sense of the proliferating information age.

* * *

1948′s Black Friday was hardly the first time that an inability to make sense out of big data sent shock waves through the United States’ intelligence apparatus. Before World War II, it was clear to U.S. military code-breakers that they could not keep up with information overload.

“The development of new cipher machines and the maturation of radio led to a critical data problem for America’s cryptanalysts,” writes Burke. “There was more and more data, and it was overwhelming those who were charged with turning it into useful information for policymakers. The failure to predict the attack on Pearl Harbor, for example, was the result of too much data. The thousands of intercepted Japanese naval messages could not be analyzed with the men and equipment available to … OP-20-G.”

Burke also references a fascinating story about a typhoon in December 1944 that swept over a naval task force led by Adm. Halsey and resulted in the loss of 800 American lives. The disaster might have been avoided, writes Burke, if the Navy had been able to efficiently process “the combined weather report” produced by Japanese weather stations. At that late stage in the Pacific conflict, the American fleet was moving faster than it could set up its own weather stations. But, again, there was just too much data for intelligence services to keep up with their clumsy pre-digital technologies.

And even World War II’s exigencies paled before what the NSA faced in the early ’50s. The problems went beyond the brute force necessary for code-breaking. Just keeping up with data gathered in the clear proved near impossible.

By 1955 there was a torrent of “noninformation” flowing into NSA. The United States had more than 2,000 round-the-clock listening positions that were sending thirty-seven TONS of intercept material to NSA each month. In addition, some 30,000,000 words of intercept were sent by teletype. The data overload grew with every year and every Soviet challenge. NSA’s posts were intercepting 2,000,000 messages a month from just one Soviet system, and the vast majority of that was plain text … Within a short time, punching 1,000,000 IBM cards a month for just one problem was common.

Decades before software solutions like Hadoop were developed, big data required big iron. But there was no commercial market for untested general-purpose computers with price tags of a million dollars and up. From the late 1930s on, the NSA and its predecessors were forced to experiment with a wide variety of cumbersome technologies, and push established companies like IBM to design new machines aimed to realize government priorities.

“NSA’s … computer purchases and its research and development contracts helped establish America as the world’s leading computer manufacturer,” writes Burke. “From the moment of its creation, it was at the cutting edge of computer technology and architecture … Based on an intense faith that new machines would work cryptanalytic and data processing miracles, NSA became involved in many projects to create new generations of computer technology.”

Most of “It Wasn’t All Magic” is a fairly dry, jargon-packed recitation of experiments in computer technology that sounds to modern ears as if it was written in Etruscan. Only in computer history does the archaeology of 50 years ago seem as distant as Sumer. Ironically, despite the claims made for NSA’s long-term influence, a fairly common theme of the narrative is one of failure. For almost the entire period that Burke examines, the NSA and SIS and OP-20-G geeks were unable to realize the full extent of their dreams. Practical exigencies and theory did not match up.

But the pressure and funding steered toward the private sector unquestionably provided a powerful incentive for technological progress. The history of the computer industry offers as strong an argument for the importance of government investment in pushing forward the development of new technology as can be found in the entire domain of industrial policy. The free market, left to itself, would have required decades longer to make the same advances. There simply wasn’t any money to be made cracking Soviet ciphers.

If it hadn’t been for the Cold War, it’s difficult to imagine what our digital lives might be like today. When World War II ended, writes Burker, there was a brief moment in time when the pressure to attack big data seemed to relax.

For contemporaries, the future was uncertain. No one imagined that America was going to build a multi-billion dollar intelligence bureaucracy that seemed to have a life of its own. In fact, for those in the cryptanalytic organizations in early 1945, there were signs they might return to the isolated and have-not world of the 1930s, an era when American politicians condemned “reading other gentlemen’s mail.”

Unfortunately, Burke’s tale of “one of the great techno-gambles in American history,” reaches no closure in “It Wasn’t All Magic.” The narrative ends, rather abruptly, around 1960. We are teased with the declaration that by “the early 1960s, NSA’s ‘basement’ became one of the world’s great computer data processing centers,” but we don’t get any hints about what was accomplished in that basement. And considering the fact that there are sections of Burke’s book concerning events as far back as the late 1940s that are still redacted for security reasons, one assumes that the later we get in time, the less we will be able to find about about what the NSA was up to, during the 1960s and beyond.

But this much we do know. The early 1950s, writes Burke, are now considered “the Dark Ages of American Cryptanalysis.” We can guess, thanks to the efforts of Edward Snowden, that “dark ages” isn’t the operative description for what we’re living through now.

Andrew Leonard

Andrew Leonard is a staff writer at Salon. On Twitter, @koxinga21. More Andrew Leonard.

You Might Also Like

More Related Stories

Featured Slide Shows

7 motorist-friendly camping sites

close X
  • Share on Twitter
  • Share on Facebook
  • Thumbnails
  • Fullscreen
  • 1 of 9

Sponsored Post

  • White River National Forest via Lower Crystal Lake, Colorado
    For those OK with the mainstream, White River Forest welcomes more than 10 million visitors a year, making it the most-visited recreation forest in the nation. But don’t hate it for being beautiful; it’s got substance, too. The forest boasts 8 wilderness areas, 2,500 miles of trail, 1,900 miles of winding service system roads, and 12 ski resorts (should your snow shredders fit the trunk space). If ice isn’t your thing: take the tire-friendly Flat Tops Trail Scenic Byway — 82 miles connecting the towns of Meeker and Yampa, half of which is unpaved for you road rebels.
    fs.usda.gov/whiteriveryou


    Image credit: Getty

  • Chattahoochee-Oconee National Forest via Noontootla Creek, Georgia
    Boasting 10 wildernesses, 430 miles of trail and 1,367 miles of trout-filled stream, this Georgia forest is hailed as a camper’s paradise. Try driving the Ridge and Valley Scenic Byway, which saw Civil War battles fought. If the tall peaks make your engine tremble, opt for the relatively flat Oconee National Forest, which offers smaller hills and an easy trail to the ghost town of Scull Shoals. Scaredy-cats can opt for John’s Mountain Overlook, which leads to twin waterfalls for the sensitive sightseer in you.
    fs.usda.gov/conf


    Image credit: flickr/chattoconeenf

  • Nordhouse Dunes Wilderness Area via Green Road, Michigan
    The only national forest in Lower Michigan, the Huron-Mainstee spans nearly 1 million acres of public land. Outside the requisite lush habitat for fish and wildlife on display, the Nordhouse Dunes Wilderness Area is among the biggest hooks for visitors: offering beach camping with shores pounded by big, cerulean surf. Splash in some rum and you just might think you were in the Caribbean.
    fs.usda.gov/hmnf


    Image credit: umich.edu

  • Canaan Mountain via Backcountry Canaan Loop Road, West Virginia
    A favorite hailed by outdoorsman and author Johnny Molloy as some of the best high-country car camping sites anywhere in the country, you don’t have to go far to get away. Travel 20 miles west of Dolly Sods (among the busiest in the East) to find the Canaan Backcountry (for more quiet and peace). Those willing to leave the car for a bit and foot it would be remiss to neglect day-hiking the White Rim Rocks, Table Rock Overlook, or the rim at Blackwater River Gorge.
    fs.usda.gov/mnf


    Image credit: Getty

  • Mt. Rogers NRA via Hurricane Creek Road, North Carolina
    Most know it as the highest country they’ll see from North Carolina to New Hampshire. What they may not know? Car campers can get the same grand experience for less hassle. Drop the 50-pound backpacks and take the highway to the high country by stopping anywhere on the twisting (hence the name) Hurricane Road for access to a 15-mile loop that boasts the best of the grassy balds. It’s the road less travelled, and the high one, at that.
    fs.usda.gov/gwj


    Image credit: wikipedia.org

  • Long Key State Park via the Overseas Highway, Florida
    Hiking can get old; sometimes you’d rather paddle. For a weekend getaway of the coastal variety and quieter version of the Florida Keys that’s no less luxe, stick your head in the sand (and ocean, if snorkeling’s your thing) at any of Long Key’s 60 sites. Canoes and kayaks are aplenty, as are the hot showers and electric power source amenities. Think of it as the getaway from the typical getaway.
    floridastateparks.org/longkey/default.cfm


    Image credit: floridastateparks.org

  • Grand Canyon National Park via Crazy Jug Point, Arizona
    You didn’t think we’d neglect one of the world’s most famous national parks, did you? Nor would we dare lead you astray with one of the busiest parts of the park. With the Colorado River still within view of this cliff-edge site, Crazy Jug is a carside camper’s refuge from the troops of tourists. Find easy access to the Bill Hall Trail less than a mile from camp, and descend to get a peek at the volcanic Mt. Trumbull. (Fear not: It’s about as active as your typical lazy Sunday in front of the tube, if not more peaceful.)
    fs.usda.gov/kaibab


    Image credit: flickr/Irish Typepad

  • As the go-to (weekend) getaway car for fiscally conscious field trips with friends, the 2013 MINI Convertible is your campground racer of choice, allowing you and up to three of your co-pilots to take in all the beauty of nature high and low. And with a fuel efficiency that won’t leave you in the latter, you won’t have to worry about being left stranded (or awkwardly asking to go halfsies on gas expenses).


    Image credit: miniusa.com

  • Recent Slide Shows

  • Share on Twitter
  • Share on Facebook
  • Thumbnails
  • Fullscreen
  • 1 of 9

Related Videos

Comments

Loading Comments...

follow salon

Most Read

Popular on Reddit

links from salon.com

vote
We crush them with debt to go to college -- and today, rates are actually set to double. Are we out of our minds?
3285 points3286 points | 4073 comments
vote
Before Edward Snowden: “Sexual deviates” and the NSA - When the last major defection hit the NSA, the government smeared the leakers as gay. New documents show they lied
2679 points2680 points | 185 comments
vote
This man is still lying to America: A smoking gun shows Director of National Intelligence James Clapper is not coming clean to us -- again
754 points755 points | 43 comments
vote
We must hate our children. We crush them with debt to go to college -- and today, rates are actually set to double. Are we out of our minds?
110 points111 points | 17 comments
vote
James Clapper is still lying to America
84 points85 points | comment

From Around the Web

Presented by Scribol
listener_new_rect The Listener: "The Joker": Laughing at life's deepest sorrows