Code that kills, for real

Future military combat systems will require ever more complicated code, but writing software that is bug free and ready for a firefight is a challenge that gets tougher every day.

Topics: U.S. Military,

“When everyone decides for themselves what frequency to use, what protocols to use, what standards to use, then you get systems that don’t talk to each other. And it’s killing us.”

That sort of lament is a staple at technology conferences, and its dire language is usually a matter of executive hyperbole: Somewhere in corporate America, perhaps, a bottom line is breathing its last, and we’re supposed to care.

But when the speaker is in uniform, and the incompatible systems he’s describing belong to the armed forces, then you sit up straight in your seat and realize that the words are meant all too literally. As Adm. Michael Sharp of the U.S. Navy went on to say, in a talk last month in Salt Lake City, “Software errors, timing errors, can get real critical — killing the wrong people, or not killing the right people and leaving our people unprotected.”

Here’s David Cook, senior research scientist at Aegis Technologies: “It has been said that, without software, the F-16C is nothing more than a $15 million lawn dart. There are stories that I know for a fact of airplanes that have been flying cross-country that land at a base that’s not where they’re supposed to land, and while they’re there, somebody modifies the software. And the airplane flat stops in midair when they turn on the radar unit. Why? Because there are incompatible versions of this certain piece of radar software, one of which they never thought would be on that particular model.”

The U.S. military is like every other institution in our society: It runs on software. And all software has bugs. But unlike, say, your Windows PC, where a crash could cause you to curse at some lost work, or your bank’s ATM, where a software error could leave you with an incorrect cash balance, or even your precinct’s voting machine, where a security hole could cast doubt on a close election’s outcome, bugs in Pentagon programs carry potentially fatal risks. Americans in wartimes past were warned that “loose lips sink ships”; today, it’s loose code that sinks ships, and downs airplanes, and costs lives.

I recently spent four days at the Systems and Software Technology Conference, a 16-year-old Pentagon-sponsored event for Defense Department contractors and software developers, because I wanted to know more about how the defense establishment approaches the ever more complex job of writing the literally hundreds of thousands of individual programs that our tax dollars fund and our armed forces depend on. Private-sector software is notoriously late, expensive and full of flaws; no one has a more urgent incentive to solve those problems than the people who write code for “the warfighter” (a Pentagon terminology upgrade for “soldier”). Have they figured out anything the rest of us can learn from?

The headlines from Iraq and the Pentagon since then may make that question seem trivial. During the week I spent at the Utah conference, casualties mounted in Iraq, and the latest defense-contracting scandal at Boeing was hitting the front pages. But the world of “defense systems” moves at a pace that is, in contrast to the flow of news, geological: The average “acquisition cycle” for a military product, according to conference speakers, is 10 years — and that’s an optimistic figure. Between the time the Pentagon commissions a system and a contractor delivers it, whole generations of private-sector computing software and hardware have come and gone.

That scale and pace makes defense-establishment technology a world unto itself. This parallel universe of software has a lengthy and checkered history of dependence on a dauntingly complex armamentarium of methodologies and practices and disciplines, aimed at codifying good engineering practices, producing reliable plans and schedules, and reducing flaws in the product. (One epicenter for these standards is the Pentagon-backed Software Engineering Institute at Carnegie Mellon.)

Like everything else in this field, these schemes kick up dust storms of acronyms. Although the “high ceremony” required by such approaches — including rigorous documentation, exhaustive testing and precise measurements — can look an awful lot like arthritic bureaucracy, many programmers swear by them, insisting that their tenets reduce costs and save time in the long run.

In a talk titled “The Need for Speed,” conference speaker Jon Ogg, director of engineering and technical management at the Air Force Materiel Command, laid out what he calls the “software divergence dilemma” the military faces today. In the past 50 years, the amount of code in a typical military system has increased a hundredfold: For instance, a 1960s-era jet fighter might have had 50,000 lines of code, whereas the new Joint Strike Fighter will demand 5 million. Meanwhile, in that same span of time, the average productivity of programmers has only doubled. That means we’ve seen the “person-months of effort now required to develop a capability” increase by a factor of 50, Ogg says.

As Barry Boehm, professor of software engineering at the University of Southern California and a pioneer in the field of cost estimation, put it, “The amount of software DoD needs is growing exponentially, and you’ll never get it done in a finite amount of time.”

And so today’s Pentagon is deep in the throes of a convulsive sea change toward a more nimble and “agile” method of making software — one that parallels recent developments both in the private sector and in the operational transformations the Rumsfeld-era military has espoused. The military-industrial elephant, in other words, is trying to learn dance steps from the private-sector menagerie.

From what I could tell from the people I talked to and the sessions I sat in on at SSTC, there’s a pitched battle taking place in the defense technology world between die-hards of the old disciplines and the proponents of the new agility, with its emphasis on “working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.” The latter scorn the stultifying paperwork of their more traditional colleagues and are staging a “rebellion against an overemphasis on process,” in one speaker’s words. Meanwhile, among the old guard, as another put it, “there’s almost a feeling that lightweight is cheating.”

The Defense Department honchos who spoke at the conference left little doubt about which road they favor. They’re abandoning custom-tailored, white-elephant “legacy systems” and buying COTS — commercial, off-the-shelf systems. In their push toward a “netcentric” vision of “warfighters” connected by the GIG — a secure, global broadband network — they are emphasizing the adoption of standards from the commercial world, including many familiar technologies and buzzwords.

“XML and Web services are crucial for protecting America,” according to deputy undersecretary of defense Sue Payton.

David Wennergen, CIO of the Department of the Navy, had tough words for the crowd of contractors and developers: Get with this new program, or your project will get cancelled. “We have the hardest time when we try to invent the solution all on our own,” said Wennergen. “Our future is a future of Web services, as we move toward an enterprise portal structure. Too often we’re tortured by the idea of sunk costs. But if you’re not part of that transition, it really doesn’t matter how much we’ve already spent on your system.”

Which is not to say that there aren’t skeptics about the new marching orders, who view at least some of these directives as futile or misguided. Critics point out that agile and “extreme programming” approaches fit small teams and projects best and may not offer much help to the vast hordes of developers laboring over the Defense Department’s next-generation megaprojects.

One of the five winners of the conference’s Top Five Quality Software Projects awards, a software framework for the Patriot Excalibur missile, started out as a troubled project paralyzed by delays and low morale; the team of about 20 started to click only when they switched to what they described as a “modified extreme programming” technique. Every other project in the Top Five, though, involved much bigger groups of roughly 100 or more developers.

And while the government’s budget monitors may dream of saving money by plugging in ready-made commercial software rather than handcrafting one-of-a-kind products, history and experience suggest that road is pocked with its own craters. Randall Jensen, a Hughes Aircraft veteran now with Hill Air Force Base’s Software Technology Support Center (the conference host), questioned the Pentagon’s COTS mania: “It’s a bolt-on! It’s free! That’s the current philosophy. But you have to understand it, build the interfaces, and test it. Reusable software is not free. It’s going to kill you…. It’s dangerous stuff.”

As the armed forces prepare almost unimaginably complex new technologies — like the Army’s Future Combat System — for an age of “network-centric warfare,” it may be that the adoption of “agility” as a watchword is simply bowing to the inevitable. These projects may differ from their predecessors in their decentralization, their compatibility (or “interoperability”) and their automation, but they share one thing with their forebears: They still take forever to move from the drawing board to the field. If you’re writing software today for a system that’s going to take years to deploy, you have no choice but to plan on everything changing around you. You’re Sisyphus, and you might as well get used to that stone rolling back down the hill.

Or, as Aegis’ Dave Cook put it, in a pep talk on quality:

“What are the odds your software is going to change? Oh, 100 percent. It’s going to be around for 10 years. You’re going to have to modify it more times than you can imagine. You’re gonna have to change it for different hardware and software environments, change languages three times, and go through four personnel turnovers. What you have to do is fight as hard as you can to keep quality up so that 10 years from now you have something left. It’s like, when you grab the software in your hands, it’s sand. And as you stand here doing the best you can, sand starts leaking between your fingers. Ten years from now, there’s a few grains of sand left, and that’s all you have to work with. Your job is to keep your hands as tight as possible.”

Salon co-founder Scott Rosenberg is director of He is the author of "Say Everything" and Dreaming in Code and blogs at

More Related Stories

Featured Slide Shows

  • Share on Twitter
  • Share on Facebook
  • 1 of 11
  • Close
  • Fullscreen
  • Thumbnails

    Ten spectacular graphic novels from 2014

    Beautiful Darkness by Fabien Vehlmann & Kerascoët
    Kerascoët's lovely, delicate pen-and-watercolor art -- all intricate botanicals, big eyes and flowing hair -- gives this fairy story a deceptively pretty finish. You find out quickly, however, that these are the heartless and heedless fairies of folk legend, not the sentimental sprites beloved by the Victorians and Disney fans. A host of tiny hominid creatures must learn to survive in the forest after fleeing their former home -- a little girl who lies dead in the woods. The main character, Aurora, tries to organize the group into a community, but most of her cohort is too capricious, lazy and selfish to participate for long. There's no real moral to this story, which is refreshing in itself, beyond the perpetual lessons that life is hard and you have to be careful whom you trust. Never has ugly truth been given a prettier face.

    Ten spectacular graphic novels from 2014

    Climate Changed: A Personal Journey Through the Science by Philippe Squarzoni
    Squarzoni is a French cartoonist who makes nonfiction graphic novels about contemporary issues and politics. While finishing up a book about France under Jacques Chirac, he realized that when it came to environmental policy, he didn't know what he was talking about. "Climate Changed" is the result of his efforts to understand what has been happening to the planet, a striking combination of memoir and data that ruminates on a notoriously elusive, difficult and even imponderable subject. Panels of talking heads dispensing information (or Squarzoni discussing the issues with his partner) are juxtaposed with detailed and meticulous yet lyrical scenes from the author's childhood, the countryside where he takes a holiday and a visit to New York. He uses his own unreachable past as a way to grasp the imminent transformation of the Earth. The result is both enlightening and unexpectedly moving.

    Ten spectacular graphic novels from 2014

    Here by Richard McGuire
    A six-page version of this innovative work by a regular contributor to the New Yorker first appeared in RAW magazine 25 years ago. Each two-page spread depicts a single place, sometimes occupied by a corner of a room, over the course of 4 billion years. The oldest image is a blur of pink and purple gases; others depict hazmat-suited explorers from 300 years in the future. Inset images show the changing decor and inhabitants of the house throughout its existence: family photos, quarrels, kids in Halloween costumes, a woman reading a book, a cat walking across the floor. The cumulative effect is serene and ravishing, an intimation of the immensity of time and the wonder embodied in the humblest things.

    Ten spectacular graphic novels from 2014

    Kill My Mother by Jules Feiffer
    The legendary Pulitzer Prize-winning cartoonist delivers his debut graphic novel at 85, a deliriously over-the-top blend of classic movie noir and melodrama that roams from chiaroscuro Bay City to Hollywood to a USO gig in the Pacific theater of World War II. There's a burnt-out drunk of a private eye, but the story is soon commandeered by a multigenerational collection of ferocious women, including a mysterious chanteuse who never speaks, a radio comedy writer who makes a childhood friend the butt of a hit series and a ruthless dame intent on making her whiny coward of a husband into a star. There are disguises, musical numbers and plenty of gunfights, but the drawing is the main attraction. Nobody convey's bodies in motion more thrillingly than Feiffer, whether they're dancing, running or duking it out. The kid has promise.

    Ten spectacular graphic novels from 2014

    The Motherless Oven by Rob Davis
    This is a weird one, but in the nervy surreal way that word-playful novels like "A Clockwork Orange" or "Ulysses" are weird. The main character, a teenage schoolboy named Scarper Lee, lives in a world where it rains knives and people make their own parents, contraptions that can be anything from a tiny figurine stashable in a pocket to biomorphic boiler-like entities that seem to have escaped from Dr. Seuss' nightmares. Their homes are crammed with gadgets they call gods and instead of TV they watch a hulu-hoop-size wheel of repeating images that changes with the day of the week. They also know their own "death day," and Scarper's is coming up fast. Maybe that's why he runs off with the new girl at school, a real troublemaker, and the obscurely dysfunctional Castro, whose mother is a cageful of talking parakeets. A solid towline of teenage angst holds this manically inventive vision together, and proves that some graphic novels can rival the text-only kind at their own game.

    Ten spectacular graphic novels from 2014

    NOBROW 9: It's Oh So Quiet
    For each issue, the anthology magazine put out by this adventurous U.K.-based publisher of independent graphic design, illustration and comics gives 45 artists a four-color palette and a theme. In the ninth issue, the theme is silence, and the results are magnificent and full of surprises. The comics, each told in images only, range from atmospheric to trippy to jokey to melancholy to epic to creepy. But the two-page illustrations are even more powerful, even if it's not always easy to see how they pertain to the overall concept of silence. Well, except perhaps for the fact that so many of them left me utterly dumbstruck with visual delight.

    Ten spectacular graphic novels from 2014

    Over Easy by Mimi Pond
    When Pond was a broke art student in the 1970s, she took a job at a neighborhood breakfast spot in Oakland, a place with good food, splendid coffee and an endlessly entertaining crew of short-order cooks, waitresses, dishwashers and regular customers. This graphic memoir, influenced by the work of Pond's friend, Alison Bechdel, captures the funky ethos of the time, when hippies, punks and disco aficionados mingled in a Bay Area at the height of its eccentricity. The staff of the Imperial Cafe were forever swapping wisecracks and hopping in and out of each other's beds, which makes them more or less like every restaurant team in history. There's an intoxicating esprit de corps to a well-run everyday joint like the Imperial Cafe, and never has the delight in being part of it been more winningly portrayed.

    Ten spectacular graphic novels from 2014

    The Shadow Hero by Gene Luen Yang and Sonny Liew
    You don't have to be a superhero fan to be utterly charmed by Yang and Liew's revival of a little-known character created in the 1940s by the cartoonist Chu Hing. This version of the Green Turtle, however, is rich in characterization, comedy and luscious period detail from the Chinatown of "San Incendio" (a ringer for San Francisco). Hank, son of a mild-mannered grocer, would like to follow in his father's footsteps, but his restless mother (the book's best character and drawn with masterful nuance by Liew) has other ideas after her thrilling encounter with a superhero. Yang's story effortlessly folds pathos into humor without stooping to either slapstick or cheap "darkness." This is that rare tribute that far surpasses the thing it celebrates.

    Ten spectacular graphic novels from 2014

    Shoplifter by Michael Cho
    Corinna Park, former English major, works, unhappily, in a Toronto advertising agency. When the dissatisfaction of the past five years begins to oppress her, she lets off steam by pilfering magazines from a local convenience store. Cho's moody character study is as much about city life as it is about Corinna. He depicts her falling asleep in front of the TV in her condo, brooding on the subway, roaming the crowded streets after a budding romance goes awry. Like a great short story, this is a simple tale of a young woman figuring out how to get her life back, but if feels as if it contains so much of contemporary existence -- its comforts, its loneliness, its self-deceptions -- suspended in wintery amber.

    Ten spectacular graphic novels from 2014

    Through the Woods by Emily Carroll
    This collection of archetypal horror, fairy and ghost stories, all about young girls, comes lushly decked in Carroll's inky black, snowy white and blood-scarlet art. A young bride hears her predecessor's bones singing from under the floorboards, two friends make the mistake of pretending to summon the spirits of the dead, a family of orphaned siblings disappears one by one into the winter nights. Carroll's color-saturated images can be jagged, ornate and gruesome, but she also knows how to chill with absence, shadows and a single staring eye. Literary readers who cherish the work of Kelly Link or the late Angela Carter's collection, "The Bloody Chamber," will adore the violent beauty on these pages.

  • Recent Slide Shows



Comment Preview

Your name will appear as username ( settings | log out )

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>