Silicon Follies

Chapter 43: Geek apocalypse: Four days to ship, and the bugs are everywhere.

By Thomas Scoville
Published August 11, 1999 8:00PM (UTC)
main article image

There comes a time in every programmer's life when, after months of design, development and debugging, he discovers an architectural oversight, a flaw in basic reasoning or a mistaken key assumption. In a rush it dawns on him that the whole project must be scrapped and rethought. Desolation reigns.

Paul started the test suite, tapping out the keystrokes on the command line. He watched the results trickle in.


# whiptest -d -f /usr/eng/test/V3.6

init ==> passed pingnode ==> passed pushpacket ==> passed recv_interrupt ==> passed thrupt_50 ==> passed Error 920: incorrect segment length Error 220: missing data Error 6: assembly failure

thrupt_200 ==> failed Defect(s) detected. Terminating.

Debug mode? [Y/N] n Trace dumped to file ~/whiptrace.2157


He exhaled sharply and leaned back in his chair, testing its spring-loaded, ergonomic correctness to the very limit. Gripping his brow between thumb and forefinger, he listened to waves of despair and panic sloshing around in his skull.

This was worse than bad. This was the apocalypse. And they were supposed to be shipping in four days.


- - - - - - - - - - - - - - - - - - - - -

Any serious software development effort included the construction and maintenance of a "performance/regression test suite." The test suite was a sort of obstacle course for a computer program -- a series of digital jumps, hoops and contortions that defined the minimum standard of behavior for the system in question. As the programmers continued to build and refine their program, they periodically ran it against the test suite to make sure none of their recent modifications had negatively impacted their gold standard of functionality. In other words, the test suite existed to provide periodic reassurance they hadn't inadvertently broken anything while they weren't looking.

And WHIP was flunking spectacularly. It was failing the simplest tests in the suite, the easy stuff it had been breezing through for months -- until now.


Of course, this was the worst possible timing for such a sobering event. Whip Technologies' collective technical backs were up against the wall, with the combined expectations of the industry pressed up against them. Not only were they supposed to deliver product shortly, but they were the darlings of the hour. All eyes were upon them.

Any of the engineers would have given their compilers for a chance to regroup and re-code. But the Silicon Valley path to glory was in every way like a roller coaster: Once the process of revolution was set in motion, there were no opportunities to pause, and getting off was suicide -- an embarrassing, highly visible one at that.


Paul reviewed the previous weeks' grim slide in his mind. How could it have come to this? WHIP was on deck to be the Next Big Thing. The technology was supposed to be bulletproof by now. What the hell had gone wrong?

Other engineers throughout the lab were caught up in similarly black ruminations. Though their project leaders projected the usual "can-do" attitude, there was no way to put a positive spin on it: WHIP's technical coolness was crumbling in their hands. It was time to question sacred assumptions.

- - - - - - - - - - - - - - - - - - - - -


For a while, WHIP had at least appeared stable. The original design specifications had been prudently conservative: Spare, modest, direct, they had ensured that the early struggles with WHIP's essential complexity weren't too bloody.

But then came the bells, whistles and chrome. A number of after-the-fact "requirements" and "extensions to functionality" -- all conceived and commanded by factions increasingly distant from the actual engineering and programming -- had tickled a number of WHIP's shortcomings and design flaws.

At first there had been vigorous warnings and dire predictions from technical quarters. Changing specs mid-project was bad policy; such "creeping featurism" was always a recipe for disaster. But these qualms were overcome by a shameless appeal to the engineers' vanity: WHIP was so beautiful, management had assured them, so well-designed that the very integrity of its architecture would preserve it from ruin. There was no need for worry; it was, they insisted, foolproof.


Nothing is foolproof, because fools are so ingenious. As WHIP was diddled, tweaked, dithered and extended, the hoops in the test suite began to fall. At first this was no great cause for alarm; minor ripple effects were not uncommon, and a modest number of reversals was always expected. But as weeks passed, the ripples became increasingly obscure and remediation decreasingly effective. One set of patches would result in a whole new constellation of falling hoops in far-flung locations, and each time the explanations proved more elusive.

And so the human behavior in the lab became more buggy, too. The programmers became paranoid and defensive about their craft. A spontaneous cult of secrecy and denial developed. No one could bring themselves to face the issue squarely, even as they could feel their creation coming apart. The stakes were too high now. The revolution was in progress, and the TeraMemory spinoff was committed. Full steam ahead, damn the torpedoes -- and for God's sake, everybody ignore the iceberg.

Thomas Scoville

Thomas Scoville is either an Information Age savant or an ex-Silicon Valley programmer with a bad attitude. He is the author of the Silicon Valley Tarot.

MORE FROM Thomas Scoville

Related Topics ------------------------------------------

Fiction Satire Silicon Valley