“I think I see it,” said Clarke.

“Where?” asked Collins.

“Right here,” Clarke said, pointing toward a line of source code on the screen. “Looks like you left out the second ‘i’ in ‘original’ on line 172.”

Within seconds, Collins, the designated typist, had fixed the error. With a click of the return button, the program was running once again through a battery of internal tests written by the two-man coding team over the last two weeks. This time around, the offending red tinge signifying a failed test was gone. The program’s status bar showed solid green. Zero failures. Their newest feature, a point-and-click editing command, had received the green light, literally, and Collins and Clarke were ready to move on.

“Ah yes,” said Clarke, leaning back while Collins piloted the machine. “Yet another example where having a second set of eyes helps.”

In an era of cheap, portable computers and universal connectivity, the image of two people working over the same keyboard seems about as quaint as an RCA vacuum tube. For a growing number of programmers, however, it’s the latest thing: “pair programming,” a cornerstone tactic in an emerging grass-roots software development methodology sweeping the industry.

The name used to describe that methodology varies. Some call it “agile programming.” Most call it “extreme programming,” or “XP” for short. Whatever the name, the methodology’s tenets boil down to an intriguing mix of age-old developer wisdom, newfangled coding tactics and a sugary-sweet layer of marketing-speak. Depending upon on whom you talk to in the software industry, it’s either the biggest breakthrough since object-oriented programming or the biggest pile of hype since “push” technology.

“XP improves a software project in four essential ways: communication, simplicity, feedback, and courage,” reads one XP booster site.

“XP is like a ring of poisonous snakes, daisy-chained together,” counters another. “All it takes is for one of them to wriggle loose, and you’ve got a very angry, poisonous snake heading your way. ”

The truth about XP, which is no relation to the Microsoft operating system Windows XP, actually lies somewhere in the middle. Communication is certainly the key component of any XP project. Designed to overcome the endemic problem of programmers promising one thing and delivering something totally different, the XP methodology is built around putting the face-to-face interaction between developers and customers — not to mention developers and developers — over the keyboard instead of over the conference table. XP’s underlying article of faith is that if programmers and customers just communicate better, quality software will be the natural result.

Extreme programming also means giving up a little control — acknowledging, at the outset, that software development is an imperfect process.

“It’s the idea of admitting on Day 1 of a software project that you have no idea what your project’s going to look like on a given future date and saying, ‘I’m OK with that,’” says Colin Strasser, managing principal of Union Square Internet Development.

The alternative approach, as Strasser and other XP boosters are quick to point out, is the now familiar tale of software industry woe: Company A hires Company B to develop a multimillion dollar software program. The companies draft a contract, complete with specifications, and part ways while Company B writes the software. Eighteen months later, when Company B finally delivers, the resulting software looks absolutely nothing like the program specified in the contract. Or, even worse, Company B meets all the specifications but delivers a product riddled with bugs.

The history of the software industry is rife with high profile examples of this process. In 1997, the U.S. Internal Revenue Service scuttled a software upgrade to the tune of $3.5 billion. In 1995 the Denver Airport spent an extra $88 million, not including lost revenue, to debug its $193 million baggage handling system. Such examples, however, are less scandalous than the overall culture of shoddiness that seems to have infected the high tech industry over the last decade.

“There’s a real tendency for business people and geeks to see development as a zero-sum game,” says Kent Beck, the Oregon programmer now credited with coining the term “extreme programming.” “The suits say, ‘Give me more stuff, more stuff!’ The geeks say, ‘But I have to do a worse and worse job to give you more stuff.’ I wanted to get away from that.”

Beck traces the XP story back to early 1996. At the time, the Chrysler Corp. had called him in to help bring the company’s new payroll system up to speed. In the process of putting the system through its paces, Beck made a troubling discovery: All the paycheck amounts were coming out false. Despite 18 months of development and millions of dollars, the software lived up to the original specifications but couldn’t deliver correct data from one component to the next. After a few weeks, Beck found himself in the unwelcome position of telling Sue Unger, Chrysler’s chief information officer, that she and her fellow executives had a giant lemon on their hands.

Instead of helping to overhaul the system, Beck offered his professional advice. He recommended that Unger dump the software and start over from scratch. To his lasting surprise, Unger accepted the advice — with a Solomon-like twist. She put Beck in charge of the new project.

“I said, ‘You don’t understand. I’m a consultant. I don’t do in charge,’” recalls Beck, laughing. “She said, ‘You don’t understand, I’m Sue Unger. You’re in charge.’”

In the mad scramble to give Chrysler something it could use, Beck called upon many of the best practices he’d come across during his career. Early on, he decided to discard the usual whiteboard planning and view system design as an evolving, feedback-driven process.

“It was a combination of preparation and panic,” Beck says. “I decided we were going to divide the project into stories. Each story would involve a specific feature that the customer wanted. It would also be something we could test, so we could show that we had fulfilled the customer request.”

Writing tests before writing the code eliminated the need for exhaustive specifications, Beck says. It also eliminated the need for a large “quality assurance” team, which in most large-scale projects makes sure a software program obeys its specs.

Beck turned to pair-programming out of a similar desire for economy. The way he saw it, pair programming eliminated the need for extensive software documentation. If a programmer could communicate his ideas clearly to the programmer sitting over his shoulder, chances are, the programmer inspecting the code two years later would find it easier to understand. What’s more, having a copilot on each machine would reduce the overall number of bugs.

The more Beck explained, the more solid the approach became. “By the 15th person, it had become fully streamlined,” Beck says.

By the middle of 1996, Beck decided to give his management concept a name. Because other methodologies emphasized planning over programming, Beck decided that “programming” should be in the title. Still, “I needed an adjective that would be catchy, descriptive and defensible,” Beck says.

Beck says decided to name his methodology “extreme programming” after noting the intensity of extreme athletes. “I wanted my teams to have that same sense of fearlessness in the face of challenge,” he says.

Ron Jeffries, a friend and colleague called in to coach the Chrysler team, grabbed onto the term immediately. “The way it was described was we were taking all these practices that the best programmers already did and turning the dials all the way up to 10,” Jeffries says.

Beck’s strategy worked. By 1997, Chrysler had a new payroll system, and Beck and Jeffries were writing the first of many books on the XP “methodology” and convincing others to give it a try.

XP’s ensuing rise from ad hoc method to next big thing is as much a testament to the evangelization skills of Beck, Jeffries et al. as it is to the virtues of the XP method itself. Five years after the fact, the number of XP testimonials trails well behind the number of XP books — currently numbering more than a dozen on Amazon.com.

Such numbers prompt skeptics to wonder if “extreme programming” shouldn’t be changed to “extreme hype.” Doug Rosenberg, president of Itrix Software, a Santa Monica, Calif., company whose consulting and tutorial services now compete head-to-head with the growing number of XP tutorials in the marketplace, likens the XP phenomenon to a “brush fire” — heavy on the flash but light on the fuel.

“Some of the ideas are actually pretty interesting,” says Rosenberg. “They’ve actually contributed positively to the industry in terms of having people pay more attention to unit testing and automated testing frameworks. [But] the way they’ve marketed it, creating this whole fad phenomenon is objectionable to me.”

Matt Stephens, another XP critic, prefers to zero in on the weaknesses of the XP approach itself. His August 2001 essay “A Case Against Extreme Programming” is the one that likens the XP approach to a ring of poisonous snakes. The ring analogy comes about, Stephens says, because of the XP proscription against “dabbling” — adopting only certain elements of the XP approach. In order to make the XP methodology work, project managers must employ all the elements, according to XP proponents.

But Stephens considers certain elements risky. For example, XP places a low priority on source-code documentation, a common developer bugbear, on the rationale that well-written, well-tested code will be easier to understand by third party programmers than well-documented, poorly written code.

“If everything goes according to plan, XP might just get you there,” Stephens warns. “If something goes wrong (e.g. your key programmer leaves; the on-site programmer, who is essentially the walking requirements spec, is hit by a bus; the janitor accidentally vacuums up your pile of story cards) then things can really go awry.”

Beck downplays the argument that XP is an all-or-nothing approach. Still, he does note that the project leaders who report the most success with the methodology tend to be the ones who put aside the cultural obsession with rigid specs, an obsession inherited from less flexible fields such as civil and mechanical engineering, and approach XP’s design-on-the-fly philosophy wholeheartedly.

“My strategy has been to say, ‘Let’s push it,’” Beck says. “If you think you’re doing XP, but all you’re really doing is a little bit of testing and weekly planning sessions, you haven’t fundamentally changed the social contract yet.”

After all, Beck notes, it was the gaping disconnect between management and software development that prompted him to develop XP in the first place. Rather than paint XP as yet another revolutionary trend in the software business, Beck prefers to describe it as a much-needed “return to civility.” After nearly two decades of working on both sides of the line, Beck says he was looking for a way to eliminate, or at least diminish, the sociological disconnect that seems to characterize most software projects.

“Going into XP, I was very conscious that this had to be good for both sides,” he says. “The suits can’t say, ‘We need these three things by Friday,’ and the geeks can no longer say ‘That’s impossible.’ With XP, you’ve got a situation where the programmers come back and say, ‘We can give you two out of three. Which do you prefer?’ Then the negotiating begins.”

Whether or not other programmers and managers accept the bargain, XP is already lowering stress levels in some corners of the industry. Tom Clarke, the Union Square Internet Development programmer currently using techniques such as pair programming to fulfill his company’s latest development contract, a Web site rebuild, says his first brush with XP was a very positive experience.

“I was working on a project for another company that involved sending out 50,000 e-mails every night,” says Clarke, recalling his first brush with XP. “I’d go to bed wondering if the e-mails were going to the right addresses or if they were even going out at all. I had no way of knowing for sure. When I stumbled onto the notion of unit testing, I started experimenting with it. By the time I was done, I came up with a few tests that verified that, yes, the software did what I said it did. Suddenly, I found I was sleeping a lot better.”

Since then, Clarke has helped to form a New York City XP interest group which meets in the Union Square Internet Development offices one night a week. As for his programming mate, Eli Collins, he reports a similarly positive take on the XP methodology.

“There’s none of this worry about breaking things three days before release, which is a serious concern on most projects,” he says. “[With XP], you can sit down with a client and if they say they want to do something new, you can say, ‘OK. We’re ready.’”

Recalling the earlier pair programming episode, Collins adds with a smile, “It’s also a confidence booster when you know that stuff like a missing ‘i’ isn’t going to come back and bite you.”

The pile, a collection of recently published papers investigating the topic of software evolution, a topic Lehman helped inaugurate back in the 1970s, is something of a taunting tribute. Written by professional colleagues at other universities, each paper cites Lehman’s original 1969 IBM report documenting the evolutionary characteristics of the mainframe operating system, OS/360, or his later 1985 book “Program Evolution: Processes of Software Change,” which expands the study to other programs. While the pile’s growing size offers proof that Lehman and his ideas are finally catching on, it also documents the growing number of researchers with whom Lehman, a man with dwindling office space and even less in the way of support, must now compete.

“And to think,” says Lehman, letting out a dry laugh. “When I first wrote about this topic, nobody took a blind bit of notice.”

Software evolution, i.e. the process by which programs change shape, adapt to the marketplace and inherit characteristics from preexisting programs, has become a subject of serious academic study in recent years. Partial thanks for this goes to Lehman and other pioneering researchers. Major thanks, however, goes to the increasing strategic value of software itself. As large-scale programs such as Windows and Solaris expand well into the range of 30 to 50 million lines of code, successful project managers have learned to devote as much time to combing the tangles out of legacy code as to adding new code. Simply put, in a decade that saw the average PC microchip performance increase a hundredfold, software’s inability to scale at even linear rates has gone from dirty little secret to industry-wide embarrassment.

“Software has not followed a curve like Moore’s Law,” says University of Michigan computer scientist John Holland, noting the struggles of most large-scale software programs during a 2000 conference on the future of technology. “In order to make progress here it is not simply a matter of brute force. It is a matter of getting some kind of relevant theory that tells us where to look.”

For Lehman, the place to look is within the software development process itself, a system Lehman views as feedback-driven and biased toward increasing complexity. Figure out how to control the various feedback loops — i.e. market demand, internal debugging and individual developer whim — and you can stave off crippling over-complexity for longer periods of time. What’s more, you might even get a sense of the underlying dynamics driving the system.

Lehman dates his first research on the topic of software evolution back to 1968. That was the year Lehman, then working as a researcher at IBM’s Yorktown Heights facility, received an assignment to investigate IBM’s internal software development process. Managers at rival Bell Labs had been crowing about per-developer productivity, and IBM managers, feeling competitive, wanted proof that IBM developers were generating just as many lines of code per man-year as their AT&T counterparts.

Lehman looked at the development of OS/360, IBM’s flagship operating system at the time. Although the performance audit showed that IBM researchers were churning out code at a steady rate, Lehman found the level of debugging activity per individual software module to be decreasing at an equal rate; in other words, programmers were spending less and less time fixing problems in the code. Unless IBM programmers had suddenly figured out a way to write error-free code — an unlikely assumption — Lehman made a dire prediction: OS/360 was heading over a cliff. IBM, in stressing growth over source-code maintenance, would soon be in need of a successor operating system.

Although IBM executives largely ignored the report, Lehman’s prediction was soon borne out. By 1971, developers had encountered complexity problems while attempting to install virtual memory into the operating system, problems which eventually forced the company to split the OS/360 code base into two, more easily manageable offshoots. The linear growth curve that seemed so steady in the 1960s suddenly looked like the trail of a test missile spiraling earthward.

Lehman’s report would eventually earn a small measure of fame when University of North Carolina professor and former OS/360 project manager Frederick P. Brooks excoriated the IBM approach to software management in his 1975 book “The Mythical Man Month.” Using Lehman’s observations as a foundation for his own “Brooks Law” tenet — “adding manpower to a late software project makes it later” — Brooks argued that all software programs are ultimately doomed to succumb to their own internal inertia.

“Less and less effort is spent on fixing original design flaws; more and more is spent on fixing flaws introduced by earlier fixes,” wrote Brooks. “As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress.”

By 1975, Lehman, with the help of fellow researcher Laszlo Belady, was well on the way to formulating his own set of laws. A quarter century after their creation, the laws read like a mixture of old developer wisdom and common textbook physics. Take, for example, Lehman’s “Second Law” of software evolution, a software reworking of the Second Law of Thermodynamics.

“The entropy of a system increases with time unless specific work is executed to maintain or reduce it.”

Such statements put Lehman, who would leave IBM to take a professorship at Imperial College, into uncharted waters as a computer scientist. Halfway between the formalists, old-line academics who saw all programs as mathematical proofs in disguise, and the realists, professional programmers who saw software as a form of intellectual duct tape, Lehman would spend the ’70s and ’80s arguing for a hybrid point of view: Software development can be predictable if researchers were willing to approach it at a systems level.

“As I like to say, software evolution is the fruit fly of artificial systems evolution,” Lehman says. “The things we learn here we can reapply to other studies: weapon systems evolution, growth of cities, that sort of thing.”

That Lehman conspicuously leaves out biological systems is just one reason why his profile has slipped over the last decade. At a time when lay authors and fellow researchers feel comfortable invoking the name of Charles Darwin when discussing software technology, Lehman holds back. “The gap between biological evolution and artificial systems evolution is just too enormous to expect to link the two,” he says.

Nevertheless, Lehman aspires to the same level of intellectual impact. While he was in retirement during the early 1990s, his early ideas jelled into one big idea: What if somebody were to formulate a central theory of software evolution akin to Darwin’s theory of natural selection? In 1993, Lehman took an emeritus position at Imperial College and began work on the FEAST Hypothesis. Short for Feedback, Evolution and Software Technology, FEAST fine-tunes the definition of evolvable software programs, differentiating between “S-type” and “E-type”: S-type or specification-based programs and algorithms being built to handle an immutable task, and “E-type” programs being built to handle evolving tasks. Focusing his theory on the larger realm of E-type programs, Lehman has since expanded his original three software laws to eight.

Included within the new set of laws are the Law of Continuing Growth (“The functional capability of E-type systems must be continually increased to maintain user satisfaction over the system lifetime”) and the Law of Declining Quality (“The quality of E-type systems will appear to be declining unless they are rigorously adapted, as required, to take into account changes in the operational environment”). For added measure, Lehman has also thrown in the Principle of Software Uncertainty, which states, “The real world outcome of any E-type software execution is inherently uncertain with the precise area of uncertainty also unknowable.”

While the new statements still read like glossed-over truisms, Lehman says the goal is to get the universal ideas on paper in the hopes that they might lead researchers to a deeper truth. After all, saying “objects fall down instead of up” was a truism until Sir Isaac Newton explained why.

“Whenever I talk, people start off with blank faces,” Lehman admits. “They say, ‘But you haven’t told us anything we didn’t already know.’ To that I say, there’s nothing to be ashamed of in coming up with the obvious, especially when nobody else is coming up with it.”

For extra ammo, Lehman also has expanded the graphs and data from his original studies in the 1970s. Taken together, they show most large software programs growing at an inverse square rate — think of your typical Moore’s Law growth curve rotated 180 degrees — before succumbing to over-complexity.

Whether the curves serve as anything more than a conversation-starter is still up for debate. Chris Landauer, a computer scientist at the Aerospace Corporation and a fellow guest speaker with Lehman at a February conference on software evolution at the University of Hertfordshire, was impressed by the Lehman pitch.

“He has real data from real projects, and they show real phenomena,” Landauer says. “I’ve seen other sets of numbers, but these guys have something that might actually work.”

At the same time, however, Landauer wonders if the explanation for similar growth trajectories across different systems isn’t “sociological.” In other words, do programmers, by nature, prefer to add new code rather than substitute or repair existing code? Landauer also worries about whether the use of any statistic in an environment as creative as software development leads to automatic red herrings. “I mean, how long does it take a person to come up with a good idea?” Landauer asks. “The answer is we just don’t know.”

Michael Godfrey, a University of Waterloo scientist, is equally hesitant but still finds the Lehman approach useful. In 2000, Godfrey and a fellow Waterloo researcher, Qiang Tu, released a study showing that several open-source software programs, including the Linux kernel and fetchmail, were growing at geometric rates, breaking the inverse squared barrier constraining most traditionally built programs. Although the discovery validated arguments within the software development community that large system development is best handled in an open-source manner, Godfrey says he is currently looking for ways to refine the quantitative approach to make it more meaningful.

“It’s as if you’re trying to talk about the architecture of a building by talking about the number of screws and two-by-fours used to build it,” he says. “We don’t have any idea of what measurement means in terms of software.”

Godfrey cites the work of another Waterloo colleague, Rick Holt, as promising. Holt has come up with a browser tool for studying the degree of variation and relationship between separate offshoots of the original body of source code. Dubbed Beagle, the tool is named after the ship upon which Charles Darwin served as a naturalist from 1831 to 1836.

Like Landauer, Godfrey expresses concern that a full theory of software evolution might be too “fuzzy” for most engineering-minded programmers. Still, he credits Lehman for opening the software field to newer, more intriguing lines of inquiry. “It’s the gestalt ‘Aha’ of his work that I find more interesting than the numbers,” Godfrey says.

For Lehman, the lack of a scientific foundation to the software-engineering field is all the more reason to keep digging. Fellow researchers can quibble over the value of judging software in terms of total lines of code, but until they come up with better metrics or better theories to explain the data, software engineering will always be one down in the funding and credibility department. A former department head, Lehman recalls the budgetary battles and still chafes over the slights incurred. Now, as he sits in a cramped office, trying to recruit new corporate benefactors and a new research staff, he must deal once again with those who label software development a modern day form of alchemy — i.e. all experiment but no predictable result.

“In software engineering there is no theory,” says Lehman, echoing Holland. “It’s all arm flapping and intuition. I believe that a theory of software evolution could eventually translate into a theory of software engineering. Either that or it will come very close. It will lay the foundation for a wider theory of software evolution.”

When that day comes, Lehman says, software engineers will finally be able to muscle aside their civil, mechanical and electrical engineering counterparts and take a place at the grown-ups’ table. As for getting bigger offices, well, he sees that as a function of showing the large-scale corporations that fund university research how to better control software feedback cycles so their programs stay healthier longer. Until then, the search for a theory has rendered Lehman less of a Darwin and more of an Ahab — a man in search of both fulfillment and a little revenge.