Software is hard

Salon's Scott Rosenberg explains why even small-scale programming projects can take years to complete, one programmer is often better than two, and the meaning of "Rosenberg's Law."

By Andrew Leonard

Read more: Books, Technology & Business, Andrew Leonard, Software, Scott Rosenberg, Windows, Interviews, Computers, Authors, Books Interviews

Feb. 3, 2007 | One way to look at Salon co-founder Scott Rosenberg's new book, "Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software," is as an attempt to tell the story of a specific software development project -- the effort by industry legend Mitch Kapor and a band of ace programmers to create Chandler, a kind of turbo-powered personal information management program that would dazzle users with its ability to enhance their productivity.

On that level alone, the book is successful, even though Chandler's development process has been bumpy, and some four years after the project began, it's still not finished. Like an anthropologist recounting the behavior of some hitherto unknown tribe, Rosenberg brings us inside the world of this particular group of software developers and documents their quest with detail and insight.

But the story of Chandler is also a platform for Rosenberg to explore much larger questions about the nature of software. Such as, why is it so hard to pull off a big project on deadline? What is it that software developers actually do? How is the art of writing software different from other forms of creative endeavor or technological construction?

With the years-past-deadline rollout of Microsoft Vista this past week, Rosenberg's questions (and answers) are especially timely. And given his long association with Salon, we are delighted to showcase them here, even as he dashes between speaking engagements at Microsoft and Google and Yahoo and writing Op-Eds for the Washington Post.

Disclaimer: Scott Rosenberg was responsible for Salon's hiring me 10 years ago, was my editor and boss for many years, and is a close personal friend. My daughter baby-sat his twin sons as I "interviewed" him. So I'm utterly biased and completely partial. But "Dreaming in Code" is still a darn good book.

Is it fair to summarize "Dreaming in Code" by saying that it is basically an attempt to answer the question: "Why is writing software so hard?"

Yes. That was the question I started with. It dates back to the experience at Salon that I start the book with. I had moved from being a journalist and an editor to being the managing editor of a Web site, and we had decided to build, on our own, some software for Salon: a new content management system. But as I tell it in the book, it turns out to be a software disaster, a classic death march situation in which everything went wrong and nothing worked as planned and when we deployed it everything broke.

And it was inconceivable to me, as it was happening, that this could be so. The fact that we were undertaking this thing that many people had done before us, we weren't the first, and that it could be so out of control, so unpredictable -- that threw me. I thought, OK, I don't know anything about this, I'm a writer and a journalist. I'm ignorant, but let me go educate myself. And so I did, and the first book I read was "The Mythical Man Month" [Frederick Brooks' classic study of the difficulties inherent in large software projects]. And I quickly realized that our experience was not unique at all. In a way it was the norm.

I've noticed that this week you've been in demand as a commentator, both in the Washington Post and on Marketplace, with respect to the launch of Microsoft Vista. Are you saying that there is a common thread between the problems that plagued our little Web site, and the delays experienced by the biggest software company in the world in rolling out its newest version of Windows?

Well, they're clearly hugely different undertakings. But there is probably more than one common thread. The one that comes first to mind is simply the incredible difficulty of estimating the time it takes to do this stuff, whether you are building a little content management system for a relatively modest-size Web company or whether you are building the operating system that will be used by three-fourths of the known universe. The difficulty in saying, A) How long will it take to do what you want to do? And B) When are you done doing what you want to do?

But you know, there are people in the field of software development who would look at Salon's story and say: "This is a very ill-disciplined project," and they would look at Chandler, the project at the center of "Dreaming in Code," and they would say, "That is a very ill-disciplined project," and they would look at Vista and say, look, they spent a couple of years trying to rewrite the file system and take on all these ambitious things -- "They were really ill-disciplined."

Can these people point to actual examples of disciplined projects?

Yes, they can, but usually they are extremely narrowly defined, well-known things -- situations where you have a very specific need, like an embedded system, something that is running on a single piece of hardware that is part of some larger system, and all you need to do is write the code for it and everything is specified in great detail. Those kinds of systems can come in on schedule.

But any time you look at a project that is trying to do something new, you have a different story. In the book this is what I call, in a sort of tongue-in-cheek way, Rosenberg's Law. I got to this point in my writing where I realized that everybody else in the software world has a law -- there's laws galore! -- and surely, if I'm going to contribute anything in this book, I have to find something that I can hang the label "Rosenberg's Law" on. But in fact, in the course of writing this story, it came to me in the middle of an argument that I was making that as long as you are not trying to do something new -- if you are doing something that has already been done -- then you have a frame of reference to estimate how long it is going to take, and to guess how many people are going to be required, and so on. And of course in other fields of engineering where we do the same thing over and over, the same is true. If we are going to build a house, there are always imponderables, and every site is different, but you have a rough idea: two-story house, and such and such a set of features, it is going to take a certain amount of time.

Next page: "Programmers are programmers because they like to code"