considered harmful: adj. [very common] Edsger W. Dijkstra’s note in the March 1968 “Communications of the ACM,” “Goto Statement Considered Harmful,” fired the first salvo in the structured programming wars … use of such titles has remained as a persistent minor in-joke (the ‘considered silly’ found at various places in this lexicon is related).
That entry in Eric Raymond’s edition of the Hacker’s Dictionary was my first encounter with pioneering computer scientist Edsger Wybe Dijkstra, but thanks to the dedicated work of volunteers at the University of Texas at Austin, it was very far from my last. These volunteers maintain the massive and growing EWD archive. It’s a tremendous and erudite proto-blog, the extraordinary record of an exemplary life, and it’s one of my favorite places on the Web. A year after his death, a computer scientist who devoted himself to teaching people how to think is still on the podium, delivering gem after gem of insight.
Born in the Netherlands in 1930, Dijkstra was a witty and thoroughly engaging writer in his nonnative English (“I have learned to be very suspicious of ideas I cannot express well in both Dutch and English,” he noted, late in life. “As nice as it is to have the union at one’s disposal, it is wise to confine oneself to the intersection.”)
Over a 40-year period that began in the early 1960s, Dijkstra wrote prolifically on timely and compelling topics: from his experience of the evolution of universities on both sides of the Atlantic from the post-WWII era to the beginning of the 21st century; to meditations on the science and art of teaching; to incredibly rich and detailed accounts of his own intellectual methods (don’t miss EWD 666: “A problem solved in my head,” which contains the endearing aperçu: “Goldbach’s Conjecture — I had never thought that I would ever use that!”)
Like entries in a modern weblog, many of the informal pieces collected in the EWD archive were never published in any traditional sense. Instead they were copied (and later photocopied), numbered sequentially from EWD 0 (sadly lost to history) to EWD 1317 (“From van IJzeren’s correspondence to my aunt & uncle,” written a few months before his death in August 2002) and circulated from the greedy hands of one computer scientist to another like Eastern European samizdat or fourth-generation copies of the Lions books.
For years I have been dipping into this priceless archive (or at least its English language subset; is there a great Dutch-English translator out there who would do the world the incalculable favor of translating the rest?) and I have yet to scratch the surface of its treasures. But I continue to follow the trail; the archive is redolent of the spoor of Dijkstra’s intellectual evolution, the physical evidence of a great mind thinking aloud. A fine, clear light shines through it all, the light of intelligence unmarred by any particular arrogance or egotism — the set of personal qualities I tend to think of as integrity.
Dijkstra is at his iconoclastic best on, for example, academic hypocrisy:
“Today’s mathematical culture suffers from a style of publication, in which the results and the reasoning justifying them are published quite explicitly but in which all the pondering is rigorously suppressed, as if the need to ponder were a vulgar infirmity about which we don’t talk in civilized company.”
Or the relationship between programming and mathematics:
“Programming is one of the most difficult branches of applied mathematics; the poorer mathematicians had better remain pure mathematicians.”
Or the truth itself, however unpalatable:
“French science is poisoned by politics.”
One particularly apposite piece (EWD 696) is titled “Written in anger.” What’s illuminating about it is what rouses Dijkstra’s ire: those who try to simplify or abstract away the innate difficulty of solving problems. The particular occasion of his anger is a 1978 Ph.D. thesis recommending the use of “easily managed graphic display systems” for designing computer programs. Dijkstra calls the suggestion “the most severe disservice to program design that I can imagine.” He explains:
“Most people, including mathematicians, are amateur thinkers in the sense that they have not been taught how to think effectively. They have not been told to throw the crutch away and as a result, have never learned how to run.”
The crutch in this context is the use of visual imagery. Dijkstra makes various good arguments against the use of images in thinking about geometry and mathematics and programming (diagrams are invariably overspecific, for example, and in set theory may be actively misleading.) He concludes:
“A major component of learning how to throw the crutch away is the ‘unlearning’ of the use of pictures. (And ‘unlearning’ is very difficult, as your past remains your past: the only thing you can do is superimpose a new past on top of the old one, and pray that the more recent past will be dominant.)”
This is a remarkable argument, not least from a European who came of age in the aftermath of 1945. What Dijkstra rejects is oversimplification. What he demands in its place is something like direct confrontation with the problem, as nearly as it can be defined. He’s perfectly willing to acknowledge how extremely difficult it is — both at a professional and at a personal level — to grapple with the world’s complexity so profoundly; and yet he is not prepared to settle for less, either in his students or in himself.
Reading Dijkstra’s account of a January 1974 meeting in Albuquerque, N.M., at which he turned out to be the main and only attraction — a prospect that frankly terrified him — made me envy the participants.
“I was very afraid … people come with such utterly unrealistic expectations that one is bound to disappoint them … In order to save the situation I used the first half of the morning to display all my misgivings, all my feelings of uncertainty … I felt that honesty alone could save me.”
As an eight-year veteran of computing and programming conferences, academic and otherwise, I can only imagine how charming and beguiling such a humble confession must have been — all the more for coming, as it did, from a man already world famous for his contributions to computer science. Not surprisingly, then:
“What happened during that one-man show of five days is unbelievable and exceeded my wildest hopes … once they picked up the rules of the game they inspired me to an extent that I have not experienced since a long time and I am perfectly willing to believe the many participants that told me on the last day that this week had been a unique experience.”
The rules of what game? What was it that Dijkstra was trying to teach? As these excerpts suggest, it was only nominally the teaching of programming. Indeed, many self-styled “educators” in today’s world of “information technology” would scarcely recognize it as teaching programming at all.
In a pivotal essay, EWD 473, written in 1975, Dijkstra explicitly rejects the acquisition of laundry lists of programming languages (in his day, FORTRAN, ALGOL 60, COBOL; today’s equivalents might be Java, Perl, C++). He calls such courses “driving lessons” and snipes that “no one tells [the student] that all those bells and whistles — those so-called ‘powerful features’ — belong more to the problem set than the solution set.”
How disconcerting is that? Take away pictorial aids, take away the very programming languages, and what on earth is left to teach? Dijkstra’s reply is more disconcerting still.
“I shall confine my attention to the difficulties of solving intrinsically hard problems.”
Dijkstra tried to teach his students how to think. Characteristically, he immediately acknowledges the difficulties involved, defines his terms, and narrows the scope of his inquiry. He’s not talking about neuroscience. He’s not talking about writing poetry. He’s talking specifically about reasoning — a term he uses to describe formal manipulations: think of arithmetic in primary school, algebra in secondary school, and symbolic logic in college. This kind of reasoning, he admits, is very demanding, and so he calls for a preliminary step aimed at reducing the amount of reasoning required.
This preliminary step is what he calls “pondering.” It’s so unfashionable even to talk about it, he laments, that many mathematicians can’t describe how they do it — which means, in practice, that they don’t even understand how they do it. The notion that the ability to ponder is not innate, that it could be known and taught, is laughed off or worse: “experienced as a threat upon their ego.”
What is a poor computer scientist to do?
“Suppose you stop teaching results and solutions, but start to solve problems in the lecture room and that you try to be as explicit as possible about your own pondering. What will happen?”
Well, most likely, all hell will break loose. But if you’re lucky:
“The need to get some sort of verbal grip on your own pondering will by sheer necessity present your ponderings as something in which, as time progresses, patterns will become distinguishable. Once you have established a language in which to do your own pondering, in which to plan and to supervise your reasoning, you have presented a tool that your students could use as well, for the planning and supervision of their reasoning.”
Geek that I am, I find this passage incredibly touching. It’s the combination of Dijkstra’s searing integrity and his humility and willingness to make a complete ass of himself, by actually standing up and pondering aloud in front of his students, for their sake, that gets me every time. I wonder if the success of the scientific method does not depend on exactly this combination of integrity and humility? Dijkstra doesn’t just advocate it. He models it.
It took me a while to realize it, but thanks to the EWD archive volunteers, my envy for the participants at the 1974 conference in Albuquerque is redundant. Reading the archive offers the same experience. Brilliant as the content is, the performance is better still. In my master’s steps I trod, where the snow lay dinted; heat was in the very sod which the saint had printed. “Ah,” you begin to think, “so that’s how he did it!” or even, as a tiny spark ignites somewhere inside you: “Maybe that’s how I could do it…” Dijkstra’s great gift, it would seem, was to inspire (tempt?) others, not to replicate his own feats of mind but to attempt their own.
Chronologically, the EWD archive may qualify as a proto-blog; qualitatively, it’s what the very finest blogs aspire to be.
The EWD archive is a priceless cultural artifact of the computer age. To its maintainers I say, wholeheartedly: Thank you. To anyone who has ever cared about programming or teaching or simply thinking hard and well about difficult problems I say, ardently: GOTO.
When Andrew Hume presented the Usenix Lifetime Achievement Award in San Diego in June, he managed to say exactly two words — “Richard Stevens” — before a standing ovation drowned him out. “I sat next to Richard’s family at the presentation,” says Tom Christiansen, a well-known figure in the Perl programming community who had known Stevens on and off for years. “It was stunning. I don’t know if his family did, but I sure noticed a lot of the audience in tears.”
“Usenix,” (a word coined to get around trademark restrictions on the word “Unix”) is the Advanced Computing Systems Association. W. Richard Stevens is the author of “TCP/IP Illustrated” and “Unix Network Programming,” each of which runs to three volumes, and “Advanced Programming in the Unix Environment.” Their influence among Unix users is hard to overstate. Thousands of programmers all over the world consider Stevens a guru and his works essential to their jobs.
“It blew my mind,” says his sister, Claire Stevens. “I knew he wrote those books, but it never made a dent. I had absolutely no idea that all these people knew and were touched by him.” Claire and Richard’s wife, Sally, accepted the award on Stevens’ behalf. Stevens died on Sept. 1, 1999. He was 48 years old.
His death hit the close-knit Unix community hard. Fiercely intelligent and deeply private, Stevens set an example for everyone in the Unix world. What he didn’t know, he determined to find out; what he did know, he strove to pass on to anyone who was interested. A year after his death, memories of one of the Unix community’s most beloved experts are still fresh and vital.
Christiansen’s recollection is typical. The two were casual acquaintances from the academic conference circuit. “I remember I was doodling around on the piano, and Richard came over and said, ‘I heard you doing that at some other conference and it inspired me to take up the piano again,’” Christiansen says. “On subsequent meetings he would tell me all about his progress. As with everything he talked about that he really loved, his eyes just kind of sparkled.”
With Christiansen, Stevens talked about music. With cryptographer Greg Rose, a fellow pilot, he talked about flying. With Dave Hanson, who sat on the committee to assess his doctoral thesis, Stevens talked about yet another shared passion, skiing. To everyone who knew him, it seemed he cared about the things that mattered most to them.
“He was a very good listener and he knew something about every subject,” says Claire. “He could always contribute something, or at least sound intelligent.”
Yet Stevens was also an extraordinarily private man. Christiansen, Rose and Hanson all knew him for years, yet none felt that they knew him well. “I wouldn’t say he was complex, but because of his intelligence he could come across that way,” Claire admits.
He brought the full force of his intellect to bear on his work, and it showed. Stevens’ books are regarded as models of the genre. “I remember he shocked me, because I ran into him at some conference and he said, ‘I wish I’d brought the ‘Perl Cookbook’ so I could have had you autograph it for me,’” says Christiansen. “I don’t consider my books to be in anywhere near the same league of serious, hard work as his. That he would say to me ‘I should have got your autograph’ was like Dennis [Ritchie, one of the original authors of Unix] saying it.”
“He gave extremely cogent explanations of what’s going on,” says Rose. “His books are so much more readable than most computer books. A lot of authors look at the documentation and rewrite it with little or no interpretation. He took the time to understand.”
His perseverance was an artifact of an apparently ferocious curiosity. “When I hit something that I don’t understand, I take a detour and learn something new,” said Stevens in an interview two years before his death. “This often makes my books late by a few months, but I think accuracy and completeness are essential.”
It may have been Stevens’ sense of himself as an outsider that inspired this level of dedication. He and his brother and sister were copper-mining brats, born in Africa to a metallurgical engineer; they divided their childhood among northern Rhodesia (now Zambia), Utah, New Mexico, Washington and South Africa. Stevens didn’t start out in computer science at all, but in aerospace. It just happened that he graduated in 1973, when Boeing was laying off thousands of aerospace engineers. Programming, like piano, was a late acquisition that took.
“I really believe that my background is fundamental to the success of ‘Unix Network Programming’ and my other books,” he said. “That is, I was not one of the developers at Berkeley or AT&T, so the writing of UNP was not a ‘memory dump.’ Everything that is in the book I had to dig out of somewhere and understand myself.”
In fact, Stevens belonged to the generation of Unix geeks who pored over not-exactly-legal copies of John Lions’ legendary commentary on the Unix source code. Lions firmly believed that no one could understand the theory of computer science without concrete examples, ideally embodied in C programs. Stevens seems to have agreed.
“Certainly his books do that. He tends to walk you through the code,” says Hanson, who led the graduate seminars at the University of Arizona where Stevens discovered the Lions books. “It’s not like you can read this stuff while you’re sitting in front of the television. It’s a particular style of learning about software that requires a very heavy investment upfront. But it pays off.”
It’s not as easy, or as obvious, as it sounds. Publishing code legibly is hard work, and calls for stubborn authors who insist on the best tools for the job. Stevens greatly admired and strove to emulate Donald Knuth, who wrote “The Art of Computer Programming,” and Brian Kernighan, “The C Programming Language,” whose books are as beautifully laid out as they are brilliantly written.
For his own work, he insisted on a text-formatting tool even more venerable and revered than Knuth’s Tex. “Rich had told me that he gets to send ‘troff,’” says Christiansen in awed tones, referring to a fairly archaic layout program. “He still used all the cool old tools, and he prepared camera-ready copy for the publisher. It’s very, very rare that authors aren’t forced to use Microsoft Word or unadorned SGML.” Controlling his own means of production allowed Stevens to explain difficult concepts visually as well as verbally — hence the apparently perverse theme of “TCP/IP Illustrated.” Visualizing a protocol? It sounds crazy, but it works.
His books are so good that they have come to symbolize intelligence. In “Wayne’s World II,” Garth’s girlfriend carries a copy of “Unix Network Programming.” Stevens discovered this when he took his 13-year-old son to see the film. His son grabbed his arm and said, “Dad, that’s your book!”
“I couldn’t believe it,” he told programmer Trent Hein. “My book was used to define the ultimate geek, and suddenly my son thinks I’m really cool.”
His son was right.
12 July 2000
“Incidentally, Napster was Shawn Fanning’s nickname in high school, after he got an extremely short haircut. Just in case, y’know, anyone was still wondering about that.”
“rachel”
Let me say, right upfront, that I don’t deserve to be on Advogato, and that all the people who have certified me as an “apprentice” are just being kind. (Thanks, people!) I use MacOS at home, Windows (gasp) at work and Linux only occasionally. My sole qualification for membership in the open-source community is that I enjoy the company of engineers very much, and seem to get on with them fairly well. It is, therefore, the height of cheek for me to keep a diary on Advogato, but I do.
Advogato — it’s a play on the word advocate, but the place is routinely called avocado or guacamole — is an eight-month-old Web site designed to make life a little easier and more fun for free-software developers. It is both a community hangout where hackers cluster, jotting down daily tidbits of info in publicly accessible diaries, and a forum for discussion, like Slashdot. There’s also an extra twist: On Slashdot, readers can rate the value of posts as part of a not-always-perfect filtering mechanism. But at Advogato, people rate one another.
Advogato has three elements: certificates — apprentice, journeyer, master — that measure your status in the community and that are awarded by members to other members; a Slashdot-like article feed, less frenzied but arguably much more thoughtful; and the diary-hosting feature. Thrown in almost as an afterthought, the diaries are by far the best part of the site. The “recent entries” page is particularly addictive. A compilation of the 10 or 20 freshest diary entries, haphazardly juxtaposed, it’s a continually refreshing portrait of the free-software community, a gestalt dream journal, a gossip column, a webcam in prose.
Advogato was built by Raph Levien, a UC-Berkeley computer scientist, as part of his Ph.D. research into the gnarly problem of creating “webs of trust.” There are two basic approaches to building such webs.
You can assemble a strict hierarchy with “God” at the top, authorizing the priests beneath him, who oversee the commoners beneath them in their turn. This is what VeriSign, the largest commercial encryption security company, has done. The company’s model, in which it plays the role of God, corresponds closely to open-source evangelist Eric Raymond’s idea of the “cathedral” style of software development, in which coders are coordinated from on high as part of a master plan. Cathedrals tend not to be popular in the free-software community, where everyone wants to do his own thing.
Advogato is an experiment in the alternative, peer-to-peer model, heavily inspired by Phil Zimmerman’s Pretty Good Privacy site. “I found that the traditional trust metric framework was pretty limited, but discovered a new technique, which I call a group trust metric,” Levien explains. “Basically, you put in a bunch of certificates of the form ‘I, person A, vouch for the fact that person B is a member of this group,’ and the trust metric tells you who it thinks is really in the group.”
In other words, to qualify as a master on Advogato requires that the community of your peers agree that you’re a hotshot.
12 July 2000
“chinese buffet for lunch, not the good one either. my stomach is kind of rumbling around in circles, i dont even want to look at the AIX failovers. oh well, back to work.”
“spot”
What’s revolutionary about Advogato’s model is that like the Net itself — and unlike VeriSign’s top-down bureaucracy — it’s self-organizing, self-repairing and therefore hard to corrupt or otherwise compromise. “In all previous systems, once you get a certain number of wrong certificates, the whole thing falls apart,” says Levien. In Advogato, at least in theory, the system should continue to function even if abuse is widespread. But his theory sounded too good to be true. Without hard evidence, Levien found, peer reviewers kept rejecting his papers.
At the same time, he was starting to feel the need for a community Web site for free-software developers. In spite of its efforts at moderation, Slashdot had become a victim of its own success, its signal-to-noise ratio infamously low. “It occurred to me that I could kill two birds with one stone, and build a community site that used the group trust metric to define the outline of the community,” says Levien. “With these goals in mind, I started hacking like crazy, and Advogato was born.” Article zero went up Nov. 6.
And suddenly, the hardcore hackers who had grown disillusioned with the venerable and increasingly creaky Slashdot had somewhere far more interesting to go. The latest stage in the evolution of online community had come to pass.
12 June 2000
“I got a call from my cousin today. My grandmother died Saturday night. Life sucks. My initials are entirely appropriate right now.
“The sky is a terribly brilliant blue today.”
“sad”
Ever since I discovered Advogato, I’ve hit the recent-entries page three or four times a day. Not a great deal happens, to be honest. There have been one or two deaths in the extended family, two or three births, four or five breakups, six or eight episodes of depression. I’ve learned, for example, that Deb Richardson of the Open Source Writers Group — “dria” — has moved to Montreal, where she’s having a very satisfying time furnishing her apartment.
“I’m still occasionally questioning my sanity in this ‘I need a huge dining suite’ thing,” she writes, “but it’s way too late now.” Meanwhile Stephane, Eskil and Maciej were lucky enough to find a rental in San Francisco’s sought-after Castro District, although they were all greatly puzzled by Clause 26 of their lease — “Time of Essence: Time is of the essence.”
Not everything on Advogato is quite so quirkily benign. Rob “lilo” Levin (certified master) and Elise Shapiro (journeyer) used to work at Linuxcare, until Rob was laid off and Elise was fired. Dave Sifry (journeyer), who founded the company, also keeps a diary at Advogato. “I wish the best for lilo, and Elise and others caught up in recent goings-on at the job,” he wrote in May, in the thick of the layoffs. “I see your diary entries and I’m glad that things are working out well for you all.” It sounded pretty awkward, but what else could he have said?
“The worst thing, I think, was the ‘dimwit’ fiasco,” says Levien. “People had been continually asking me to add new levels. I finally broke down when Elliot Lee offered to join up if he could get certified as ‘dimwit’… I thought people would have fun trying to jockey for low status, sort of a joke on the people who were getting wrapped up in having a high cert[-ificate] level on Advogato. Unfortunately, some people didn’t take it in the intended spirit of fun, and some feelings were hurt. It’s gone now.”
3 July 2000
“I’ve been messing around with doubleclick.net’s heinous cookie system and come up with a programming problem and an anti-tracking page. I wonder what kinds of responses I’ll get.”
“dmarti”
Public exposure has its price. Philip “bryce” Copeland, a Red Hat programmer, got burned by what he calls “the infamous Piranha security hole fiasco.” A security consultancy announced that it had found a “back door” into a Red Hat software utility called Piranha.
“I originally wrote up a response to the press critics in as fair and as unbiased a fashion as I possibly could,” says Phil. “Unfortunately a few press houses decided they’d blatantly misrepresent what I’d written to sensationalize the story. I don’t mind being quoted; however, I would have liked it to have been in context of the spirit in which I’d written it.” Now there’s a disclaimer on his Advogato home page: “For people that want to use these scribblings for news articles regarding Red Hat, please, please, please contact the good people in Red Hat’s PR group *first*.”
So who does contribute, and why? Members include famous Linux and BSD (Berkeley Software Distribution) hackers like Jordan Hubbard, Miguel de Icaza, Bruce Perens, Eric Raymond and Jamie Zawinski, although few of them are active on the site. Others purport to be God, Satan, Nietzsche, Richard Stallman and Need to Know’s Danny O’Brien, but these claims have yet to be confirmed.
Seth David Schoen, the hacker behind Linuxcare’s bootable business card — a full Linux installation on a cut-down CD — is one of Advogato’s most prolific posters. “I like the opportunity to keep a diary on the Web,” he says. “I would have wanted to do that earlier, but I didn’t have any kind of excuse.” Seth, it must be said, is something of an exhibitionist: “Most of the time the idea of sharing some part of my life on the Net is pretty appealing to me. I used to have a thing called whereami … I would send myself e-mail with keywords and that page would update immediately with a map and sometimes a picture of where I was. Some days I updated that six or seven times a day.”
Like other open-source projects, Seth noted, Advogato goes through cycles of activity and quiescence. “A bunch of people from some community (e.g., Bay Area Linux user groups, employees of a particular company, developers on a particular project) will get involved, and all enjoy reading one another’s diaries all the time. That keeps them coming back.” Then the novelty wears off. “When some of them get bored, they stop updating their diaries, which makes Advogato less interesting for other people … People have certainly come and gone in waves.”
Among the groups that have fallen in and out of love with Advogato are the technical staff at big-name companies (Red Hat, Linuxcare and VA Linux Systems), lesser-known start-ups (Eazel, Helixcode, Tuxtops and Canada’s Zero Knowledge Systems), mailing lists (Linux elitists and Crackmonkey) and semilegendary secret society/defunct coffeehouses like the Linux CABAL. But given the very large extent to which these cliques intersect, it’s hard to say who actually joined through which connection.
It should be emphasized that the above list is San Francisco-centric simply because these are the people I know personally and the connections that have caught my eye. Though Advogato was born in Berkeley, it now has users all over the world. The site is fun for the same reason the free-software community is fun. Both are full of smart, interesting people, many of whom are also witty and all of whom are engaged in and passionate about their work.
Advogato’s creator is a case in point. “The best payoff for me is seeing it strengthen the community,” says Levien. “People routinely help each other out with technical problems, and it’s also a great way to get a feel for what’s happening in the free-software world. I’ve gotten to know a number of people through Advogato, several of whom I’ve met in real life. Plus, I now have a real certificate graph I can use in my research!”
11 July 2000
“Today is a sad day for Debian. It is odd how you can feel you know someone as a friend, and yet have never met that person in the flesh. To our friend, who will remain unnamed for a few more days … you shall always be remembered as a part of what some people (quite aptly, perhaps) call the most dysfunctional family there can be. We may quarrel, but we all share a passion for the cause. We shall all miss you.”
“tausq”
Not many Linux-come-latelies know this, but Linux was actually the second open-source Unix-based operating system for personal computers to be distributed over the Internet. The first was 386BSD, which was put together by an extraordinary couple named Bill and Lynne Jolitz. In a 1993 interview with Meta magazine, Linus Torvalds himself name-checked their O.S. “If 386BSD had been available when I started on Linux,” he said, “Linux would probably never have happened.”
Linux obviously did happen. Why? Eric Raymond, the open-source evangelist, believes it came down to a question of personal style. In his A Brief History of Hackerdom he praises 386BSD at the expense of the “crude” versions of Linux that were around at the time.
The deciding factor, argues Raymond, was not technological but social. Torvalds, even while practicing rigorous quality control in determining what goes into the Linux kernel and what stays out, nevertheless welcomes contributions and is remarkably generous in sharing credit.
The Jolitzes had a very different style. Like Torvalds, they placed a premium on quality control, but unlike him, they seem to have tried to control quality by doing most of the work themselves. This inevitably made their release cycle slow, but it was also an implied snub to would-be collaborators — who took their contributions elsewhere.
The Jolitzes’ insight that the world needed an open-source Unix-like operating system running on Intel’s x86 microprocessors has been triumphantly borne out by history. Yet outside the Unix community they remain virtually unknown. In trying to earn most of the credit for their insight, they lost nearly all of it.
Bill and Lynne Jolitz are charming and forthcoming. Their chemistry as a couple is evident; they interrupt each other fluidly and finish each other’s sentences. Their pride in their achievements is equally evident. Lynne is heated in her defense of Bill’s reputation. So is Bill.
In 1989 or 1990, Lynne remembers, the Berkeley distribution had gone from being available on the most relevant machines to being limited to what the Jolitzes saw as the most irrelevant. “There was an HP [Hewlett-Packard] port in progress and nothing else,” she says. “Since we were looking for recreation, we offered to do one for the 386.” “Completely as a lark,” Bill adds.
Intel’s now-ubiquitous chip design for personal computers, the x86, was then in its infancy. This daunted the Jolitzes not at all. “The choice of architecture was obvious to us then,” Lynne says. “People said why not [Digital Equipment Corp.'s rival chip] Alpha? But it was clear that the x86 architecture was going to be the dominant player.” “Some of the most promising people I knew at the time were hoping the 386 would die an early death,” Bill observes. “It was a harbinger,” says Lynne, “way ahead of its time.”
“I had been talking to Intel engineers,” Bill says. “Their plans were to double performance every 18 months, and they believed they could keep that up for close to a decade.” (They did.) “The rate of improvement on a VAX was not nearly so competitive. The 386 was already faster than a DEC mainframe even without that happening. Yet a senior researcher at DEC asked me, ‘When will you stop using Unix on these penny-ante machines?’”
“They figured it was a DOS chip, so it couldn’t be anything interesting,” says Lynne, witheringly. In fact, the Intel architecture was far from perfect. “Parts of the chip were visionary, parts were terrible,” she recalls. “It was like a camel.” Bill agrees: “We had a joke that one day we’d be working on the 1-billion-86, and they still wouldn’t get segments [a method for storing memory] right.” In spite of its problems, the Jolitzes believed in Intel’s chip, and the Unix port went ahead.
The Jolitzes released Version 0.0 of 386BSD on St. Patrick’s Day 1991. It was barely functional, but available to anyone with 30 floppy disks to spare. The system shipped under a BSD-style software license, permitting it to be freely distributed and modified as long as copyright attribution remained intact. But the Jolitzes knew even as they announced it that this release was not enough.
“As we were doing and documenting the port, we realized we’d eventually have to do a complete release,” Lynne says. “This was a very arduous process. We had to throw away all the licensed portions. We started to do novel work. We realized that a lot of what we had thrown away were first cuts. It’s ironic that people have spent so much time and money on stuff developers just threw in.”
The Jolitzes believed that from its origins as a series of quick, if elegant, hacks, Unix had hardened into a series of unquestioned rituals for getting things done. Many of these rituals were fossils — work-arounds for hardware that no longer existed. “It’s amazing how much we were still tied to the past,” Lynne says. “The physical machines had been hauled away, but elements of the operating systems are still being implemented in the same way.” She believes today’s Linux and BSD developers are carrying on that unhelpful tradition.
“It’s like they’re doing incantations,” she says. “They’re repeating what they’ve been taught, and they don’t know what it means.”
“There are all these artifacts of minicomputers still left in,” Bill agrees. One example is the allocation of address space and garbage collection. Put simply, this is how the machine decides where to put numbers it needs to remember, and when to clean up numbers it no longer needs. Bill criticizes what he describes as the “linear” idea of address space enshrined in Unix: Memory is assigned to the next free block of appropriate size. “It’s adequate for simple memory allocators but a massive problem for large allocations and clusters,” he says. “You end up with lots of little pools of memory.”
“There are ways to solve this. It’s cut and dried,” says Lynne. “You can ask anyone. Yet even trying to implement the changes necessary to improve the system results in incredible controversy, because you’re removing something somebody knows. You have to have been there to know that a lot of these elements of the operating system were very rough first cuts.”
“The Unix systems then offered eight or 10 different competing mechanisms to do basically the same thing,” Bill says. “The question in 386BSD was, How many of those could we get rid of? We increased the capabilities and reduced the size by a factor of 35.”
This slash-and-burn approach to the work on the new release spawned a series of articles in the respected journal Dr Dobbs. Those articles may have been even more influential than the code. A generation of O.S. developers — including Torvalds, then a student in Finland — studied and learned from them. On Aug. 25, 1991, Torvalds posted a fateful note to a Usenet news group devoted to the experimental operating system Minix: “Hello everybody out there using minix,” he wrote. “I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since April, and is starting to get ready.”
If this blithe note represented a serious threat to their work, it didn’t register with the Jolitzes. The articles in Dr Dobbs primed their audience for the release of 386BSD 0.1 on Bastille Day 1992. The tone of their announcement forms an illustrative contrast with that of Torvalds:
“We are pleased to announce the official release of 386BSD Release 0.1, the second edition of the 386BSD operating system created and developed by William and Lynne Jolitz and enhanced further with novel work and contributions from the dedicated 386BSD User Community. Like its predecessor, 386BSD Release 0.0, Release 0.1 comprises an entire and complete UNIX-like operating system for the 80386/80486-based AT Personal Computer.”
There were 250,000 downloads of 386BSD 0.1, Lynne remembers. “We had people showing up on our doorstep,” Bill says. “It was a fun time,” Lynne agrees. “We thought there would be maybe a few hundred downloads. So much for the power of the press!” Work continued apace. The Jolitzes aimed high. “When you do a release, it takes months and months to shape it up to make it usable,” says Lynne, demonstrating a distinctly different approach from the Linux mantra: Release early, release often.
Instead, in December 1993, there came 386BSD 1.0, a revised version with new internal interfaces, allowing the individual components of the operating system to talk to each other more cleanly and efficiently. But somehow, in the 18 months between 0.1 and 1.0, the project lost its momentum. Version 1.0 was widely criticized for being too little, too late.
“The idea behind what we were after was a system that always had network access, that could detect faults and repair itself on the fly,” explains Bill, heaving a sigh. “Sometimes you get too far into the future,” he remarks. “You have to stop and wait to see the future catch up.”
By the time 1.0 was released, the x86BSD user community had fragmented. Some developers had moved to the more active and open NetBSD and FreeBSD teams. Others had jumped the fence to Linux. A lawsuit between Unix copyright holder AT&T and University of California at Berkeley was also partly to blame. But the Jolitzes, too, were criticized for their autocratic style. The strength of their convictions did not endear them to people who wanted to do things differently.
In return, they say their intellectual heirs have largely missed the point. “The xBSDs don’t have any understanding of where we were heading,” says Lynne. “They’re looking to the past. We look to it as: Don’t repeat the mistakes of history. When you haven’t been involved, you don’t know that Bill Joy threw something in late one night, or Dennis Ritchie wrote something up as an expedient. We take so much for granted, all the knowledge that we’ve accumulated, that we forget how much we worked for it. Someone writes it up very simple and they’ve solved it for us. We miss the rigor of solving it for ourselves.”
“Yet we continually re-solve the same problems,” Bill says. “Why something and why not another thing? It’s a difficult discipline. Ninety-nine out of 100 things may work, but only one may be the ultimate. Striving for the ultimate best is part of the Berkeley experience.”
Strive as they might, 386BSD lost its early lead over Linux, never to regain it. The Jolitzes ended as they began: voices crying in the wilderness. Their great insight into the software that came before them was the extent to which it was human and subjective and fallible, the partly accidental product of the conditions that brought it into being. Their great flaw, perhaps, was to fail to realize that the same principle applied equally to their own work.
Their code lives on in FreeBSD, NetBSD and OpenBSD, which many believe are technically superior to Linux. FreeBSD is often called the Net’s best-kept secret. It keeps Yahoo up and running and is at the heart of Apple’s Darwin operating system. The recent merger of Walnut Creek CD-ROM, the largest distributor of FreeBSD, with the commercial BSDi O.S., should infuse the operating system with fresh life.
If the glory of being Torvalds evaded them, the Jolitzes certainly helped prepare the way. With their most recent venture, InterProphet, the Jolitzes aim with typical hubris to eliminate the TCP/IP (Transmission Control Protocol/Internet Protocol) bottleneck and move bits around the Internet at light speed. Whether or not they succeed, they could hardly have chosen a more revealing name.
Before there was an Open Source Initiative, before the Free Software Foundation was even a twinkle in St. iGNUcius’ eye, Unix hackers were fighting lawyers and commercial interests for the right to copy and distribute source code. The fight began, in part, due to the beliefs of an avuncular Australian professor named John Lions, who thought that by making source code available and using it as a teaching tool, he could encourage the highest possible standards in programming. As the first anniversary of his death approaches and the open-source movement kicks into higher and higher gear, it seems a propitious time to remember Lions’ contribution.
I stumbled across Lions’ books in 1996. I’d majored in literature and seemed to have spent my entire life searching for a witty, literate man. When I finally found the man who would become my fianci, he was a Unix hacker. This baffled me. I couldn’t begin to imagine how the arid world of over-lit computer labs and humming server rooms could have produced someone so much droller and more insightful than my fellow humanities graduates. So I did what I always do when I want to get inside someone’s head: I browsed his bookshelves.
There I found — and devoured — Eric Raymond’s hilarious “The New Hacker’s Dictionary.” Written up from the legendary Jargon file, it describes a literary culture weirdly like my own, complete with movements, jokes, manifestoes and Great Works. And speaking of great works, I also found on those shelves the first editions of the Lions books. The books — a two-volume set titled “Source Code and Commentary on Unix Level 6″ — include not only the entire source code for the Unix Version 6 kernel but also a detailed and often witty discussion of it written in the mid-1970s.
The genesis of Lions’ books is, of course, tied to the Unix tradition that began 25 years ago when the magazine Communications of the Association for Computing Machinery — technology’s equivalent to Nature — published a paper by Ken Thompson and Dennis Ritchie, the two Bell Labs employees who created Unix and the C programming language. The paper was called “The Unix Time-Sharing System” and included a description of the operating system, a justification of its design and a few notes on why it was built in the first place. The article captured the imagination of many a programmer, including Ken Robinson, a teacher at the University of New South Wales (UNSW), who wrote away for a copy of the new operating system. When it arrived, Lions, his colleague, read the source.
Source code is the blueprint for software, like a set of spells that humans can read and use to control machines. Given source code, a programmer can modify an application by fixing bugs or adding features. But most commercial software is sold only in machine-readable form. Access to source code is power.
Lions’ career had followed the classic path for an Australian academic of his generation. In 1959, he took a first-class honors degree from Sydney University and promptly left the country. He earned his doctorate at Cambridge in 1963 and spent the next decade working for Burroughs Corp. in Canada and Los Angeles. By 1972 he was married and had a young family. He moved back to Australia and took a position as senior lecturer in UNSW’s department of computing. He would teach there for the rest of his working life.
The Unix code base enchanted Lions — so much so that he decided to make significant changes to two of the courses he taught. Until then, most teachers of operating systems loftily imparted general principles about programs their students had probably never seen, or encouraged students to build toy operating systems of their own. Unix offered a third approach. It ran on a comparatively affordable computer system, a Digital Equipment Corp. PDP 11 — a machine that UNSW already owned. Unix was compact and accessible but offered a remarkable set of features. To top it off, in Lions’ words, it was “intrinsically interesting.” Unix could be read and understood with far less effort than IBM’s bloated OS/360 and TSS/360 (which, funnily enough, are pygmies by modern standards), but it had the industrial-strength functionality to which homemade toys could never aspire. One student, Greg Rose, remembers, “John expressed this by saying, ‘The only other big programs they see were written by them; at least this one is written well.’”
The first editions of Lions’ books were slender computer printouts covered with red cardboard and stamped with the UNSW crest; they bore the titles “Unix Operating System Source Code Level 6″ and “A Commentary on the Unix Operating System.” Lions had originally prepared them for his students, who were astonished at the availability of the source code. Here was an entire operating system you could hold in your hand. “The whole documentation is not unreasonably transportable in a student’s briefcase,” noted Lions. Later he would joke that in subsequent editions of Unix, this had been fixed.
The books worked. “In general, students seem to find the new courses more onerous, but much more satisfying than the previous courses,” Lions dryly observed. His students describe the classes as a revelation. “He enjoyed seeing if any of us were alert enough to catch any of the few bugs or inexplicable oddities in the code as we read through it. I don’t think we managed it often,” remembers Lucy Chubb, now president of the Australian Unix and Open Systems Users Group (AUUG). “Serious reading of someone else’s code was something that no other subject gave us.”
One of Lions’ greatest insights was that what Thompson and Ritchie had written deserved to be studied in this way. “You will find that most of the code in Unix is of a very high standard,” he wrote in an introduction to the books. “Many sections which initially seem complex and obscure, appear in the light of further investigation and reflection, to be perfectly obvious and ‘the only way to fly.’”
Unix was beautiful and useful and it rewarded close attention. As programmer Peter Reintjes observed, “We had acquired what amounted to a literary criticism of computer software.”
The implications were huge. Thompson and Ritchie’s succinct and elegant code lent itself to investigation and play. Now a cadre of well-trained students was equipped to do just that. “Enhancements from institutions around the world began to be exchanged, and contributed in no small way to the growth of Unix,” Rose recalls. “These two volumes made it far easier to get started with this kind of experimentation, and contributed greatly to the success of Unix during the late 1970s and early 1980s.”
News of the revolution trickled back to its instigators. “The first Ken and I heard of John was when the original of the ‘Commentary’ book arrived, I think, out of the blue,” says Ritchie. “We were much impressed by the quality of the work and highly flattered by its very existence.” Lions had worked hard to understand what Thompson and Ritchie had been trying to achieve. In their view, he succeeded. “After 20 years, this is still the best exposition of the workings of a ‘real’ operating system,” Thompson has said.
As much as he admires the course notes, Ritchie praises Lions’ teaching still more. “Probably the most important contribution John made was to start, at UNSW and indirectly at Sydney Uni, a very strong group of Unix people, many of whom have visited or stayed here, and whom we have often visited,” he says. Lions also founded the AUUG, and it is at least partly to his credit that Unix thrives in Australia to this day. (The group recently celebrated Lions’ contribution with a John Lions Award for Research Work in Open Systems.)
A visionary teacher, a wonderful teaching tool, a commentary drenched with wit and insight, an environment in which talented students thrived — this is all building up to a happy ending, right? Yes and no. Even when they were first published, Lions’ books were technically only available to licensees of sixth edition Unix. The operating system’s new owner, Western Electric, didn’t want just anyone learning the inner workings of the Unix kernel.
“By the time the seventh edition system came out, the company had begun to worry more about the intellectual property issues and ‘trade secrets’ and so forth,” Ritchie explains. “There was somewhat of a struggle between us in the research group who saw the benefit in having the system readily available, and the Unix Support Group … Even though in the 1970s Unix was not a commercial proposition, USG and the lawyers were cautious. At any rate, we in research lost the argument.”
The seventh and subsequent licenses explicitly prohibited the kind of teaching that Lions had been doing. The USG also sought to control distribution of the commentary. It failed. Even then, Unix hackers were not the types to take this sort of treatment lying down. Seventh edition Unix was released in 1979, at about the same time that photocopiers were becoming affordable. Engineers added two and two. The Lions books were secretly copied and passed from hand to hand. Fourth- and fifth-generation photocopies became treasured possessions. The books were literally samizdat, the homemade literature of the resistance.
“Because we couldn’t legally discuss the book in the university’s operating systems class, several of us would meet at night in an empty classroom to discuss the book,” said Reintjes. “It was the only time in my life when I was an active member of an underground.”
This awkward situation lasted nearly 20 years. Even as USG became Unix System Laboratories (USL) and was half divested to Novell, which in turn sold it to the Santa Cruz Operation (SCO), Ritchie never lost hope that the Lions books could see the light of day. He leaned on company after company. “This was, after all, 25-plus-year-old material, but when they would ask their lawyers, they would say that they couldn’t see any harm at first glance, but there was a sort of ‘but you never know …’ attitude, and they never got the courage to go ahead,” he explains.
Finally, at SCO, Ritchie hit paydirt. He already knew Mike Tilson, an SCO executive. With the help of his fellow Unix gurus Peter Salus and Berny Goodheart, Ritchie brought pressure to bear. “Mike himself drafted a ‘grant of permission’ letter,” says Ritchie, “‘to save the legal people from doing the work!’” Research, at last, had won.
In 1996, Peer to Peer Communications reprinted the Lions books. It was fortuitous timing; John Lions was seriously ill. The moment the books arrived in Australia, Goodheart took several copies to him. “As you can imagine, his face just lit up,” Goodheart wrote at the time. “He was thrilled with delight and we all celebrated the occasion with champagne. His health is much worse than I thought but he was able to understand that his work was finally published. He tends to fall asleep during even the shortest conversation and very rarely says anything. Marianne is not sure if he understands everything but I think he does …” Lions wondered aloud what had become of the UNSW crest.
“He actually burst into loud hysterics when I read out the comment on the back cover, ‘The Most Famous Suppressed Manuscript in Computer History,’” Goodheart wrote. “Marianne said it had been a long time since she’d seen him so happy. John’s daughter Liz was there and she popped a question to him. ‘Dad, do you really understand this gobbledygook?’ John replied ‘No,’ and burst into further laughter …”
John Lions died on Dec. 5, 1998. Three years or so before his death, he had discovered a box of first editions of the “Commentary and Source” in his basement, slightly water-damaged but otherwise fine. He gave a set to a young programmer he knew. They were among the eight or 10 irreplaceable books my fianci brought with him when we moved to San Francisco last year. He’s working on the Linux kernel now — a system rich with brilliant programming because its source code has always been available.
I like to think John Lions would approve.
