Technology is changing our world -- and we should be afraid! Sun Microsystems chief scientist Bill Joy envisions a frightening future of self-replicating machines.



Are we headed for mass destruction? Are technological innovations laying the foundation for machines to take over the world? These are the questions plaguing Sun Microsystems co-founder and chief scientist Bill Joy.

Joy, who was the principal designer of the Berkeley version of Unix and one of the lead developers of Java, last made a big splash in the press in 1998, when he was all pumped up about Sun’s new networking technology Jini. Having spent most of his life working on ways to make computer networks work better, he was preaching the gospel of a technology that Sun said would enable truly user-friendly, plug ‘n’ play computing.

But these days Joy is engaged in an altogether different kind of crusade: He wants us to slow down the technological race, and rein in technologies — specifically nanotechnology, genetic engineering and robotics — that he believes could ultimately endanger human life as we know it. Joy spelled out in a recent issue of Wired magazine his fear that modern technologists may unthinkingly be heading down the same path as the physicists who built the atomic bomb.

At a recent Stanford University forum entitled “Will Spiritual Robots Replace Humanity by 2100?” he squared off against optimists, including computer scientist Hans Moravec, who argued that the future’s technological organisms — even if they wipe out humanity — represent a higher form of evolution that we should be proud to usher in. In the midst of such claims, Joy’s call for limiting or regulating research in nanotech, genetic engineering and robotics, played well to the crowd of about 1,000 who fought for space in the auditorium, but his fellow scientists remained unconvinced. That’s not stopping Joy, however, who is determined to get people to listen and take action.

Celera Genomics announced last week that it has finished sequencing the human genome, moving the debate about genetic engineering one step further from abstraction toward reality. Given your views on the potential dangers of genetics, nanotechnology and robotics, how do you think this new discovery and others should be controlled?

We have to do something. Because otherwise — within 20 or 30 years, probably sooner — if we give everybody personal computers that are a million times as powerful, they’ll have the ability to manufacture whatever they can design. We’ll have an untenable situation, because crazy people can design new diseases and build them on their own computers. Somebody has to take responsibility for that and figure out a way to not let that happen.

I mean, the people doing genetic engineering are doing really good stuff. It’s just that I don’t believe they’ve dealt with the confluence of their field and Moore’s Law. If the tools they had were just like the ones they always had, it wouldn’t be a problem. But look at Celera, finishing the sequence. How long was it supposed to take? I haven’t been keeping track, but they got it done ahead of schedule. And yet, I still don’t see that we have adequate mechanisms or the psychological desire to put any restrictions in place.

Then how do you remedy that situation, with genetics and the other technologies? How do you set up a system that keeps companies like Celera in check?

I’m not worried about the commercial companies. Some people could worry that companies could do something bad. But I just don’t want to get to the point where crazy people can do it. In the nuclear age we had two organizations with unlimited power, the Soviet government and the U.S. government. Others have nuclear weapons, but not at the level to destroy civilizations.

We may have many companies that have the technology in their labs to do very disastrous things. That’s probably difficult to manage, but it’s not inconceivable. But every person in the world? That’s inconceivable. We can’t ethically go there. That’s not acceptable. We can’t let ourselves get to that situation. There’s this great quote from Woody Allen: “More than any other time in history, mankind faces a crossroads: One path leads to despair and hopelessness, and the other to total extinction. Let us pray we have the wisdom to choose correctly.”

So in some sense people despair of having to go down a path where they would limit anything, and they feel hopeless to change it — that’s one of Woody’s two paths. On the other hand, going out and giving everybody all the information, then letting any crazy person destroy things, is completely and morally unacceptable. But we have a choice, and this is a real choice. We don’t face these very often.

Should we put Microsoft … I mean, Microsoft? They’re a bunch of people that don’t have much personal ethics, and they break the law all the time, so we kind of futz around. We do this to them or we do that — but it’s not like society’s going to crumble. Certainly if we continually fail to enforce the laws of reasonable conduct, our code of ethics in business will decline to the long-term detriment of our society, but the whole thing isn’t going to collapse because we failed to enforce the clear law in this case.

But something like this seems much harder. We’re talking about real consequences here. We’re not talking winners and losers. In my mind, everyone’s a loser on this path.

So then, who controls the technology, and saves us from ourselves? An international body? Scientists?

You Might Also Like

Well, the problem is that there are at least seven ways in which this technology differs from the nuclear technology of the 20th century, all of which makes the problem of regulation harder.

What are those differences?

The first thing is that everything’s happening so fast. Moore’s law is happening so fast. We don’t think things are possible, but in fact they’re already going to happen, for sure. The second is this amplification, where self-replication lets an individual do something that has a wide impact. For example, [in February's denial of service attacks] one person probably brought down the Web; that’s an example of replicated code. Then there’s the irreversibility of the actions. Even if you create a nuclear [war] winner who blows up the cities, it’s not the end [of all life on Earth]. But if you release something deadly into the environment, you probably can’t call it back. And they are commercial technologies, not military, so they’re really loose in the commercial sector, and they have enormous value. These things are going to create millions of dollars in wealth. It’s no surprise that the NASDAQ is going crazy. The technologies are very democratic: If it’s information, all you need to design it is a PC — maybe one that’s 10,000 or a 100,000 times faster than we have today. But it’s not like having a nuclear, plutonium and uranium refining facility — that’s not something everyone’s going to have. The information revolution fundamentally democratizes access to the tools.

The other thing is that these things really blur the lines between machines and life. We understand biology and we understand machines, but these things that these technologies are making are different. Say in nanotechnology, it’s machines that reproduce. They’re living. And if we mess with genetic engineering, we have to understand [cells] at a mechanistic level. And the last thing is, these things are so powerful … we really can’t foresee what the outcome will be. We can know where there’s a situation of great danger, but to predict anything with specificity is extremely difficult because there’s so much progress being made in parallel on so many fronts. It’s like Dolly — that was a complete surprise. The motto of the age has to be, expect to be surprised; forget fiction, read the newspaper. So each of these seven things makes the problem more difficult.

Couldn’t companies simply limit access to dangerous products, thus keeping them from the “crazy people”?

Companies clearly have to behave, but I don’t think companies can prevent individuals from getting the technology. An example of something we might have is some kind of secret patent. We may decide that it’s good to study some virus, in the hopes of coming up for a cure for something. But maybe we don’t want everyone to know everything about it. And yet we want the company to have intellectual property rights. So we need some kind of patent that isn’t published.

I’m not suggesting that’s the solution. We’d have to think that through. That could actually make the situation worse. I’m not sure how you’d have the patents be largely secret and still enforce them. It’s scary because it sounds like some of these trials where you don’t know what you’re accused of; it sounds like Kafka.

But see, the problem we have here is that if we’re trying to protect against something that’s so extreme, we can’t rule things out. If we want to have all this incredibly powerful technology, we may have to give up some other things. Fundamentally, individuals can’t be free to use anything that’s that powerful. Therefore, something has to be restricted: their actions, the information itself, the tools to do it.

Haven’t you spent most of your life pushing technology to its boundaries? Given your new apocalyptic views of nanotechnology, robotics and genetics, how has your own work been affected? Has your relationship to technology changed?

I’ve become more troubled about the progress in these fields because while there’s tens of thousands of good things we can do and enormously wonderful wealth creation, longer lives, all these things — I don’t think we’ve dealt with the downsides, and aren’t on a course to deal with the risks. So that’s obviously a depressing thing to believe. It’s kind of strange to write a piece, and have your fondest wish be that someone would stand up and tell you that you’re totally wrong. It’s not a message that’s a fun message to be delivering.

Did anyone come forward and allay your fears?

No. In terms of the analysis of the problem, there isn’t much dispute. I think some people say we can’t do anything, let’s just go faster — the fatalist approach. Other people say it’s all Darwinism, whatever happens happens. That’s really the same argument. The other thing people say is that we have to seek truth at all costs — that’s the Classical thing. But then, if you start looking at the ethical consequences, and try to say what you can do if you believe that something should be done, it isn’t easy. Because it involves people. Management would be easy if it didn’t involve people, right?

Then, do you still feel justified in your own work — on networks and Jini, which could be considered an aid to the super intelligent machines that you fear?

I’m actually not working on Jini or the network aspect of it right now. I’m trying to make machines more reliable. I’d like to have systems that worked, that were administered in a way that when you touched it, it didn’t break. Every time I touch anything I break it, because change has a higher probability of causing an error.

I don’t think [my work on reliability] contributes to the problem because the problem I’m concerned about is offensive uses of these things, and that doesn’t require reliability. It requires overwhelming force. Defense requires more care. For example, a burglar can go down the street and look for the house where the burglar alarm is broken; I have to have one that works if I expect my house to be safe. It’s the defense that needs the reliability.

So improvement in software reliability improves our ability to deal with these situations to the extent that we can abate them with technology. But in the end, I don’t think there are technical fixes to a lot of these problems anyway. We’re going to need to have some ethical and political and other fixes.

I’m interested in the kinds of systems that provide services to people who can’t tolerate failure — like air traffic control, hospitals and public safety. We need more and more systems that work safely and reliably in those environments, especially as medicine gets more computerized.

I don’t see that it’s contributing to this other problem. If I felt it was in any substantive way, then I’d really have a problem. But the rate of progress of the speed of personal computers is largely out of my hands. I sent copies of the article to Andy Grove and Gordon Moore [of Moore's law], but that’s all I can do.

Wired News ran an article saying Silicon Valley didn’t pay much attention to your warnings — but personally, how would you gauge the reaction? Did anyone from Sun respond?

I sent copies to all the vice presidents and directors and other people who have equivalent categories in Sun — that’s a few hundred people — and I would say I got 25 responses all of which were supportive. I suppose you could say that maybe some people didn’t like it and didn’t say anything because they were afraid. That’s not really the culture though; people tend to question everything. I have one old friend who works at Intel who sent me mail saying he thought it was all way, way farther out. Then I sent him back a couple of references trying to move him gently to where I think the truth is. Now I just got mail from him a half-hour ago saying his daughter was thinking about going to school in nanotechnology, and now he’s talking to her about it.

What about Hans Moravec’s argument that we, as humans, are just parts of this evolutionary whole and that the advent of techno-humans represents a higher form of evolution — one that we should be honored to bring about, even if they kill us off?

Nah. There’s two things going on there. One: Science has always been fatalistic, saying we’ll just pursue truth and whatever happens happens. Moravec is espousing, really, the Darwinian world. But humans haven’t been subject to Darwinism for a long time. In fact you can make a strong argument that evolution stopped when we came along because evolution is no longer genetic, it’s cultural. So he’s arguing for bringing back something that has not been the driving force and somehow transferring it from a side effect of capitalism, creating it without anybody asking for it, and then just sort of doing surgery on ourselves. It might happen because we’re not being careful, but it’s an awful crazy thing.

Aristotle said, “We’re clearly rational beings,” and we clearly are. We didn’t set up our present system without thinking. The Constitution, the whole economic system we have was designed. So I’m hopeful if only because I believe we can choose what’s in our interest independent of whether it maximizes someone’s notion of profit or destiny. We can choose to have a social safety net, when in fact business would be more efficient if it didn’t have to pay those costs, and people had to starve in the streets. We can decide that’s not acceptable and we have in most Western societies. That’s not a totally Darwinian choice. The total Darwinian choice would just be to let it go. But we’ve decided that’s not the kind of society we want to live in.

Do you think we’ll actually reach the point where these technologies are considered dangerous enough to be regulated? Will Congress ever address the issue?

I think we have to be realistic that it takes a long time … We’re a very long way from any prospect of any action. And we would have to deal with the scientific community’s enormous desire for lack of interference, and businesses’ enormous desire for a lack of interference, and governments’ desire to not do anything. Everybody’s pretty happy with the status quo at this point. It’s going to take some real leadership, and it’s going to take time to develop.

Let’s just hope no one has the breakthrough to wild self-replication. Luck may be an important aspect here. We might get lucky. Let’s hope we do.

Damien Cave is an associate editor at Rolling Stone and a contributing writer at Salon.

More Related Stories

Featured Slide Shows

  • Share on Twitter
  • Share on Facebook
  • 1 of 11
  • Close
  • Fullscreen
  • Thumbnails
    Martyna Blaszczyk/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 1

    Pond de l'Archeveche - hundreds thousands of padlocks locked to a bridge by random couples, as a symbol of their eternal love. After another iconic Pont des Arts bridge was cleared of the padlocks in 2010 (as a safety measure), people started to place their love symbols on this one. Today both of the bridges are full of love locks again.

    Anders Andersson/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 2

    A bird's view of tulip fields near Voorhout in the Netherlands, photographed with a drone in April 2015.

    Aashit Desai/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 3

    Angalamman Festival is celebrated every year in a small town called Kaveripattinam in Tamil Nadu. Devotees, numbering in tens of thousands, converge in this town the day after Maha Shivratri to worship the deity Angalamman, meaning 'The Guardian God'. During the festival some of the worshippers paint their faces that personifies Goddess Kali. Other indulge in the ritual of piercing iron rods throughout their cheeks.

    Allan Gichigi/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 4

    Kit Mikai is a natural rock formation about 40m high found in Western Kenya. She goes up the rocks regularly to meditate. Kit Mikai, Kenya

    Chris Ludlow/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 5

    On a weekend trip to buffalo from Toronto we made a pit stop at Niagara Falls on the Canadian side. I took this shot with my nexus 5 smartphone. I was randomly shooting the falls themselves from different viewpoints when I happened to get a pretty lucky and interesting shot of this lone seagull on patrol over the falls. I didn't even realize I had captured it in the shot until I went back through the photos a few days later

    Jassen T./National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 6

    Incredibly beautiful and extremely remote. Koehn Lake, Mojave Desert, California. Aerial Image.

    Howard Singleton/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 7

    Lucky timing! The oxpecker was originally sitting on hippo's head. I could see the hippo was going into a huge yawn (threat display?) and the oxpecker had to vacate it's perch. When I snapped the pic, the oxpecker appeared on the verge of being inhaled and was perfectly positioned between the massive gaping jaws of the hippo. The oxpecker also appears to be screeching in terror and back-pedaling to avoid being a snack!

    Abrar Mohsin/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 8

    The Yetis of Nepal - The Aghoris as they are called are marked by colorful body paint and clothes

    Madeline Crowley/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 9

    Taken from a zodiac raft on a painfully cold, rainy day

    Ian Bird/National Geographic Traveler Photo Contest

    National Geographic Traveler Photo Contest Entries

    Slide 10

    This wave is situated right near the CBD of Sydney. Some describe it as the most dangerous wave in Australia, due to it breaking on barnacle covered rocks only a few feet deep and only ten metres from the cliff face. If you fall off you could find yourself in a life and death situation. This photo was taken 300 feet directly above the wave from a helicopter, just as the surfer is pulling into the lip of the barrel.

  • Recent Slide Shows



Comment Preview

Your name will appear as username ( settings | log out )

You may use these HTML tags and attributes: <a href=""> <b> <em> <strong> <i> <blockquote>