Interview with Bill Joy, Sun Microsystems Inc.

September 6, 2000

John McIntyre

JRMc: Is 2001 very different from 1945 in terms of being at a technology crossroads? In 1945 there was the atomic bomb, in 2001 the GNR problem that you mentioned in your Wired article?


BJ: Well, the world is at peace I suppose.If we were in a global conflict with the kind of fear that we saw around Hitler, propelling people to pursue research for weapons purposes, I think that we would be in worse shape.In that sense, the driving forces last time were military, this time they are commercial.


JRMc:†† Do you think that there are qualitative as well as quantitative differences- the kind of problems that arise in connection with technology are not as clear cut as they were back in 1945?


BJ: No.I think these problems are very clear-cut and are understood by todayís scientists who are creating the technologies.The nanotechnologists, the robotic scientists and certainly a number of biologists recognize both the safety and ethical issues with their work.There has been some debate about genetically modified foods but in a sense it has been focused on the least of the dangers.The human safety testing is not nearly as difficult as ecological safety or the cascading unintended consequences that are possible.


JRMc: You have stressed three particular families of technology - of scientific revolutions for lack of a better word - genetics, nanotechnology and robotics as harbingers of a threat to manís survival.Why have you focused on these particular three classes of technologies?


BJ: The power of these three technologies all comes from information technology.The genetic kind of revolution comes from biology becoming more of an information science.You can do things with robots and computers rather than beakers and microscopes.And we saw this in the human geno project.Nanotechnology is about being able to design things at the atomic scale, using the whole periodic table and not just organic materials.Robotics is about building machines, perhaps with some biological aspects, which either use traditional computing or perhaps computationally embedded evolutionary models.So, all three are about information.Since they can be designed on a personal computer, the kind of information that is involved can be spread around using things like the Internet.Because computers can be used to simulate these designs there is such large danger. In the case of biological weapons, if you have something as dangerous as Ebola Ė if you donít have access to the Ebola material its hard to do much with it.Or Smallpox, if you donít have access to whatís conceivably now a very small quantity of Smallpox in the world, you canít abuse it.But information flows much more freely- so the danger really comes not because large scale mistakes arenít possible, but today they require large organizations, certain nation states, or access to material. Once the technology involved is fundamentally an information technology, itís much more difficult to control and the scale of the activity can be much smaller because you donít need a large laboratory.


JRMc: In a way, what distinguishes 2001 as an arbitrary point from 1945 is the issue of diffusion and scale of the danger?


BJ:Itís several things coming together.There is the dematerialization of the information you need to do harm.It is becoming weightless- itís just bits.The other thing is that these technologies also can be used to develop things that replicate.†† With a nuclear weapon, the chain reaction must stop.†† It has a certain amount of destruction.Obviously it can also spread fallout but the total amount of destruction is bounded, very large but bounded.If you release something into the environment that can reproduce, there is almost no limit to how much damage a single action can do.For example, if you created a version of malaria that wasnít limited to the tropics in some way, that would be a very bad thing to do.Bringing the West Nile virus to New York was a very bad thing to do.†† It was an accident probably, but it can have much larger consequences over a period of time than even dropping a small nuclear device somewhere because the effects of the nuclear reaction eventually would be limited.Obviously a nuclear weapon is much more destructive than the West Nile virus but those are just examples.You can imagine a more destructive, more communicable disease like the 1918 flu with its extreme severity and transmissibility.


JRMc: In the area of intelligent robots, what are the threats and distinct possibilities of abuses that concern you?


BJ:The threats from robotics donít require intelligence.A robot does not have to be intelligent to be destructive.People could use real robots in war- and make war much more destructive, the Armyís version of the Air Forceís Drone aircraft.The real troubling prospect and the one that is a number of years away is that we might create a species and set it lose on the world to evolve on its own.The species is essentially robots not built out of proteins, not built out of organic materials.The problem is that itís not necessarily the case that there is any resistance.In other words if you look at the biosphere, we might not have any resistance to such machines because it hasnít seen them before and there is no direct immunity to fight back.That is potentially very dangerous and a number of people have proposed that the guidelines prohibit artificial life experiments on nonbiological materials because of this danger. Itís fear of an ice 9 or grey goo kind of accident.I think thatís very responsible.What would happen if we put in different amino acids in biology?The number of amino acids that are used in DNA is a small number and there are other ones that arenít used.What would happen if we used those? Has evolution tried this?So when we start experimenting at the substrate level with technology that can replicate, we have to be exceedingly cautious and double/triple check everything that we do.


JRMc:What about the argument that from an ecological and environmental point of view that robots are the answer to manís problems in terms of consumption of energy sources and so on?Is that a valid argument?


BJ: Well, as long as you donít make a species.As long as they are not autonomous, self-replicating and evolving, you are probably safe.The problem is that the autonomy is really useful so the robot can act on its own.And letting them evolve is a good way to let them be smarter, basically use evolution to have them learn behavior.So that leaves only self- replicating and we already saw in the newspaper just in the last week an example of a robot that could make more of itself in a certain environment.Self-replication is also extremely tempting when you do nanotechnology.If you want to assemble materials out of atoms basically by tick and place itís a very slow process and you need an enormous number of assembles.The easiest way to make an enormous number of assemblers is to have the assemblers assemble assemblers.The dangerous stuff is very tempting and thatís the problem.


JRMc: Iíve read some reactions to your piece in April by nanotecnology scientists who said that you overestimated the danger in nanotechnology.They stated that technology was nowhere near this point nor did they ever anticipate reaching that point.


BJ:The truth is that they have dreamed of these assemblers for many years.†† Eric Drexler wrote of these dangers and of this dream in his book Engines of Creation which I believe was published in 1986.So I just donít think thatís true.I think that the combination of this being an information science and the Internet and the difficulty of controlling information perhaps wasnít fully appreciated.But Eric in fact argued that there is enormous danger and that means that we out to go faster because we want to get there before our enemies do.That does not deny the danger.


JRMc: Enemies in the commercial or military sense?And what would be the threat from the US standpoint?


BJ: Military.The Iraqis might get the atomic bomb or we think that the North Koreans are going to launch all of these missiles at us.I donít think that thatís a rational fear from anything that I know nor is it rational to think that we can deflect them with a missile shield.Does the political process make it profitable to take these points of view?The Star Wars shield is a dream.†† People want it to be true despite evidence to the contrary.The same is true here. A lot of these technologies are so powerful that if they fall into bad hands there is enormous danger and there is unlikely to be any defense.Defense has to be perfect.If offense gets through even a little bit or sneaks through, you have a big problem.


JRMc: Do you agree with this particular stream of reasoning?


BJ: It would be easier to build a bad pathogen that attacked some part of the biosphere than to defend the biosphere against all such attacks.I donít know how you can defend against something when you donít know what it is that you are defending.The problem is that in the biosphere we donít understand all of the mechanisms.In order to defend against an unknown threat we would have to determine whether a certain mechanism was proper or improper.Without a complete model of the biosphere, we could never know.You just end up creating an autoimmunity.If you build a defense system trying to defend against these unknown threats, you might identify part of the biosphere as an invader and thereby have an immune reaction without an infection.And in some sense the defensive shield would probably be in grave danger of attacking the thing that itís trying to defend.


JRMc: You have proposed as an alternative to limit the development of technologies that are too dangerous.And this has been characterized as too drastic and too conservative.


BJ: If we agree that this technology is too dangerous how can there be any other conclusion?If we had said just dangerous, thatís one thing.If we all agree that this technology is too dangerous, we would be better off not doing it.For example, the US made this conclusion basically 30 years ago in the Nixon administration with biological weapons.Having them was worse than not having them.Even if our enemies had them, we donít want to make them better if we canít defend against them.If we make them, they would be likely to fall in bad hands.It makes more sense to not have them.How many huge weapons of mass destruction do you need anyway?

JRMc: Do you think that 19th Century man would have gone ahead with nuclear engineering?Nuclear engineering technologies probably would have been defined as too dangerous in turn of the century terms.


BJ:Iím not sure.I think that itís possible to make nuclear reactors which are safe such that they basically shut themselves down.I know that some of the more advanced designs in France are designed this way.†† Unfortunately, the Russian designs are not.The difficulty with nuclear energy is basically the life cycle management of the waste.What we discovered here is that energy conservation is cheaper.You have energy production to meet demand but you can also reduce the demand by intelligent engineering.There is evidence that 90% of the energy is wasted.I donít think that itís clear that peaceful uses of nuclear energy are inherently bad.We had a proposal in the 1945-1947 timeframe to internationalize the non-peaceful uses of atomic energy.Had we done that I think we would not have had the arms race and I think that we could still have peaceful uses.Itís probably the case that there would be some risk of somebody cheating but the risk of a bad war is arguably greater.There was an article in the newspapers today that the Russian early detection satellites are in bad shape and greatly increasing the risk of an accidental war.†† Apparently, the Russians cannot tell whether we have launched at them or not.So the nuclear threat isnít over, but I think people of goodwill could have separated these two ends.We are just fortunate that we didnít have an accident.Iíd like to understand better why we didnít.Of course, if we had had a really bad accident we would not be here to talk about it.


JRMc: You anticipated my next question which relates to comparing relinquishment to export controls.Donít we need certainty that everyone else is moving along in terms of preventing diffusion or relinquishing research and development?


BJ: It is extremely hard because itís commercial technology.But the truth is that if we assume that this technology is so powerful that any individual who has access to it could create almost unbounded harm.If the information is on the Internet or it is stolen and put on the Internet, we donít know how to ever recall it.We canít assume that the crazy people arenít out there.So what do we do?I think that we have to do a lot of things.There is going to be no elimination of all risk but we have to apply something more like the precautionary principle. There hasnít been an accident yet but that doesnít mean that we cannot anticipate the nature of an accident and try to reduce the risk.


JRMc: Those precautionary principles, are they an ethical guideline?


BJ: Yes.Itís largely used in Europe.Itís a principle used in managing risk largely in ecological situations where you look at what the potential harm is.You donít have to have the proof of harm.You can use reason to say that it is clear that there is a lot of danger so letís not do this.On the other side, people say that it has not happened yet.What about situations where by the time itís happened, itís too late?The Physicians for Social Responsibilityís motto is we have to prevent what we canít cure. In the case of viral diseases, once you are infected with a viral disease, thatís basically it.We donít today have the technology to cure viral diseases.†† Your body can fight them off.We have vaccines but they are only effective if they occur up front.The precautionary principle is the same idea.Letís prevent it instead.


JRMc: I wanted to raise the ancient Greek question: who should guard the guardians?With those guidelines and principles, who is going to apply them and decide where to draw the line?

BJ: Well, you know that we have this funny situation in this country where at least half of the people have an enormous distrust of the government.They have a distrust of collective institutions and I would have to say that it appears that the collective institutions have worked so well that no one remembers what they are about anymore.The collective institutions have brought down the threat of communism and have caused a very profitable global economic system to be created.We have relative peace in the world despite lots of regional wars.People have forgotten that government is about giving us the mechanisms to have a society of laws and collectively expressing our will.The collective scientific institutions, the OECD, the G7, the GATT, the UN, NATO- all of these different organizations can be used to come up with things that we all collectively want.We need mechanisms for getting positive outcomes.Americans in particular, who are the leading change agents for causing these technologies, seem to have the most aversion to these kinds of international institutions. They are essential if we are going to have any ability to reduce the danger with these technologies.


JRMc:Is the nature of capitalism and innovation flawed when it comes to highly dangerous technologies?


BJ: I donít think that this is the system you would chose if your first order of business is to manage these technologies.You probably would not use the free market.One way to look at it is that capitalism is about discovering and creating a whole lot of wealth at some speed.There are other systems that would probably do it eventually but they would do it at a different speed.When you are proceeding into unchartered waters, the highest speed isnít necessarily the best.Capitalism doesnít really have many breaks.†† Itís quite clear that itís dangerous to go into unchartered territory at high speed.No reasonable person that I know would deny that there is enormous danger.

JRMc: Some folks might even say that capitalism is the handmaiden of technology.Even though itís not the right system to regulate, itís the right system to give it birth or to spawn it?


BJ: Capitalism is an aspect of the Darwinian worldview.A lot of these sciences have a biological nature, so we use this evolutionary model.The evolutionary model is not about aiming at outcomes.†† Itís about setting a process in motion whose outcome is relatively unpredictable.Evolution produced a creature that has a thumb, a large forebrain, speech, and language.There is an evolutionary experiment going on to see if thatís a good idea or not.And if we extinct ourselves, it will be judged as though it was a bad idea.We sit here as the moderns with our science and all of our toys, not realizing that sometimes this danger is just in discovering the laws of physics.The physicists werenít sure when they lit the bomb whether it wouldnít burn the atmosphere up.The precautionary principle doesnít say that you shouldnít do anything if there are risks but that you should use the scientific method to look at the risk in a reasonable way.


JRMc: Comparing your outlook to Kurzweilís The Age of Spiritual Machines, are you more techno pessimistic?

BJ:No I think thatís more the style in which Ray wrote the book.If you look in there you will see the part where he expresses his concern and he says that we have a better than even chance of making it through.He thinks the chance of catastrophe is 50/50.Iím not pessimistic.If I were pessimistic, I wouldnít bother.I believe as Aristotle said, we differ from the beasts in that we have reason.We donít have to decide that we will put everything in the fate of a kind of evolutionary process.We can decide that we will use our reason to choose what outcome we want.That will require international cooperation and collective behavior.It will not be perfect and there will still be some risks but itís got to be better than just putting on blinders and accelerating into the unknown.


JRMc:Some folks have accused you and I would say unfairly but its worth a response on your part of being a killjoy.Whatís you reaction to such off hand comments?


BJ:Iím not advocating that we abandon all technology.†† Iím saying is that we canít afford to put unlimited power in the hands of everybody because there are crazy people and there is too great a possibility of accidents.Either we deny that we are going to be able to create this unbounded power or we have to figure out how to control it.It seems clear that we are creating the power so letís get on with doing what obviously needs to be done.


JRMc: Quick question on the Internet.The US of course remains a leader in the Internet.This is an area in which you were one of the founding fathers.Whatís your general view?


BJ: Well, we call it the six webs theory.The kind of Internet that we have now largely goes to desktop computers (Macs and PCs).The second web we saw is the wireless web and I think that Japan is largely ahead right now with the I-Mode phones and Europe is not far behind with the wonderful GSM system.Japan has a disadvantage in that even if they are technically advanced, there is a language/character set type of barrier.Europe has a disadvantage since there are so many languages in Europe that you have to make multi-lingual content.From a technical/infrastructure standpoint, the US wireless industry is fragmented.Europe or Japan may end up being a sustainable leader in wireless.The third web, the entertainment web, depends on high-speed connections to the home.Japan may be the leader there.†† They are talking about bringing fiber to every home.There is a lot of entrepreneurial activity in the US but its more difficult here because of geography.Itís probably difficult in Europe because of the age and the nature of the cities.Imagine pooling fiber in all of the city cores.The fourth web is based on voice recognition.The leading company in voice is Lernout & Hauspie which is a European company.Thatís really an emerging market.Finally, there are two other kinds of webs.One is ebusiness and I think that the US is probably ahead in using the ebusiness web.And then the last one is what I would call the pervasive computing web which is putting intelligence in devices.Appliance manufacturers like Siemens, Philips and Sony tend to be the leaders because the US tends not to have a presence in consumer electronics.So I think what you really need to do is break it out by the fact that there are at least these six ways in which the devices are being hooked together.The US is probably in the driver seat with respect to ebusiness and traditional web to a desktop.But entertainment, mobile, voice actuated and what I call pervasive and software internetworking in appliances are kind of up for grabs.Europe and Japan have some strong advantages especially in appliance webs and wireless webs.


JMRc: Given the caveats you have introduced regarding GNR, in what areas would you like to see more R&D?


BJ:Iím working on reliable systems and what that means is systems that work well even though the software has flaws and the hardware has flaws.People make mistakes and people do individual things which collectively look like mistakes.I think that we have very few systems today that are very reliable.Avionics and airplanes is one of the few.What I would like to see is the avionics quality for all other devices that use software.Thatís a big task but I think itís a worthy goal and itís what Iím working on.That will help us because any systems that we build to defend against some of these technologies will require reliable computers.If you want to have a burglar alarm in your house, it better have a battery back up and it better be pretty reliable.Otherwise when it breaks you are defenseless.The reliable systems are a defensive tool.Iíd like to see more work on reliable computing systems.


JRMc: In terms of software technologies, which areas do you think are up and coming in the next 5 years, 10 years?


BJ: The ones relating to the six webs.We will be able to use desktop web access devices, new kinds of pocket devices, new kinds of entertainment devices that hook to a high speed network, devices that are embedded in your clothing, in your car and in your door.Everywhere you can talk to objects.†† There will be more use of the ebusiness web and networking appliances to reduce the cost of ownership and make it more pleasant.Your washing machine will call the service man.






†††††††††† †††††††††††††††††††††††††††††††††††††††††††††††††