File spoon-archives/sa-cyborgs.archive/sa-cyborgs_2000/sa-cyborgs.0004, message 3

Date: Fri, 28 Apr 2000 21:27:44 +0200 (MET DST)

Greetings technologists community,

[Hi, I might be interested in knowing, what Bill Joy, Hans
Moravec and Ray Kurweil have discussed in the Symposium at Stanford
University a few weeks back, on the theme..asking: "Will Spiritual
Machines Replace Humanity by 2100?" -Details are below. Bill Joy has
raised a serious issue of Ethics, in the midst of technological
advancements and progresses. BTW, Bill Joy has also recently written an
article in WIRED, on "Why the Future doesn't Need Us. Thank you. --Arun] 

Date: Wed, 19 Apr 2000 20:35:19 -1000
From: I. Thomas <>

                         By Michael Powell
                    Washington Post Staff Writer
                  Sunday, April 16, 2000; Page F01

"I may be working to create tools that will enable the construction of
the technology to replace our species," Bill Joy says. (Steve Castillo
                                            - for The Washington Post)

rising higgledy-piggledy up a hill toward an azure sky. A hundred
Chinese ladies raising silken knees in balletic tai chi. Morning's
cool shadow slanting across the park.

On a green bench a high priest of silicon sits and talks heresy.

Heresy about a golden Silicon Valley and the scientific Darwinism that
runs amok there. About the technocratic culture he helped birth, and
how it just might be the death of us.

"We are dealing now with technologies that are so transformatively
powerful that they threaten our species," he says. "Where do we stop,
by becoming robots or going extinct?"

His name is Bill Joy, a tall, 45-year-old man with the dreamy
intensity of a scientist. He is a co-founder of Sun Microsystems, one
of the world's most profitable companies. A widely respected computer
expert who one night sat in an Aspen diner and on a place mat designed
a software language that allows your refrigerator to talk to your car.

Now he watches as three fingers of doom arise from the world he helped
shape. He looks at modern genetics, robotics and the ability to build
molecule-size machines and sees a technology exploding beyond our
ability to control it. He sees manmade plagues and malevolent robots
overshadowing even the nuclear era in the capacity for utter

He warns that our scientists, his friends, must relinquish their
Promethean fire.

"I'm sorry but there are certain technologies so terrible that you
must say no." He leans forward on the bench. "We have to stop some
research. It's one strike and you're out."

The broader crusade he and a handful of influential scientists are
embarked on is no less imposing: to challenge a scientific culture
soaring too close to the sun on wings of wax. It's a culture where
pursuit of "truth" is all, where researching the next hot thing is
what matters.

"For Aristotle, an argument based on a poem was as valid as one based
on science. We've lost that. I don't sense in this community that an
ethical, spiritual-based argument carries nearly as much weight as a
capitalist imperative or the notion that progress is the ultimate.
That whatever happens happens.

"It's scientific fatalism and it could be fatal for us."

Apocalypse now? The mind manufactures caveats and yes buts and well,
whatevers. The stock bull runs the wired decade,
multiplies, the Cold War has passed . . . and Joy's talking death by

The temptation is to say: Cut the dark talk and pass the IPO.

But that would reckon without our cultural schizophrenia. Popular
culture celebrates its bigger, faster, 999,000-megabyte future even as
it devours stories of cyber-terror and robo-cops, dystopian dreams and
fears that chill.

American scientific culture pays less heed to its shadows. In the
stock-optioned ecosystem of Silicon Valley, where scientific
brilliance and venture capital leapfrog toward the horizon, the talk
is of satisfying our deepest hungers. Scientists would stamp out
disease, use computers to make us hear and see and feel better, maybe
even beat back death itself.

And they recoil when Joy, who was co-chairman of a presidential
commission on information technology, talks of barring some research.

"The evolution of technology just continues the . . . explosion of
biological evolution," says Ray Kurzweil, a cutting-edge scientist and
author of "The Age of Spiritual Machines." "In the 21st century we
will make 20,000 years of progress."

Perhaps. But maybe the four-century-long process that freed the modern
scientific mind from the bonds of religion, a philosophical string
that runs from Copernicus to Descartes, has had a curious effect.
Maybe our culture has made science so powerful that it's replaced God
at the center of our universe.

"The technological impulse to break free from the constraints of
nature and emancipate the human being is noble and a central archetype
of mankind," says Richard Tarnas, the philosopher and author of "The
Passion of the Western Mind." "The great danger of our time is that
the quest hasn't been matched by a moral and psychological awareness
of our limits."

Joy spoke at Stanford University a few weeks back, at a symposium
asking: "Will Spiritual Machines Replace Humanity by 2100?" Midway
through, Joy peered quizzically at scientist Hans Moravec, a husky man
who has established the world's largest robotics research program at
Carnegie Mellon. Moravec is convinced he can create a conscious robot
in 30 years, and is quite enthused by the prospect.

"Isn't there a point at which you worry?" asks Joy.

"Well," Moravec allows, "I do worry about intelligent robots getting
strange ideas . . ."


Joy traces the kernel of his discontent deep into the technological
cocoon, of which he is a creature. He recalled talking with Kurzweil
about computers in the autumn of 1998. It is theory and faith among
denizens of the digital world that computer chip speed doubles every
18 months.

Joy always believed this rate would slow down as science rubs against
the boundary of the physically possible. And he drew comfort from
knowing there's a limit to the power of computers.

Now Kurzweil told him that molecular-level advances would allow for
computers a million times faster and smarter by 2030. So a world of
possibilities revealed itself, and cast Joy into darkness. He's
married, has two children and a beautiful home in Aspen; he saw the
clouds moving in.

He wrote of his epiphany in a much-noticed essay for Wired magazine:

"It was only [then] that I became anxiously aware of how great are the
dangers facing us in the 21st century. . . . We have yet to come to
terms with the fact that the most compelling 21st century
technologies--robotics, genetic engineering and nanotechnology"--pose
a more dangerous threat than any past technologies.

These computers and genes and micro machines, Joy writes, "share a
dangerous amplifying factor: They can self replicate: A bomb is blown
up only once, but one bot can become many, and quickly get out of

Genetic research is the most advanced. Already scientists splice the
fish cell with the strawberry, the salmonella bacterium with the sperm
cell. (What is not known is how these new cells and bacteria will
behave once released outside the lab; the only certainty is that cells
and bacteria replicate and mutate.)

A bit further into the future, scientists hope to build
sub-microscopic nanotech machines so small they can fly by the
billions into the atmosphere or cruise the bloodstream like galleons.
Scientists envision computer robots so smart that, decades or
centuries into the future, a metallic eye might blink and a silicon
mind awaken.

The research fueling this micro industrial age is hurtling forward. It
is not taking place in the crucibles of the nuclear age, the
government-funded labs where scientists held information confidential
on pain of imprisonment. That's the dinosaur age.

The market rules now. The venture boys erect a thousand elegantly
outfitted labs, and the drive for riches and pure research attracts
scientists of the highest caliber. Information is swapped on the Web
and the ethos is libertarian and triumphal: Leave us alone and we will
transform your world.

To its devotees, this is the technological great leap forward. Joy
divines a darker shadow.

"Silicon Valley is like the Burgess Shale of the Darwinian
experiment," he says. "Every scientist has become a new phylum and
everyone is running around thinking they'll be the biggest mammal.
They all want to be billionaires and no one wants to think about
implications of their research."

Joy acknowledges that new technologies might cure diseases and
increase crop yields, and bring great surcease from suffering. But at
the crossroads of knowledge and the individual genius, the threat to
humanity metastasizes.

Ruin might come from a laboratory experiment gone awry. Or a college
student reading a Web recipe and mixing test tube viruses. Or the
genius psychopath, shooing a boutique virus or self-replicating robots
out the door.

It is what Eric Drexler, the father of nanotechnology, calls the gray
goo problem: that someone might, inadvertently or otherwise,
manufacture a ravenous and invisible man-made machine or bacterium
that would outstrip natural competitors and, propelled by wind and
rain, turn life to dust in a matter of days.

"There is a tremendous, market-driven haste to get into invisible
technologies that are unstable and dangerous to life," says Tom
Valovic, a research manager for International Data Corp. and author of
"Digital Mythologies." "It requires this religious leap of faith that
science knows best."

Joy has renounced work on nanotechnology, seeing it as too perilous.
But he acknowledges that software he invented now enables one computer
to talk with another, and so might help another scientist in another
lab create a menace.

"I may be working to create tools that will enable the construction of
the technology to replace our species. How do I feel about this? Very


Men sometimes speak as though the progress of science must necessarily
be a boon to mankind, but that, I fear, is one of the comfortable
nineteenth-century delusions which our more disillusioned age must

--Bertrand Russell in 1924

Many scientists respect Joy, even as they dismiss his fears as
primitivist. From Faust to Mary Shelly to Einstein, they say, man has
recoiled from the shadow of the new. And as often--refrigeration,
telephones and penicillin come to mind--our technological and chemical
handiwork makes life immeasurably healthier and easier.

It was once said labor-saving machines would cause mass unemployment,
that recombinant DNA would be the end of us. Haven't we heard a
thousand siren calls of doom?

"The fact is that he is right; this could happen," says John Perry
Barlow, founder of the Electronic Frontier Foundation. "By the same
token an asteroid could hit the planet Earth and have an even worse

To which Freeman Dyson, the Princeton University physicist, adds:
"There is a hidden cost of saying no. And that's the advances in
health and food production that we will forgo."

Against such optimism lies the weight of homo sapiens history. So many
technologies, from chariots to flying machines, become killing tools
in our hands. And the bacteriological world spawns as many surprises:
the Spanish flu, Ebola, AIDS. The grandfather of these 20th century
disasters was the medieval plague.

Arriving in so many guises that 14th Century doctors termed it the
"strange chameleon," the plague ravaged Europe, killing one-quarter of
the population and causing bailiffs in London and Venice to nail shut
houses of the dead and dying. Today, scientists can identify but a few
plague strains. Many more strains have disappeared, or lie dormant.

We tend, also, to underrate our scientific ability, and our capacity
for mayhem. Scientists in the 1930s laughed at the suggestion that
they might soon explode "dangerous quantities of available subatomic
energies." The nuclear bomb, a renowned physicist argued in 1930, was
"a hobgoblin."

Fifteen years later, Edward Teller oversaw the construction of such a
bomb. Warned that it might ignite the Earth's atmosphere, he endorsed
an explosion anyway. And his colleague, Robert Oppenheimer, stared at
the nuclear dawn and quoted the Hindu God Vishnu: "I am become Death."

The point is not that technology is evil. It is amoral. Human nature
is the wild card.

Joy has been talking for hours this morning. The white around his blue
eyes is burnt red from lack of sleep. We leave a restaurant and walk
across the street in San Francisco. A brilliant western sun beats
down, a dog barks, an old man sips an espresso under a palm tree.

Joy talks of the golden valley of silicon an hour's drive to the
south. He discerns a thirst for scientific knowledge--and the money to
underwrite it--that frankly scares him.

"In the endless pursuit of truth, we're about to bring back the notion
of fate. By empowering everyone to hold the powers of destruction,
everyone controls our fate as a species."

His eyes catch yours. "That's not a pleasant thought, is it?"


"We need to distinguish between artificial intelligence and
downloading your brain into a computer, which is controversial," says
Ray Kurzweil, on the coming age of spiritual robots.

We're in a lecture hall at Stanford University, jammed with scientists
and aficionados of technology, listening to a panel of geneticists and
experts on robotics and nanotechnology discuss how and when man might
become silicon. Bill Joy is on the panel, too, in his new role as
house skeptic.

To listen to these scientists, to swim in their virtual realities, is
to understand how profoundly Joy's message disquiets them. This is a
world where evolution equals technology equals progress. Where
scientists talk about what technology "wants" and "needs" and

The passing of our carbon-based bodies is seen as inevitable, like a
butterfly shedding its chrysalis. It's no insult to call this science
fiction; science and fiction are tectonic plates that long ago
converged in Silicon Valley.

"We're in for a 100-year identity crisis as human beings, and we're
launching right now," says Kevin Kelly, an influential writer and
member of the high-tech digerati. "We should figure out what
technology wants and train it to be a good citizen."

A few of the scientists see intelligent robotic life as centuries
away. Others discern it around the bend of the next decade. Some
believe that microscopic nanotech centurions will guard our
bloodstreams against man-made viruses. Others argue that we may have
to manufacture pathogens to combat other manufactured pathogens.

Uploading human brains into computers, blending the biological with
the silicon . . . the point of agreement is that swift, irrevocable
change is inevitable. Technology will be our savior.

What's striking is the bedrock faith in the church of science as the
ultimate truth.

"Progress and technology are inevitable," Kurzweil says. "We are
talking about the next evolutionary step. In virtual reality, we will
have bodies and riches. And we will take a walk on a virtual beach in
a virtual Cancun."

Hans Moravec, the computer scientist with the blue T-shirt and the
blue work shirt with the collar pulled up clerical style, is the least
sentimental of the panel. In his view, sensory pleasure, ethics,
morality, all of these are evolutionary outcroppings that could shear
away as we evolve.

"One way to evade the biological threat is just to become
non-biological." Moravec plays with a smile that suggests he's aware
that the homo sapiens in the audience might find his proposal

"The evolution of our descendants will push them into entirely
different realms." He's still smiling. "They will become something
else entirely. I don't know why you are disturbed by that."

On and on the panelists talk, a wizard's brew of possibilities. Joy
pulls at his upper lip and listens with the detachment of an apostle
strayed. Their assumptions about the world and technology are no
longer his.

"To use the word 'want' about technology, to talk about technology as
our 'children'--" Joy shakes his head, disbelieving. "We are using an
incredibly misleading vocabulary to try to get around vast ethical

And he's not comforted by the notion of uploading the human brain into
the computer maw. Hit the delete button on that one.

"Robot existence will never be a human existence," Joy says. "It is
not three-dimensional. We're not a native form in cyberspace. If we
allow evolution to go on in there, what results will not resemble us."

Roam the world of Web thinkers and scientists, in California and
elsewhere, and it's not hard to find people who share Joy's fears. But
the prevailing ethos, that scientific research cannot be constrained
by man, is so strong that few even talk of reining it in.

Christine Peterson of the Foresight Institute, a California
new-technology think tank, counsels fear followed by acceptance.

"Bill's correct to be nervous and seriously scared," Peterson says.
"But research is not going to stop, that's not an option. If you try
to stop people, they'll go underground."

Peterson would ask research labs to agree to outside surveillance.
Other Foresight board members are working on legal and ethical
protocols for nanotech research.

As for an age of intelligent computers?

Peterson suggests we might want to be careful. It will be hard to know
what annoys them. (When you thumb your nose at a conscious computer,
will it know it's being dissed?) We'll want to keep the computers'
competitive impulses in check.

"We want a system where computers have no need to wipe us out. We want
to make sure that physical property rights [translation: Our right to
breathe and live] is protected."

She likens the human race to the Amish. Our legal and ethical system
ensures that the Amish can live happily insulated from a technological

So one day, by implication . . . ?

"Right. When computers rule, we're going to be the Amish."


"A Joyless Future" . . . "No Joy Here" . . . "Silicon Valley Killjoy"

So the headlines read in the days after Joy's essay appeared, the
heretic nailing his manifesto to the door of the high-tech temple.
National and international media interviews, scientists reacting,
columns mixing praise with dismissive shrugs.

He e-mailed copies to lawmakers, to the Supreme Court, the president
and his Cabinet secretaries. He's yet to hear back from any politician
of note. The Republicans and Democrats are clamoring to claim credit
for the technological dawn. Who wants to talk about the night?

Nor do many of his friends in Silicon Valley want to hear these
arguments. In a way, it goes to why he decamped several years ago,
left the Valley and its hungry young men and women sleeping on lab
floors. He left for the vertical beauty of Aspen. Different time,
different culture.

"The Valley culture has kind of gotten away from me," Joy says.
"They're in this Darwinian mind-set. They can't imagine not doing it."

So he hopes to take the conversation outside the church door, to
like-minded scientists and activists. Several high-tech thinkers, from
Valovic at Intel to physicist Amory Lovins to John Seely Brown,
director of the Xerox Palo Alto Research Center, now argue that our
high-tech culture suffers from technological tunnel vision, without
concern for protocols and the social good.

And the Clinton administration has taken halting steps, such as
declaring emerging diseases a national security threat.

Joy would go much further. Prohibit research on dangerous technologies
and put stringent international controls on sensitive equipment, much
as nations do with plutonium and biological war agents. Create an
international whistleblowers' fund and pay vast sums to anyone who
exposes life-threatening research work.

And unleash the lawyers. Demand that private labs take out insurance
against terrible accidents, and let the raptor tort types have at
them. Death by lawsuit, a strategy immediately recognizable to any
anti-tobacco activist.

"We need to reset the social contract," he says. "Just as we did in
response to the technologies of the first industrial age."

In the end, perhaps, it comes back to faith. A former colleague or two
has suggested that Joy has gone round the bend, become a primitive
spiritualist in a rationalist world. But talk to the digerati long
enough and you hear a sometimes unspoken belief in a "hidden hand"
that shields us from the most terrifying use of our inventions.

We've had the bomb for half a century, the argument goes, and we
haven't destroyed ourselves. AIDS rose up along with the technology
emerged to keep it at bay. We have survived so far, therefore . . . .

"There is a co-evolution between society and technology," argues
Barlow of the Electronic Frontier Society. What keeps us in balance?
"At a certain stage you just have to go theological," he says. ". . .
I do believe that there are larger forces than ourselves at work."

Joy comes to quite the opposite conclusion. He is no atheist, he left
that certainty behind in college. But the world, he believes, is in
our hands. It is the price of our maturity as a species.

He walks through the park toward his car. He stops and watches the
Chinese ladies do a slow tai chi dip and rise. There's not a cloud in
the sky.

"If you don't believe extinction can happen," he says, "check the
fossil record."
                  c 2000 The Washington Post Company
_ _ _ _ _ _ _ _ _ _
Tom Atlee  *  The Co-Intelligence Institute  *  Eugene, OR


Driftline Main Page


Display software: ArchTracker © Malgosia Askanas, 2000-2005