Home » Uncategorized » Nick Bostrom’s Superintelligence and the Metaphorical A.I. Time Bomb

Nick Bostrom’s Superintelligence and the Metaphorical A.I. Time Bomb

Superintelligence book coverFrank Knight was an idiosyncratic economist who formalized a distinction between risk and uncertainty in his 1921 book, Risk, Uncertainty, and Profit. As Knight saw it, an ever-changing world brings new opportunities, but also means we have imperfect knowledge of future events. According to Knight, risk applies to situations where we do not know the outcome of a given situation, but can accurately measure the odds. Uncertainty, on the other hand, applies to situations where we cannot know all the information we need in order to set accurate odds in the first place.

“There is a fundamental distinction between the reward for taking a known risk and that for assuming a risk whose value itself is not known,” Knight wrote. A known risk is “easily converted into an effective certainty,” while “true uncertainty,” as Knight called it, is “not susceptible to measurement.”

Sometimes, due to uncertainty, we react too little or too late, but sometimes we overreact. This was perhaps the case of the Millennium Bug (Millennium time bomb) or the 2009 swine flu, a pandemic that never was. Are we perhaps so afraid of epidemics, a legacy from a not so distant past, that we sometimes overreact? Metaphorical ‘time bombs’ don’t explode. 
This follows from the opinion that time bombs are all based on false ceteris paribus assumptions.

Artificial Intelligence may be one of the areas where we overreact. A new book by Oxford Martin’s Nick Bostrom, SuperIntelligence, Paths, Dangers, Strategies, on Artificial intelligence as an existential risk has been in the headlines since Elon Musk, the high-profile CEO of electric car maker Tesla Motors and CEO and co-founder of SpaceX, said in an interview at an MIT symposium that AI is nothing short of a threat to humanity. “With artificial intelligence, we are summoning the demon.” This was on top of an earlier tweet by Musk where he said he had been reading SuperIntelligence and A.I. is “possibly a bigger threat than nukes.” Note: Elon Musk was one of the people that Nick Bostrom thanks in the introduction to his book as a ‘contributor through discussion.’

Perhaps Elon was thinking of Blake’s The Book of Urizen when he described A.I. as ‘summoning the demon’:

Lo, a shadow of horror is risen, In Eternity!  Unknown, unprolific!. Self-closd, all-repelling: what Demon.  Hath form’d this abominable void.  This soul-shudd’ring vacuum? — Some said: “It is Artificial Intelligence (Urizen),” But unknown, abstracted: Brooding secret, the dark power hid.

Professor Stephen Hawking and Stuart Russell (Russell is the co-author along with Peter Norvig of the seminal book on A.I.) have also expressed their reservations about the risks of A.I. indicating its invention “might” be our last “unless we learn how to avoid the risks.”

Hawking and his co-authors were also keen to point out the “incalculable benefits.” of A.I.

The potential benefits are huge; everything that civilisation has to offer is a product of human intelligence; we cannot predict what we might achieve when this intelligence is magnified by the tools that A.I. may provide, but the eradication of war, disease, and poverty would be high on anyone’s list. Success in creating AI would be the biggest event in human history.

In 1951, Alan Turing spoke of machines outstripping humans intellectually:

“Once the machine thinking method has started, it would not take long to outstrip our feeble powers. … At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s Erewhon.”

Leading A.I. Researcher Yann le Cunn commenting on Elon Musk’s recent claim that “AI could be our biggest existential threat,” wrote:

Regarding Elon’s comment: AI is a potentially powerful technology. Every powerful technology can be used for good things (like curing disease, improving road safety, discovering new drugs and treatments, connecting people….) and for bad things (like killing people or spying on them). Like any powerful technology, it must be handled with care. There should be laws and treaties to prevent its misuse. But the dangers of AI robots taking over the world and killing us all is both extremely unlikely and very far in the future.

So what is SuperIntelligence?

Stuart Russell and Peter Norvig in their much cited book, Artificial Intelligence: A Modern Approach consider A.I. to address thought processes and reasoning, as well as behavior, they then subdivide their definition of A.I. into four categories: ‘thinking humanly,’ ‘acting humanly,’ ‘thinking rationally’ and ‘acting rationally.’

In Superintelligence Nick Bostrom says it is:

“Any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.”

Bostrom has taken this further and has previously defined superintelligence as follows:

“By a ‘superintelligence’ we mean an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills.

He also indicates that a “human-level artificial intelligence would probably have learning, uncertainty, and concept formation as central features.”

What will this Superintelligence do according to Bostrom?

For a good review of Superintelligence see Ethical Guidelines for A Superintelligence by Ernest Davis (Pdf link above) who writes of Bostrom’s thesis:

“The AI will attain a level of intelligence immensely greater than human. There is then a serious danger that the AI will achieve total dominance of earthly society, and bring about nightmarish, apocalyptic changes in human life. Bostrom describes various horrible scenarios and the paths that would lead to them in grisly detail. He expects that the AI might well then turn to large scale interstellar travel and colonize the galaxy and beyond. He argues, therefore, that ensuring that this does not happen must be a top priority for mankind.”

The Bill Joy Effect

Bill Joy wrote a widely quoted article in Wired magazine in April 2000, with the fear filing title: Why the future doesn’t need us, where he warned:

“If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines.”

Eminent researchers John Seely Brown and Paul Duguid, offered a strong argument to Joy’s pessimistic piece in their paper, A Response to Bill Joy and the Doom-and- Gloom Technofuturists, where they compared the concern’s over A.I. and other technologies to the nuclear weapons crisis and the strong societal controls that were put in place to ‘control’ the risks of nuclear weapons. One of their arguments was that society at large has such a significant vested interest in existential risks it works to mitigate them.

Seely Brown and Duguid indicated that too often people have: “technological tunnel vision, they have trouble bringing other forces into view.” This may be a case in point with Bostrom’s Superintelligence, where people who have worked closely with him, have indicated that there are ‘probably’ only 5 “computer scientists in the world currently working on how to programme the super-smart machines of the not-too-distant future to make sure A.I. remains friendly.” In his book presentation Authors@Google Bostrom claimed that only half a dozen scientists are working full time on the control problem worldwide (last 6 minutes). Which sounds like “technological tunnel vision,” and someone who has “trouble bringing other forces into view.”

Tunnel vision A.I. Bias

Nicholas Taleb warns us to beware of confirmation bias. We focus on the seen and the easy to imagine and use them to confirm our theories while ignoring the unseen. If we had a big blacked out bowl with 999 red balls and 1 black one, for example, our knowledge about the presence of red balls grows each time we take out a red ball. But our knowledge of the absence of black balls grows more slowly.

This is Taleb’s key insight in his book Fooled by Randomness, and it has profound implications. A theory which states that all balls are red will likely be ‘corroborated’ with each observation. Our confidence that all balls are red will increase. Yet the probability that the next ball will be black will be rising all the time. If something hasn’t happened before or hasn’t happened for some time we assume that it can’t happen (hence the‚ this time it’s different syndrome). But we know that it can happen. Worse, we know that eventually it will.

In every tool we create, an idea is embedded that goes beyond the function of the thing itself. Just like the human brain, every technology has an inherent bias. It has within its physical form a predisposition toward being used in certain ways and not others.

It may be this bias that caused Professor Sendhil Mullainathan, whilst commenting on the Myth of A.I., to say he is:

“More afraid of machine stupidity than of machine intelligence.”

Bostrom is highly familiar with human bias having written Anthropic Bias, a book that since its first publication in 2002 has achieved the status of a classic.

A.I. Black Swan

In 2002, Nick Bostrom wrote of A.I. and SuperIntelligence Existential Risks:

When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question. [“Existential Risks”, 2002]

With my behavioral economics hat on I know that the evidence that we can’t forecast is overwhelming, however we must also always plan for and do our best to mitigate risks or ‘black swan’ events as best we can… it appears that the artificial intelligence community are doing a pretty good job of that.

MIT has an entire division, the Engineering Systems Division, that brings together researchers from engineering, the hard sciences, and the social sciences to identify and solve problems of complex engineered systems. One promising technique for engineering complex systems is known as axiomatic design, an approach conceived by Nam Suh, the former head of MIT’s Department of Mechanical Engineering. The idea of axiomatic design is to minimize the information content of the engineered system while maintaining its ability to carry out its functional requirements. Properly applied, axiomatic design results in airplanes, software, and toasters all just complex enough, and no more, to attain their design goals. Axiomatic design minimizes the effective complexity of the engineered system while maintaining the system’s effectiveness.

Professor Joanna Bryson has a trove of good information and research papers showing some of the efforts researchers are taking when it comes to mitigating A.I. risks.

The UK Government’s Chief Science Officer is addressing A.I. risk and what will be needed to govern such risk. The Association for the Advancement of Artificial Intelligence (AAAI) has a panel of leading A.I. researchers addressing the impact and influences of A.I on society. There are many others.

unPredictable

Of course I am aware that one counterintuitive result of a computer’s or A.I.’s fundamentally logical operation is that its future behavior is intrinsically unpredictable. However, I have a hard time believing an A.I. will want to destroy humanity and as much as I take the long-term risk of A.I. seriously I doubt it (Superintelligent A.I.) will happen in 5 or 10 years. We’re still not a paperless society. I can’t see a programmer, or mad scientist for that matter, capable of inventing a super intelligent A.I. programming it with: “Your mission, should you choose to accept it, is to eliminate all humans, wherever they may rear their head.”

I have gleamed many good insights from reading Superintelligence and recommend Bostrom’s book. I do not think human ingenuity will merely allow us to become lumbering robots, survival-machines entirely controlled by these super-machines. There is still something about being wiped out by a superintelligent A.I. that’s like squaring the circle. It doesn’t quite add up.


9 Comments

  1. Luke Muehlhauser says:

    > people who have worked closely with him, have indicated that there are ‘probably’ only 5 “computer scientists in the world currently working on how to programme the super-smart machines of the not-too-distant future to make sure A.I. remains friendly.” In his book presentation Authors@Google Bostrom claimed that only half a dozen scientists are working full time on the control problem worldwide (last 6 minutes). Which sounds like “technological tunnel vision,” and someone who has “trouble bringing other forces into view.”

    But did you see this clarification on the Financial Times story? Nick and I are aware of many different subfields of AI, different kinds of AI safety work, different catastrophic risks, etc. We just mean to say that there are vanishingly few people “explicitly doing technical research on the problem of how to ensure that a smarter-than-human AI has a positive impact even as it radically improves itself.” If you think otherwise, please point me to examples. This isn’t tunnel vision on our part, it’s just a simple fact that very, very few people are working in this newly emerging subfield of research.

  2. Luke Muehlhauser says:

    > Bostrom is highly familiar with human bias having written Anthropic Bias, a book that since its first publication in 2002 has achieved the status of a classic.

    Bostrom is pretty familiar with the cognitive biases literature, but that has nothing to do with his book Anthropic Bias, which is not about psychology but is instead about a problem in normative epistemology.

  3. Luke Muehlhauser says:

    > I have a hard time believing an A.I. will want to destroy humanity and as much as I take the long-term risk of A.I. seriously I doubt it (Superintelligent A.I.) will happen in 5 or 10 years.

    But neither Bostrom nor I believe either of these things, either. See here.

  4. Nice article. I thought Superintelligence was really interesting too, but I’m not entirely sure what to make of it yet. I agree with your prediction that it wouldn’t occur in 10 years from now. I wrote a short review of it in my most recent post, with a good lecture video by Bostrom added at the end. I can send you the link if you’d like to read it. By the way, you are probably familiar with Bostrom’s thought experiment about virtual reality/simulation. I haven’t talked to anyone about it, curious what your thoughts are on that

  5. Colin Lewis says:

    AI researcher Professor Yann Le Cunn has posted a comment which supports my opinion piece above:

    “A lot of opinions about the potentials benefits and dangers of AI in the comments to this piece by Jaron Lanier.

    There are several comments from people who know something about AI, and many comments by people who don’t know much about it.

    People who have been “in the trenches” trying to make progress towards AI are less worried about robots taking over the world than others because:

    (1) we know we are a long way from human-level AI, even if we believe that we are on the cusp of significant progress with many applications;

    (2) there are ways to build AI systems that will ensure their benevolence towards humans, the same way normally-constituted humans have built-in social behaviours that ensure the survival of the species;

    (3) AI can be useful without having a direct influence on the physical world: we can build AIs that are purely “problem-solving”, or “predictive”, not robot warrior, but artificial philosophers and scientists confined to virtual ivory towers.

    (4) as a last resort, there are ways to build inviolable benevolent behaviours and “kill switches” that even a super-intelligent AI could not disable;

    (5) we can build benevolent AIs to protect us against hypothetical malevolent ones.

    Regarding the seemingly endless exponential growth of computer power (a phenomenon put forth by proponents of the singularity), Neil Gershenfeld says something similar to what I’ve been saying for a while: “The history of technology advancing has been one of sigmoids that begin as exponentials.”

    http://edge.org/conversation/the-myth-of-ai

  6. […] Lewis plays with the idea of William Blake’s The [First] Book of Urizen as a prophetic anticipation of […]

  7. […] Lewis plays with the idea of William Blake’s The [First] Book of Urizen as a prophetic anticipation of […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: