Top positive review
or a terrible, suddenly awakening monster which tries to destroy all ...
22 February 2018
Superintelligent computers in fiction tend to take one of two forms: a harmless calm-voiced sidekick (“Captain, I compute that this course of action has only a 13 per cent chance of success”), or a terrible, suddenly awakening monster which tries to destroy all life and must be defeated by a band of plucky humans who know what love means. According to Nick Bostrom, both of these prognoses are too optimistic.
The quest to create artificial intelligence is usually thought of as a crankish pursuit. But, says Bostrom, it could happen, perhaps within a few decades, perhaps within a few centuries. If it happens, then it will be the greatest change in human history. It could very easily be the end of human history.
Much of his warning sounds rather like the ancient Jewish myth of the golem, which destroyed its creator by following its instructions literally. If we build a machine that is much more intelligent than we are to do our bidding, without taking enormous care in defining what our bidding is, it could backfire in the most spectacular way. (His book opens with a parable of a group of sparrows saying that they really ought to find an owl chick and raise it as a servant.)