The intelligence explosion, explained

The argument in four steps

1. Designing machines is intellectual work. It is done by people thinking.
2. Suppose a machine exists that out-thinks every person at every intellectual task. Good called this an ultraintelligent machine.
3. Then it is better than us at designing machines too. That includes designing a better version of itself.
4. The better version repeats the trick. Each generation designs the next, faster and better than the one before, and human intelligence is left far behind.

Good called the result an “intelligence explosion”. His conclusion is the line this site is named after:

“Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”

The first half of that sentence is quoted everywhere. The second half is the part that matters most to the people now working on AI safety.

Who was I. J. Good?

Irving John Good was born Isadore Jacob Gudak in London in 1916. He studied mathematics at Cambridge and in 1941 joined the codebreakers at Bletchley Park, where he worked under Alan Turing on the German naval Enigma and later on the Colossus computers.

After the war he worked in computing and statistics in Britain, then moved to the United States in 1967 and spent the rest of his career as a professor of statistics at Virginia Tech. He advised Stanley Kubrick on the film 2001: A Space Odyssey, whose computer HAL 9000 is one of fiction’s best-known thinking machines. He died in 2009.

The paper appeared in volume 6 of the journal Advances in Computers.

Fast or slow?

If the loop starts, how quickly does it run? This is known as the question of takeoff speed.

• Fast takeoff. Capability goes from roughly human to far beyond in days or months, too quickly for anyone to react. Eliezer Yudkowsky has argued for this view.
• Slow takeoff. The change plays out over years or decades, with many systems improving in step and the wider economy changing along the way. Robin Hanson and Paul Christiano have argued versions of this.

What decides it is how much each gain in intelligence helps with the next one, and how much the process depends on things that cannot be hurried: computer chips, energy, and experiments in the physical world.

Reasons to doubt it

• Diminishing returns. Each improvement may be harder to find than the last, so the loop slows down instead of speeding up.
• Bottlenecks. Better ideas still need chips to run on and experiments to test them. Factories and laboratories do not speed up just because the thinking does.
• Intelligence is not one number. The AI researcher François Chollet, among others, has argued that intelligence is tied to a task and an environment, so nothing can simply turn a dial marked “smarter”.
• The premise may never be met. The argument starts from a machine that is better than us at everything. Getting there may be much harder than progress so far suggests.

Why it still matters

None of the objections removes the loop; they argue about its speed. And a mild version is already visible. AI systems help write software, including the software used to train AI, and they have been used to help lay out computer chips. People remain firmly in that loop today. The open question is what happens as their share of the work shrinks.

Good’s condition is the practical point. If machines do most of the inventing, the thing left for people to get right is making sure those machines are working toward what we actually want. That problem has a name, alignment, and a growing research field.

From explosion to singularity

In 1993 Vernor Vinge gave Good’s idea its most famous name. He compared the arrival of superhuman intelligence to a singularity in physics: a point past which our models stop making predictions. The term technological singularity stuck, helped by Ray Kurzweil’s 2005 book The Singularity Is Near.