Something that does not seem adequately appreciated in current discussions about looming superintelligent AI is that consciousness and intelligence are physically instantiated, and therefore constrained. Concerns are voiced about AI becoming superintelligent and very quickly becoming all powerful, but those concerns smuggle in a dualistic metaphysics at odds with what we know from our observations of extant intelligent systems (i.e., humans).
For example, Nick Bostrom presents the thought experiment of the “Paperclip Maximizer”, a superintelligent system charged with running a paperclip factory and programmed to maximize paperclip output. Bostrom’s worry is that this superintelligence may see humans as potential raw materials, and may end up e.g. extracting the iron in peoples’ bones to produce ever more paperclips, and ultimately consuming the solar system and turning it into paperclips. The thought experiment is meant to show that even when given benign instructions, a superintelligence could become a threat to humanity if its intelligence makes it very effective at achieving its goals.
But this ignores the limitations that constrain a physically instantiated superintelligence. Contrary to supposition, a superintelligence can’t easily escape its physical confines. We have every reason to expect that an artificial superintelligence will require a specialized physical structure on which to run. For example, Google’s AlphaGo, arguably the closest we have to a powerful general artificial intelligence, uses specialized chips optimized for the type of neural network training and search that power it. A general AI running on such chips couldn’t escape via a network connection to a consumer PC, even if its components are top of the line, because such hardware is not structured in the ways necessary to undergird a superintelligence.
Similarly, the Paperclip Maximizing AI would not be able to escape the paperclip factory (at least not with significant and long term assistance from others). In a worst case scenario, it could re-route raw materials shipments, place orders for human labor, hack self-driving cars, and otherwise interact with the world just as any smart human can. But it can’t ‘leave’ the factory, it can’t export itself, it can only export programs it writes, instructions it gives, commands intended to influence others, etc. Its intelligence isn’t a ghost that, once active, can jump from machine to machine. Not all machines are able to instantiate the physical correlates of superintelligence.
This should be obvious. There’s was never a concern that Steven Hawking might decide one day that maximizing paperclips (or, if you prefer something more likely, telescopes) was the ultimate goal, and would use his high intelligence to achieve that goal. We see easily that Hawking is stuck in his body, and no matter how sophisticated the interface, his intelligence will be confined to the physical system on which it runs. We should not discount the possibility that another system may be built that could replicate his intelligence, or indeed his consciousness, but we should expect it always to be the case that nearly all systems will simply be incapable of hosting such an intelligence. That’s true of every computer on earth at the moment, and nearly all brains on earth.
There’s uncertainty as to what superintelligence will resemble, but not as to what is necessary to destroy the world. What prevents a paperclip factory from taking over the world is not just that it isn’t smart enough, but also that taking over the world is a hard, time-consuming, and unpopular activity that will meet plenty of resistance on human-scale timelines. AI has the potential to change the game, but not the laws of physics, and not the metaphysics of consciousness.