Box it. I'm not worried about AI itself, since Eliezer's quote is an anthropomorphism: an AI won't "want" to do anything unless it is programmed in such a way. The concern is about people running around with boxed AIs that can answer all their questions very accurately and thereby make them uber-powerful.
An AI won't be a super Watson. The idea behind an AI with human like intelligence is that it will be able to change itself. It will learn and rewrite its own program. Maybe it will be possible to give it some starting values and desires. Maybe not. But it will have its own desires eventually.
No, it won't. Desires are something humans have for evolutionary reasons. An AI is just a bunch of code. It can rewrite itself, but it can no more develop desires when it it loaded into a computer than it could if it were written down in book form. It will never "want" anything unless someone programs it to simulate such behavior, though as Cryddit points out there is a
literal genie effect to worry about, where in attempting to execute an instruction it could wreak havoc on the real world.
That's why I say box it. By "box it" I mean never give it the ability to affect the real world other than by providing information. It will affect its own world, where it can learn mathematical properties and reason out things based on given physical constraints (3D shapes cannot pass through each other; given these shapes, what can be built?). To the extent it can remake the outside world in its sandbox, it can potentially provide us with a lot of useful information without being able to directly affect anything. It can probably tell people how to make a dangerous device very easily, or how to override their own conscious and go postal. So limiting its communication ability might also be good. Maybe at first only allow it to answer yes/no questions, with yes and no.
You will say, "No, it's supremely intelligent and will find a way out of the box. It will outsmart us and figure out how to affect the real world." (Eliezer Yudkowsky's argument.) But this is akin to a category error. It assumes that faster processing power equates with God-like abilities, or even the ability to do the logically impossible. Can an AI tell you what the sum of Firlup + Waglesnorth is? Not if it doesn't know what you mean by those terms. Similarly, it is functionally impossible to for an AI to even
conceptualize the fact that it is, from our perspectives, in a box. An impossible thought is not thinkable even for a super-intelligence. We cannot conceptualize an infinite set, and neither can any AI. (In case this seems untrue, Eliezer also believes infinite sets are nonsense, to his credit. Whatever your position on infinite sets, if we for now accept that the notion is nonsense, an AI would not be any better at conceptualizing infinite sets, only better at noticing the meaninglessness of the term.)
You will say, "No, Nick Bostrom's simulation argument demonstrates how humans can reason about our own world being a simulation, where we are effectively in a box, as it were." But it doesn't. Bostrom's argument is merely a confusion based on equivocation. An AI would see this, as many people have. For example, scroll down to Thabiguy's comments
here. It is clear that there is no way to even conceptualize the notion of "we are living in a simulation" coherently. If you think you can visualize this, try drawing it. It's just high-falutin wordplay. If the notion is not even coherent, there is no way the further notion that "an AI would figure out it is living in a simulation" is coherent either. Hence it is of no concern.
AI != God. And heck, even a god could not do the semantically impossible like prove that 2+2=5, though it may be able to convince humans that it has done so.
With the right controls, AI can be boxed. Still, as I mentioned, the danger of the operator equipped with AI-furnished knowledge is a serious one.
Poll
