Is it possible to build a safe oracle AI?

It seems to me that Eliezer's post is just wrong. His argument boils down to this:

the AI needs goals in order to decide how to think: that is, the AI has to act as a powerful optimization process in order to plan its acquisition of knowledge, effectively distill sensory information, pluck "answers" to particular questions out of the space of all possible responses, and of course, to improve its own source code up to the level where the AI is a powerful intelligence. [...] the AI needs a goal of answering questions, and that has to give rise to subgoals of choosing efficient problem-solving strategies, improving its code, and acquiring necessary information.

It's not obvious that a Solomonoff-approximating AI must have a "goal". It could just be a box that, y'know, predicts the next bit in a sequence. After all, if we had an actual uncomputable black box that printed correct Solomonoff-derived probability values for the next bit according to the mathematical definition, that box wouldn't try to manipulate the human operator by embeding epileptic patterns in its predictions or something.

Maybe you could make a case that self-improvement requires real-world goals and is scary (instead of "superintelligence requires real-world goals and is scary"). But I'm not convinced of that either. In fact Karl's post shows that it's not necessarily the case. Also see Schmidhuber's work on Goedel machines, etc. Most self-improving thingies I can rigorously imagine are not scary at all.

It is indeed true that reinforcement learning AIs are scary. For example, AIXI can and will manipulate you into rewarding it. But there are many ideas besides reinforcement learning.

ETA: I gave an idea for AI containment sometime ago, and it didn't get shot down. There are probably many other ways to build a non-dangerous strong AI that don't involve encoding or inferring the unitlity function of humanity.

ETA 2: it turns out that the connotations of this comment are wrong, thanks roystgnr.

Peter Wegner has produced dozens of papers over the past few decades arguing that Turing Machines are inadequate models of computation, as computation is actually practiced. To get a sample of this, Google "Wegner computation interaction".

I had a look, and at a brief glance I didn't see anything beyond what CSP and CCS were invented for more than 30 years ago. The basic paper defining a type of interaction machine is something that I would have guessed was from the same era, if I didn't know it was published in 2004.

I say this as someone who worked on this sort of thing a long time ago -- in fact, I did my D.Phil. with Hoare (CSP) and a post-doc with Milner (CCS), more than 30 years ago. I moved away from the field and know nothing about developments of recent years, but I am not getting the impression from the 2006 book that Wegner has co-edited on the subject that there has been much. Meanwhile, in the industrial world people have been designing communication protocols and parallel hardware, with or without the benefit of these theoretical researches, for as long as there have been computers.

None of which is to detract from your valid point that thinking of the AI as a TM, and neglecting the effects of its outputs on its inputs, may lead people into the safe oracle fallacy.

To the contrary, the danger arises because the AI will interact with us in the interim between input and output with requests for clarification, resources, and assistance. That is, it will realize that manipulation of the outside world is a permitted method in achieving its mission.

Except this is not the case for the AI I describe in my post.

The AI I describe in my post cannot make request for anything. It doesn't need clarification because we don't ask it question in a natural language at all! So I don't think you're criticism apply to this specific model.

I sometimes think that part of the seductiveness of the "safe oracle AI" idea comes from the assumption that the AI really will be like a TM - it will have no interaction with the external world between the reading of the input tape and the writing of the answer. To the contrary, the danger arises because the AI will interact with us in the interim between input and output with requests for clarification, resources, and assistance. That is, it will realize that manipulation of the outside world is a permitted method in achieving its mission.

A forecaster already has acutators - its outputs (forecasts).

Its attempts to manipulate the world seem pretty likely to use its existing output channel initially.

You can do it in a very calculation intensive manner: take all programs of less than K bits (with K sufficiently big) calculate their answer (to avoid halting problem wait for the answer only a finite but truly enormous number of step, for exemple 3^^^3 steps) and compare it to the answer given by the program predictor. Of course you can't do that in any reasonable amount of time, which is why you're using the "good enough to be improved on" program predictor to predict the result of the calculation.