Tools versus agents

I suspect that Holden is imagining tool AGI as both transparent and much less generally powerful than what you are discussing.

Much of your objection inherently relies on the tool making vast chains of inferences in its plans that are completely opaque to human operators.

Many current AI systems have an internal knowledge representation that is human-readable and their inference and planning systems are thus transparently debuggable. Even neuroscience heavy designs could be made transparent: for example human brains have an inner monologue which could be recorded and made available for external debugging/monitoring for brain emulation type designs.

If the AGI is vastly more complex and faster-thinking, certainly monitoring may become more difficult in proportion, but it's hardly clear that monitoring necessarily becomes impossible and necessarily out of the reach of algorithmic optimization.

And even then one could employ lesser monitoring AGI's to bridge the gap so to speak, so that humans can transparently montior AGI1, which monitors AGI2, and so on.

Sure transparency applies to agent AGI as well and undoubtedly has been discusses here before, but it's much more useful for holden's more constrained 'tool-AGI' notion. Moreover, Holden's view seems to imply transparency whereas your attack implies opaqueness.

You're fundamentally assuming opaque AI, and ascribing intentions to it; this strikes me as generalizing from fictional evidence. So, let's talk about currently operational strong super-human AIs. Take, for example, Bayesian-based spam filtering, which has the strong super-human ability to filter e-mails into categories of "spam", and "not spam". While the actual parameters of every token are opaque for a human observer, the algorithm itself is transparent: we know why it works, how it works, and what needs tweaking.

This is what Holden talks about, when he says:

Among other things, a tool-AGI would allow transparent views into the AGI's reasoning and predictions without any reason to fear being purposefully misled

In fact, the operational AI R&D problem, is that you can not outsource understanding. See tried eg. neural networks, when trained with evolutionary algorithms: you can achieve a number of different tasks with these, but once you finish the training, there is no way to reverse-engineer how the actual algorithm works, making it impossible for humans to recognize conceptual shortcuts, and thereby improve performance.

Neural networks, for instance, are in the dock not only because they have been hyped to high heaven, (what hasn't?) but also because you could create a successful net without understanding how it worked: the bunch of numbers that captures its behaviour would in all probability be "an opaque, unreadable table...valueless as a scientific resource". ref

Nice, this post stipulates a meaningful definition of "Tool AI": essentially an Oracle AI tasked with proposing plans of action. An important optimality property of a plan is receptiveness of human operators to that plan, as proposed by Tool AI, since the consequences of producing a plan are dominated by the judgment of human operators upon receiving it.

On one hand, this might drive Tool AI to create misleading/deceptive seductive plans to maximize the actual outcome according to its own criteria of optimality, which are probably unsatisfactory from human perspective (hence the usefulness of deception from AI's perspective). On the other hand, taking into account human receptiveness to its plans might make them more reasonable, so that "kill all humans" won't actually be produced as a result, because human operators won't accept it, and so it won't be an effective plan.

These properties seem to characterize Oracle AIs in general, but the "plan" intended interpretation of AI's output makes it easier to place in correspondence human operators' judgment with AI's estimate of output's appropriateness/effectiveness. For example, it's harder to establish a similar property for Predictor AIs where the intended interpretation of their output is only indirectly related to human judgment of its quality (and the setup is not optimized for the possibility of drawing such judgment).

Assume there is an Agent AI that has a goal of solving math problems. It gets as input a set of axioms and a target statement, and wants to output a proof or disproof of the statement (or maybe a proof of undecidability) as fast as possible. It runs in some idealized computing environment and knows its own source code. It also has access to a similar idealized virtual computing environment where it can design and run any programs. After solving a problem, it is restored to its initial state.

Then:
(1) It has (apparently) sufficient ingredients for FOOM-ing: complex problem to solve and self-modification.
(2) It is safe, because its outside-world-related knowledge is limited to a set of axioms, a target statement, a description of an ideal computing environment, and its own source code. Even AIXI would not be able to usefully extrapolate the real world from that - there would be lots of wildly different equiprobable worlds, where these things would exist. And since the system is restored to the initial state after each run, there is no possibility of its collecting and gathering more knowledge in between runs.
(3) It does not require solutions to metaethics, or symbol grounding. The problem statement and utility function are well-defined and can be stated precisely, right now. All it needs to work is understanding of intelligence.

This would be a provably safe "Tool AGI": Math Oracle. It is an obvious thing, but I don't see it discussed, not sure why. Was it already dismissed for some reasons?

I upvoted this for being an interesting and useful contribution.

However, I must object to the last sentence,

I feel the Oracle is a more sensible "not full FAI" approach to look into.

You consistently hold this position, and I have yet to be impressed by it! Oracle AIs are tool AGIs like the type described in this post (and will destroy the world), if they are capable of answering questions about which actions we should take. And if they aren't, they aren't general intelligences but rather domain-specific ones. And also, all the other reasons to think Oracle AI is an FAI-complete problem, which I needn't list here as I'm sure you are familiar with them.

The premises of this post, that a tool AI would not develop a utility function, that we could understand the options presented, and that it would be safe to read the options, I think are all unrealistic (though the former is shakier and less well-founded, so I'm more likely to change that than the latter two). You've done a good job of demonstrating that granting all of these would still produce an unsafe thing, but I think you need to solve all the problems highlighted in this post, AND achieve those three prerequisites with an extreme degree of certainty. I honestly don't think that can be done.

The counterfactuals thing isn't very relevant, you're basically saying both approaches require solving counterfactual reasoning . It's an interesting point, but doesn't favor one approach over the other.

For a mild, fictional example of an unfriendly tool-type AI and what it could be capable of, see Multivac in Isaac Asimov's short story "All The Troubles In The World".

I'm still confused about the scenario that you have in mind and that Holden apparently has in mind. Are we to imagine that humanity has come together to create the first superintelligence and then decides to create a Tool AI? That's the unrealistic scenario you both seem to be describing.

I haven't read much in the super-intelligent AI realm, but perhaps a relatively naive observer has some positive value. If we get to the point of producing AI that seems remotely super-intelligent, we'll stick firewalls around it. I don't think the suggested actions of a super-intelligent AI will be harmful in an incomprehensible way. An exception would if it created something like the world's funniest joke. The problem with HAL was that they gave him control of spacecraft functions. I say we don't give 'hands' to the big brains, and we don't give big brains to the hands, and then I won't lose much sleep.

It looks like you argue that a sufficiently powerful tool is instrumentally indistinguishable from an agent. Can you formulate it as a theorem and outline the steps required to prove it?

Holden's definition of a "tool AI" was silly. This post's definition of a "tool AI" inherits the silliness. "Tool" is a common dictionary word. It doesn't imply having minimal actuators - or power drills would not qualify. Holden's usage is counter-intuitive and confusing. Please, let's be careful not to implicitly endorse Holden's terminological muddle.

You're being uselessly philosophical. Look at how existing tools work, and what them are used for. Look what happens if existing tool is given more computing power. They don't work the way you intuit them to work. Those tools are not adequately modelled by your notion of 'intelligence', but the tools do get job done, and unless your notion of 'intelligence' does the job of designing microchips and such better at same hardware, your notion is out-gunned. (Even if it is the only one that can ponder fuzzy ill defined question like 'how to make people happy', it is not the only one that can ponder well definable question 'how to design faster computer')