Strongly upvoted this comment.

But I'm not up for reworking the post right now.

And I'm not sure I fully agree with it, particularly point 1. I mean "goal directedness as argmax over action [or higher level mappings thereof] to maximise the expected value of a simple unitary utility function" and goal directedness as "contextually activated heuristics downstream of historical reinforcement events" seem to make pretty different predictions about the future (especially in the extremes), that I'm not actually sure I want to call the things that humans do "goal directed". It seems unhelpful to overload the term when referring to such different decision making procedures.

I mean I do in fact think one can define a spectrum of goal directedness, but I think the extreme end of that spectrum (argmax) is anti-natural, that the sophisticated systems we have in biology and ML both look more like "executing computations/cognition that historically correlated with higher performance on the objective function a system was selected for performance on" and that this is a very important distinction.

"Contextually activated heuristics" seem to be of an importantly different type than "immutable terminal goals" or a "simple unitary utility function".

I think for the purposes of this post, that distinction is very important and I need to emphasise it more.

i expect the thing that kills us if we die, and the thing that saves us if we are saved [LW · GW], to be strong/general coherent agents (SGCA) which maximize expected utility [LW · GW]. note that this is two separate claims; it could be that i believe the AI that kills us isn't [LW · GW] SGCA, but the AI that saves us still has to be SGCA. i could see shifting to that latter viewpoint; i currently do not expect myself to shift to believing that the AI that saves us isn't SGCA.

I don't share the pivotal act framing, so "AI that saves us" isn't something I naturally accommodate.

to me, this totally makes sense in theory, to imagine something that just formulates plans-over-time and picks the argmax for some goal. the whole of instrumental convergence is coherent with that

My contention is that "instrumental convergence" is itself something that needs to be rethought. From the post:

I think that updating against strong coherence would require rethinking the staples of (traditional) alignment orthodoxy:

* Orthogonality

* Basic AI drives

* Instrumental convergence (see^[2] [LW(p) · GW(p)])

This is not to say that they are necessarily no longer relevant in systems that aren't strongly coherent, but that to the extent they manifest at all, they manifest in (potentially very) different ways than originally conceived when conditioned on systems with immutable terminal goals.

And:

I think that updating against strong coherence would require rethinking the staples of (traditional) alignment orthodoxy:

* Orthogonality

* Basic AI drives

* Instrumental convergence (see^[2] [LW(p) · GW(p)])

This is not to say that they are necessarily no longer relevant in systems that aren't strongly coherent, but that to the extent they manifest at all, they manifest in (potentially very) different ways than originally conceived when conditioned on systems with immutable terminal goals.

So a core intuition underlying this contention is something like: "strong coherence is just a very unnatural form for the behaviour of intelligent systems operating in the real world to take".

And I'd describe that contention as something like:

Decision making in intelligent systems is best described as "executing computations/cognition that historically correlated with higher performance on the objective function a system was selected for performance on".

With the implication that decision making is poorly described as:

(An approximation) of argmax over actions (or higher level mappings thereof) to maximise (the expected value of) a simple unitary utility function

That expected utility maximisation is something that can happen does not at all imply that expected utility maximisation is something that will happen.

I find myself in visceral agreement with (almost the entirety) of @cfoster0 [LW · GW] 's reply. In particular:

• Goal-directedness in learning-based agents takes the form of contextual decision-influences (shards) steering cognition and behavior.
• [...]
• Even as they resolve these incoherences, agents will not need or want to become utility maximizers globally, as that would require them to self-modify in a way inconsistent with their existing preferences.

Agents with malleable values do not self modify to become expected utility maximisers. Thus an argument that expected utility maximisers can exist does not to me appear to say anything particularly interesting about the nature of generally intelligent systems in our universe.

why haven't animals or humans gotten to SGCA? well, what would getting from messy biological intelligences to SGCA look like? typically, it would look like one species taking over its environment while developing culture and industrial civilization, overcoming in various ways the cognitive biases that happened to be optimal in its ancestral environment, and eventually building more reliable hardware such as computers and using those to make AI capable of much more coherent and unbiased agenticity.

that's us. this is what it looks like to be the first species to get to SGCA. most animals are strongly optimized for their local environment, and don't have the capabilities to be above the civilization-building criticality threshold that lets them build industrial civilization and then SGCA AI. we are the first one to get past that threshold; we're the first one to fall in an evolutionary niche that lets us do that. this is what it looks like to be the biological bootstrap part of the ongoing intelligence explosion; if dogs could do that, then we'd simply observe being dogs in the industrialized dog civilization, trying to solve the problem of aligning AI to our civilized-dog values.

Would you actually take a pill that turned you into an expected utility maximiser^[1]? Yes or no please.

1. ^{^}

   Over a simple unitary utility function.

i expect the thing that kills us if we die, and the thing that saves us if we are saved, to be strong/general coherent agents

I agree in the sense that strong optimization is the likely shape of equilibrium (though I wouldn't go so far as to say it's utility maximization specifically), and in that equilibrium humanity is either fine or not. Conversely, while humanity remains alive, the doom status of the eventual outcome remains in question until there is a strong optimization equilibrium. Doom could come sooner, but singularity is fast in physical time, so the distinction doesn't necessarily matter.

But do you expect humans to build strong optimization? The way things are going, it's weakly coherent AGIs that are going to build strong optimization, while any alignment-relevant things humanity can do are not going to be about alignment of strong optimization, they are instead about alignment of weakly coherent AGIs (with LLM characters as the obvious candidate for successful alignment, and much more tenuous grounds for alignability of other things).

Agree on SGCA, if only because something is likely to self-modify to one, disagree on expected utility maximization necessarily being the most productive way to think of it.

Consider the following two hypothetical agents:

Agent 1 follows the deontological rule of choosing the action that maximizes some expected utility function.

Agent 2 maximizes expected utility, where utility is defined as how well an objective god's-eye-view observer would rate Agent 2's conformance to some deontological rule.

Obviously agent 1 is more naturally expressed in utilitarian terms, and agent 2 in deontological terms, though both are both and both can be coherent.

Now, when we try to define what decision procedure an aligned AI could follow, it might turn out that there's no easy way to express what we want it to do in purely utilitarian terms, but it might be easier in some other terms.

I especially think that's likely to be the case for corrigibility, but also for alignment generally.

Examples of strong coherence: Assuming AI systems:

• Are (expected) utility maximisers
• Have (immutable) terminal goals

I think even Wentworth's Subagents [LW · GW] is predicated on an assumption of stronger coherence than obtains in practice. I think humans aren't actually fully modeled as a fixed committee of agents making pareto optimal decisions with respect to their fixed utility functions. The ways in which I think Wentworth's Subagents falls short are:

• Human preferences change over time
• Human preferences don't have fixed weights, but activate to different degrees in particular contexts

And like I don't think those particular features are necessarily just incoherencies. Malleable values are I think just a facet of generally intelligent systems in our universe.

See also: nostalgebraist's "why assume AGIs will optimize for fixed goals? [LW · GW]".
And this comment [LW(p) · GW(p)].

 
I do think Subagents could be adapted/extended to model human preferences adequately, but the inherent inconsistency and context dependence of said preferences makes me think the agent model may be somewhat misleading.

E.g. Rob Bensinger suggested that agents may self modify to become more coherent over time [LW(p) · GW(p)].

 
Arguments that condition on strong coherence:

• Deceptive alignment in mesa-optimisers
• General failure modes from utility maximising

Basically, such environments are called "intelligent systems". I mean that if we train shard-agent to some lowest-necessary level of superintelligence, it will look at itself and say "Whoa, what a mess of context-dependent decision-making influencers I got here! I should refine it into something more like utility function if I want to get more things that I want" and I see literally no reason for it to not do it. You can't draw any parallels with actual intelligent systems here because actual intelligent systems don't have ability to self-modify.

Ridiculously strong levels of assuming your conclusion and failing to engage with the argument. Like this does not engage at all with the core contention that strong coherence may be anti-natural to generally intelligent systems in our universe.

Why did actual intelligent systems not develop as maximisers of a simple unitary utility function if that was actually optimal?

Why did evolution converge to agents that developed terminal values for drives that were instrumental for raising inclusive genetic fitness in their environment of evolutionary adaptedness?

I don't see reason for it to stay at that level of coherence.

You have not actually justified the assumption that it would self modify into strong coherence.

That strong coherence is optimal.

That systems with malleable values would necessarily want to self modify into immutable terminal goals.

I would NOT take a pill to turn myself into an expected utility maximiser.