LessWrong 2.0 Reader

I think this is quite similar to my proposal in Capabilities and alignment of LLM cognitive architectures [LW · GW].

I think people will add cognitive capabilities to LLMs to create fully capable AGIs. One such important capability is executive function. That function is loosely defined in cognitive psychology, but it is crucial for planning among other things.

I do envision such planning looking loosely like a search algorithm, as it does for humans. But it's a loose search algorithm, working in the space of statements made by the LLM about possible future states and action outcomes. So it's more like a tree of thought or graph of thought than any existing search algorithm, because the state space isn't well defined independently of the algorithm.

That all keeps things more dependent on the LLM black box, as in your final possibility.

At least I think that's the analogy between the proposals? I'm not sure.

I think the pushback to both of these is roughly: this is safer how?

I don't think there's any way to strictly formalize not harming humans. My answer is halfway between that and your "sentiment analysis in each step of planning". I think we'll define rules of behavior in natural language, including not harming humans but probably much more elaborate, and implement both internal review [AF · GW], like your sentiment analysis but more elaborate, and external review [LW · GW] by humans aided by tool AI (doing something like sentiment analysis), in a form of scalable oversight.

I'm curious if I'm interpreting your proposal correctly. It's stated very succinctly, so I'm not sure.

I think this is quite similar to my proposal in Capabilities and alignment of LLM cognitive architectures [LW · GW].

I think people will add cognitive capabilities to LLMs to create fully capable AGIs. One such important capability is executive function. That function is loosely defined in cognitive psychology, but it is crucial for planning among other things.

I do envision such planning looking loosely like a search algorithm, as it does for humans. But it's a loose search algorithm, working in the space of statements made by the LLM about possible future states and action outcomes. So it's more like a tree of thought or graph of thought than any existing search algorithm, because the state space isn't well defined independently of the algorithm.

That all keeps things more dependent on the LLM black box, as in your final possibility.

At least I think that's the analogy between the proposals? I'm not sure.

I think the pushback to both of these is roughly: this is safer how?

I don't think there's any way to strictly formalize not harming humans. My answer is halfway between that and your "sentiment analysis in each step of planning". I think we'll define rules of behavior in natural language, including not harming humans but probably much more elaborate, and implement both internal review [AF · GW], like your sentiment analysis but more elaborate, and external review [LW · GW] by humans aided by tool AI (doing something like sentiment analysis), in a form of scalable oversight.

I'm curious if I'm interpreting your proposal correctly. It's stated very succinctly, so I'm not sure.

Just linkposted this too! [LW · GW]

Oh nice! My retinoid formula and moisturizer have hyaluronic acid mixed in so it's hard for me to isolate its effects, but a lot of people seem to find it hugely beneficial

Yeah. Well, since It was addressing a tribe of nomadic herders in prehistoric times, that in itself is a good thing :)

It might be relevant to note that the meaningfulness of this coherence definition depends on the chosen environment. For instance, in an deterministic forest MDP where an agent at a state $s$ can never return to $s$ for any $s$ and there is only one path between any two states, suppose we have a deterministic policy $\pi$ and let $s_1=\pi \left(s\right)$, $s_2=\pi \left(s_1\right)$, etc. Then for the zero-current-payoff Bellman equations, we only need that $V\left(s_1\right)>V\left(s^{\prime }\right)$ for any successor $s^{\prime }$ from $s$, $V\left(s_2\right)>V\left(s^{\prime \prime }\right)$ for any successor $s^{\prime \prime }$ from $s^{\prime }$, etc. We can achieve this easily by, for example, letting all values except $V\left(s_i\right)$ be near-zero; since $s_j$ is a successor of $s_i$ iff $j=i+1$ (as otherwise there would be a cycle), this fits our criterion. Thus, every $\pi$ is coherent in this environment. (I haven't done the explicit math here, but I suspect that this also works for non-deterministic $\pi$ and non-stochastic MDPs.)

Importantly, using the common definition of language models in an RL setting where each state represents a sequence of tokens and each action adds a token to the end of a sequence of length $t$ to produce a sequence of length $t+1$, the environment is a deterministic forest, as there is only one way to "go between" two sequences (if one is a prefix of the other, choose the remaining tokens in order). Thus, any language model is coherent, which seems unsatisfying. We could try using a different environment, but this risks losing stochasticity (as the output logits of an LM is determined by its input sequence) and gets complicated pretty quickly (use natural abstractions/world model as states?).

I suspect that in practice many people use the word "prioritize" to mean:

• think short-term
• only do legible things
• remove slack

“For my thoughts are not your thoughts, neither are your ways my ways,” declares the Lord.

-- Isaiah 55:8

This probably also implies: "your values are not my values".

In contrast, this almost makes it sound like you think it is plausible to align AI to its user's intent, but that this would be bad if the users aren't one of "us"—you know, the good alignment researchers who want to use AI to take over the universe, totally unlike those evil capabilities researchers who want to use AI to produce economically valuable goods and services.

If I'm being honest, I don't find this framing helpful.

If you believe that things will go well if certain actors gain access to advanced AI technologies first, you should directly argue that.

Focusing on status games feels like a red herring.

You can’t just trivially scale up the angular resolution by bolting more sensors together (or similar methods). It gets more difficult to engineer the lenses and sensors to meet super-high specs.

And aside from that, the problem behaves nonlinearly with the amount of atmosphere between you and the plane. Each bit of distortion in the air along the way will combine, potentially pretty harshly limiting how far away you can get any useful image. This may be able to be worked around with AI to reconstruct from highly distorted images, but it’s far from trivial on the face of it.