The outer objective of a language model is "predict the next token", which is not necessarily aligned. The most probable continuation of a sequence of words doesn't have to be friendly toward humans. I get that you want to set up a conversation in which it was told to be aligned, but how does that guarantee anything? Why is the most probable continuation not one where alignment fails?

If I tell a language model: "Create a sequence of actions that lead to a lot of paperclips", it is going to tell me a plan that just leads to a lot of paperclips, without being necessarily aligned. However, if I say "Create a sequence of actions that lead to a lot of paperclips and is aligned", it is going to assign high probability to tokens that create an aligned plan, because that's what I specified it to do.

And the sentence I singled out was about inner alignment; you asserted that mesa optimization wouldn't occur in such a system, but I don't see why that would be true. I also don't see why this system won't have a utility function. You can't really know this with the present level of interpretability tools.

I agree that it is possible that it could have mesa-optimisation and a utility function, but I also believe that it is possible to have neither, because that's what happened to humans. Better interpretability tools would be useful, indeed.

One problem is that if you assign negative infinity to any outcome, probably every action has negative infinity expected value since it has a nonzero probability of leading to that outcome.

I believe it should be possible to create an AI that thinks like a human: when we, let's say, go get a glass of water to drink, we don't think "I can't do that because that has a non-zero chance of killing someone", that is a very alien way to think, and I believe that human-like thinking happens by default unless you intentionally build it to compute every hypothesis about the world, which is probably computationally expensive anyway (literally anything could have a non-zero chance of happening: everybody dying, the sky turning pink, gravity being flipped, the infinite list goes on, you can't compute all of them).

But that's kind of an irrelevant theoretical point because with the current training paradigm, programmers don't get to choose the utility function; if the system has one, it emerges during training and is encoded into the neural network weights, which no one understands.

Again, it might be possible for an AGI to not have a utility function at all, though we would need good evidence to prove that it doesn't have one, which is why interpretability tools are needed.