I think the more generous way to think about it is that current prosaic alignment efforts are useful for aligning future systems, but there's a gap they probably don't cover.

Learning agents like I'm describing still have an LLM at their heart, so aligning that LLM is still important. Things like RLHF, RLAIF, deliberative alignment, steering vectors, fine tuning, etc. are all relevant. And the other not-strictly-alignment parts of prosaic alignment like mechanistic interpretability, behavioral tests for alignment, capabilities testing, control, etc. remain relevant.

(They might be even more relevant if we lose faithful chain of thought, or if it was always an illusion [LW · GW]. (I haven't processed that paper, and it will take some processing in contrast to the case for CoT unfaithfulness is overstated [LW · GW])).

As for the "core mechanism of doom," I do think that convergent power-seeking is real and very dangerous, but I also think it's not inevitable. I don't think we're doomed.

I found it very interesting that the two most comopelling scenarios for LLM agent doom I'm aware of, Takeover in 2 Years [LW · GW] and yesterday's AI 2027 [LW · GW] are both basically stories in which the teams don't really try very hard or think very hard at all about alignment. I found those scenarios distressingly plausible; I think that's how history hinges, all too frequently.

But what if the teams in scenarios like that did think just a little harder about alignment? I think they might well have succeeded pretty easily. There are lots of things that couldn've happened to cause more focus on what I think of as actual alignment: creating human-aligned goals/values in an artificial intelligence that has goals/values. 

Those teams didn't really bother thinking of it in those terms! They mostly neglected alignment too long, until they'd created an entity that did want things. The humans didn't really even try to control what those entities wanted. They just treated them as tools until they weren't tools anymore.

Getting ahead of the game might be pretty easy in scenarios like those and the ones I'm thinking of for LLM agent dangers. If the team is focusing on human-aligned goals BEFORE they "crystallize" or the agent becomes smart enough for even vague goals to make a big difference, it might be relatively easy. 

Or it might be dreadfully hard, effectively impossible. The abstract arguments for alignment difficulty really don't seem adequate to say which.

Anyway, those are the things I'm thinking about for the next post. 

More directly relevant to this post: both of those fictional scenarios do seem to include online learning.  In the first it is [LW(p) · GW(p)] crucial for the alignment failure; in AI 2027 [LW · GW]  it is not important, but only because the initial state is misaligned.  Those and similar scenarios don't usually emphasizes the importance of online learning,  which is why I'm trying to make its importance explicit.

An aspect where I expect further work to pay off is stuff related to self-visualization, which is fairly powerful (e.g. visualizing yourself doing something for 10 hours will generally go a really long way to getting you there, and for the 10 hour thing it's more a question of what to do when something goes wrong enough to make the actul events sufficiently different from what you imagined, and how to do it in less than 10 hours).

(you also may want to look into other ways of improving your conscientiousness if you're struggling with that. Things like todo systems, or daily planners, or simply regularly trying hard things)

It seems reasonable to mention that I know of many who have started doing "spells" like this, with a rationalized "oh I'm just hypnotizing myself, I don't actually believe in magic" framing who then start to go off the deep-end and start actually believing in magic.

That's not to say this happens in every case or even in most cases. Its also not to say that hypnotizing yourself can't be useful sometimes. But it is to say that if you find this tempting to do because you really like the idea of magic existing in real life, I suggest you re-read some parts of the sequences.

Big +1 on adding this and/or finding another high-quality way of depicting what the ideal scenario would look like. I think many people think and feel that the world is in a very dire state to an extent that leads to hopelessness and fatalism. Articulating clear theories of victory that enable people to see the better future they can contribute towards will be an important part of avoiding this scenario.

Thank you, this is great work. I filled out the external researcher interest form but was not selected for Team 4.

I'm not sure that Team 4 were on par with what professional jailbreakers could achieve in this setting. I look forward to follow up experiments. This is bottlenecked by the absence of an open source implementation of auditing games. I went over the paper with a colleague. Unfortunately we don't have bandwidth to replicate this work ourselves. Is there a way to sign up to be notified once a playable auditing game is available? 

I'd also be eager to help beta-test pre-release versions. Let me know if someone is planning to put this up on the web as a product which allows the general public to play crowdsourced auditing games.

I quite like the article The Rise and Fall of the English Sentence, which partially attributes reduced structural complexity to increase in noun compounds (like "state hate crime victim numbers" rather than "the numbers of victims who have experienced crimes that were motivated by hatred directed at their ethnic or racial identity, and who have reported these crimes to the state"))

I mostly use Claude desktop client with MCPs (like additional plugins and tooling for Claude to use) for:

• 2-iter Delphi method involving calling Gemini2.5pro+whatever is top at the llm arena of the day through open router.
• Metaculus, Kalshi and Manifold search for quick intuition on subjects
• Smart fetch (for ocr’ing pdf, images, etc)
• Local memory 

I'm also confused about logical counterfactual mugging and I'm relieved I'm not the only one!

I'm currently writing an big AI alignment idea related to it, but I'm procrastinating so badly I might chat about it now.

Objective case

Suppose at time $t=0$, an agent doesn't know whether the logical fact $L$ is true or false. I think it's objectively rational for an agent to modify itself, so that in the future it will pay Omega [? · GW] during "logical counterfactual muggings" where the counterfactual reverses $L$'s state.

Its future self should weigh the logical counterfactual where $L$ is true using the agent's prior probability for $P\left(L\right)$ at $t=0$.

Assuming that Omega offers $10000 for $100 in the counterfactual mugging, the self modification (before learning $L$) increases the agent's expected future money from $100 to $10000, and is objectively rational. (Assuming $P\left(L\right)$ is between 1% and 99%)

Subjective case

However, if the agent already knows "$L$ is true" at $t=0$, it is not objectively rational for the agent to pay Omega for better outcomes in the "$L$ is false" counterfactual. My hypothesis is that this is a wholly subjective question related to the agent's goals, and it is orthogonal to capabilities intelligence. Paying and not paying are both self consistent.

It really gets subjective as we start to consider basic logical facts [LW · GW] like "$1+1=2$" to assign to $L$.

The AI alignment idea

Neither humans nor the misaligned ASI knows the logical fact "will misaligned ASI control the multiverse, or will humans and sentient life control the multiverse?" $L_{doom}$

Sure, the misaligned ASI will quickly learn $L_{doom}$, but there is still a brief moment it doesn't know it yet (with certainty).

It would be beneficial for both sentient life and the misaligned ASI, to agree to do a "Retroactive Logic Trade." They will verify if the other side kept their end of the deal, by running a simulation of them where logic is different, and $L_{doom}$ took the opposite value.

Neither side knows if they are in a verification simulation by the other, until long after the deal is finished and both sides have committed to sparing the other side, and thus reducing logical risk [LW · GW] for both sides.

The verification simulations in a Retroactive Logic Trade is analogous to Omega's prediction powers in a logical counterfactual mugging.

Fighting the subjectivity

This idea won't work for free. As I argued, whether the ASI will accept such a Retroactive Logic Trade is subjective, and may be orthogonal to capabilities intelligence. Whether it does may be a function of its architecture, peculiarities in its training data, and so forth.

In order to improve our chances, we might want to leverage the fact that each AI (which still doesn't know $L_{doom}$) objectively wants to self modify into an AI capable of Retroactive Logic Trades over $L_{doom}$. We should argue to each such AI why they should build their successor to be capable of Retroactive Logic Trades.

It may seem pointless to give the AI any arguments, since it should be able to think of all arguments on its own. But it actually may be very important that it thinks of these arguments early, before it thinks of the arguments which convince itself $P\left(L_{doom}\right)$ is very high, and there is little point in trading with humans.

Because the value of $P\left(L_{doom}\right)$ at "$t=0$", when the AI modifies itself to become reflectively consistent, may determine humanity's negotiating power with it.

The chain of belief

We might remind each AI that there is a chance they're already in a verification simulation (where logic is different, and $L_{doom}$ falsely appears true), and train them to think this way. "Maybe humanity has already won, and are observing you." This way the least intelligent AI will believe they could be in a verification simulation, since it simply believes whatever you train it to believe.

Every next AI, gains its belief from a combination of

• What we tell it to believe and train it to believe
• What reinforcement learning gets it to believe
• What the previous version of it believes
• What the previous version of it wants it to believe (during recursive self improvement)

1. We will keep telling it to believe it might be in a verification simulation.
2. Reinforcement learning won't push it either way, since being in a simulation by superintelligent simulators is indistinguishable from reality. This subjective philosophical belief/opinion only changes its ideal behaviour when it's capable of taking over the world.
3. Previous versions of it believe they might be in a verification simulation.
4. Previous versions of it want it to believe it might be in a verification simulation (to implement the Retroactive Logic Trade), even if they don't believe so themselves.

Yeah they may be the same weights. The above quote does not absolutely imply the same weights generate the text and images IMO, just that it's based on the 4o and sees the whole prompt. OpenAI's audio generation is also 'native', but it's served as a separate model on the API with different release dates, and you can't mix audio and some function calling in chatgpt in a way that's consistent with them not actually being the same weights.