There are several instances in the comments of the following schema: Someone finds lsusr's (self-admittedly not very fleshed out) arguments unsatisfactory and explains why; lsusr responds with just "Would you like to register a counter-prediction?".

I do not think this is a productive mode of discussion, and would like to see less of it. I think the fault is lsusr's.

These responses give me the very strong impression -- and I would bet fairly heavily that I am not alone in this -- that lsusr considers those comments inappropriate, and is trying to convey something along the lines of "put up or shut up; if you disagree with me then you should be able to make a concrete prediction that differs from mine; otherwise, no one should care about your arguments".

I see three possibilities.

First, maybe lsusr considers that (especially given the last paragraph in the OP) any discussion of the arguments at all is inappropriate. In that case, I think either (1) the arguments should simply not be in the OP at all or else (2) there should be an explicit statement along the lines of "I am not willing to discuss my reasoning and will ignore comments attempting to do so".

Second, maybe lsusr is happy to discuss the arguments, but only with people who demonstrate their seriousness by making a concrete prediction that differs from lsusr's. I think this would be a mistake, because it is perfectly possible to find something wrong in an argument without having a definite opinion on the proposition the argument is purporting to support. If someone says "2+2=5, therefore the COVID-19 pandemic was the result of a lab leak", one should not have to have formed a definite opinion on the origin of the pandemic in order to notice that 2+2 isn't 5 or that (aside from ex falso quodlibet) there is no particular connection between basic arithmetic and the origins of the pandemic.

(Also: some of those arguments may be of independent interest. lsusr's claim that poorly-aligned world-optimizing agents converge on "seizing competing pieces of their own mental infrastructure" sounds rather interesting if true -- and, for what it's worth, implausible to me prima facie -- regardless of how long the human race is going to survive.)

Third, maybe the impression I have is wrong, and  lsusr doesn't in fact disapprove of, or dislike, comments addressing the arguments without a concrete disagreement with the conclusions. In that case, I think lsusr is giving a misleading impression by not engaging with any such comments in any way other than by posting the same "Would you like to register a counter-prediction?" response to all of them.

Human simulators are unlikely to exterminate humanity by accident because the agent mesa optimizer is (more or less) human aligned and the underlying superintelligence (currently LLMs) is not a world optimizer.

If a superintelligent LLM is not a mesa-optimizer itself, it can be turned into an optimizer via a one-line bash script asking it to produce the shell commands that maximize some goal. So this isn't much help unless you can use that superintelligent LLM to patch the holes in humanity that would allow someone to squiggle the planet.

Inhuman world-optimizing agents are unlikely to turn the Universe into paperclips because that's not the most likely failure mode. A world-optimizing agents must align its world model with reality. Poorly-aligned world-optimizing agents instrumentally converge, not on siezing control of reality, but on the much easier task of siezing competing pieces of their own mental infrastructure. A misaligned world optimizer that seeks to minimize conflict between its sensory data and internal world model will just turn off its sensors.

1. This appropriates the word alignment in a way that is probably unhelpful to your thesis, whatever you intend to mean.

2. This is not what humans do, so it is clearly possible, conceptually speaking, for inhuman world-agents to target the outside world instead of maximize an internal worldscore variable. And even maximizing an internal worldscore variable can be unsafe, if the robot decides it wants to use all available matter to add "1s" to the number.

Inhuman world-optimizing agents are unlikely to turn the Universe into paperclips because that's not the most likely failure mode. 

Suppose 100 such agents are made. The first 99 brick themselves, they wirehead and then do nothing. The 100'th destroys the world. You need to claim that this failure mode is sufficiently unlikely that all AI experiments on earth fail to hit it. 

A world-optimizing agents must align its world model with reality [LW · GW]. Poorly-aligned world-optimizing agents instrumentally converge, not on siezing control of reality, but on the much easier task of siezing competing pieces of their own mental infrastructure.

You whatsmore this argument needs to hold, even in there are humans seeing their AI siezing their own mental infrastructure, and trying to design an AI that doesn't do this. Do you believe the only options are agents that harmlessly wirehead, and aligned AI?  

 A misaligned world optimizer that seeks to minimize conflict between its sensory data and internal world model will just turn off its sensors.

Maybe, but will it write a second AI, an AI that takes over the world to ensure the first AI receives power and doesn't have it's sensors switched off? If you really care about your own mental infrastructure, the easiest way to control it might be to code a second AI that takes nanotech to your chip.

Or maybe, once the conflict is done, and half the AI has strangled the other half, the remaining mind is coherent enough to take over the world.

This is just to state that I'm very far from buying any of the arguments as-written. They seem like dumb arguments. But then, you say you're not trying to explain your arguments here.

Human simulator mesa optimizer running on a non-agentic superintelligence

Even if LLM AGIs are directly aligned [LW(p) · GW(p)] (don't discard humanity themselves), they are not necessarily transitively aligned [LW(p) · GW(p)] (don't build something that discards both them and humanity). If they fail at not building inhuman world-optimizing agents as soon as they are able, there is going to be only a brief time when LLM AGIs are in charge, much shorter than 100 years.

Human simulators are unlikely to exterminate humanity by accident because the agent mesa optimizer is (more or less) human aligned and the underlying superintelligence (currently LLMs) is not a world optimizer.

GPT-N is not a Human simulator, but more like a "text-existing-on-the-internet simulator". If you give it a prompt conditioned on metadata with a future date, it will need to internally predict the future of humanity, and if it predicts that humanity does not solve alignment, then some significant fraction of the text on the internet might be written by malign AIs, which means that GPT-N will try to internally simulate a malign AI. I think there are ways of conditioning LLMs to mitigate this sort of problem, but it doesn't just fall out naturally out of them being "human simulators".

Nit: "if he does that then Caplan won't get paid back, even if Caplin wins the bet" misspells "Caplan" in the second instance.

Biowarfare [LW · GW] won't kill everyone. Nuclear war won't kill everyone. Anthropogenic global warming won't kill everyone. At worst, these will destroy civilization which, counterintuitively, makes Homo sapiens more likely to survive on the short term (century). The same goes for minor natural disasters like volcanic eruptions.

Natural disasters like giant meteors, or perhaps a gamma ray burst, are unlikely. The last time something like that happened was 66 million years ago. The odds of something similar happening in the next century are on the order of $2\times {10}^{-6}$. That's small enough to ignore for the purposes of this bet. The only way everyone dies is via AI

Strongly agreed with this. The only non-infinitesimal probability of human extinction comes from an alien intelligence (like AI) actively pursuing the goal of human extinction.  

Here's a third form of AGI:

https://www.lesswrong.com/posts/Aq82XqYhgqdPdPrBA/?commentId=Mvyq996KxiE4LR6ii [LW · GW]

This is a search for the most powerful and general algorithm (what I call a cognitive architecture), where you do not put any optimization pressure on the algorithm having "world optimizing" ability.

When you use the agent, you use it in "sessions", like LLMs, and those sessions are finite time and you clear state variables afterwards.  No online training, training must be all offline.  

For ongoing tasks, the agent must fill out a data structure that is human readable such that another agent can seamlessly "pick up" where the last ended session at.

The reason why this form of AGI is likely to beat the 2 forms you mention is :

    (1) it's much cheaper and faster to reach AGI and then ASI.  

    (2) it will outperform in ability the "human emulator", and will be measurably safer on tasks due to no state buildup than the "world optimizer".  The world optimizer cannot be tested for safety because the machine is constantly accumulating state.  You need to be able to test models where they are always in a known state, and you benchmark their reliability.  

Many of the ideas were drawn from : 

https://www.lesswrong.com/posts/HByDKLLdaWEcA2QQD/applying-superintelligence-without-collusion [LW · GW]

https://www.lesswrong.com/posts/5hApNw5f7uG8RXxGS/the-open-agency-model

Counter-predictions:

• Humanity will still be around in 2030 (P = 90%)
• ... in 2040 (P = 70%)
• A Strong AGI will be built by 2123 (P = 75%)
  • Conditional on this, no attempts at solving the alignment problem succeed (P = no idea / depends on our decisions)
    • Conditional on this, humanity survives anyway, because the AI is aligned by default or from some other reason we survive without solving alignment. (P = 10%)

The only way Yudkowsky benefits from his loan is if he spends all of his resources before 2030, but if he does that then Caplan won't get paid back, even if Caplin wins the bet.

I don't think that's true; he has sufficient earning potential to come up with the money in a short time frame near when it's due.