LessWrong 2.0 Reader

Yes, that is the same idea. "This is a big pile of speculation that I don't take very seriously, but I feel like if we are being simulated, that's where most simulations of me would be instantiate" Why not take it seriously, if you accept high chance that (1) our reality is a simulation, (2)  we seem on track to creating a paperclip maximizer, (3) weird that I, Robert Miles, would have the personality traits that cause me to be one of the few humans so worried about humanity creating a paperclip maximizer if I'm right about us being on track to probably create one?

this experiment has been done before [EA · GW].

If you have a framing of the AI Doom argument that can cause a consensus of super-forecasters (or AI risk skeptics, or literally any group that has an average pDoom<20%) to change their consensus, I would be exceptionally interested in seeing that demonstrated.

Such an argument would be neither bad nor weak, which is precisely the type of argument I have been hoping to find by writing this post.

I think this cumulative argument works:

1. there are dozens of ways AI can prevent a mass extinction event at different stages at its existence.

2. ...

If you make a list of 1000 bad things and I make a list of 1000 good things, I have no reason to think that you are somehow better at making lists than prediction markets or expert surveys.

Imagine a circuit

A --> B --> C
D --> E --> F
C + F --> output

Denoising (restoring) B + E is sufficient to restore behaviour, but B + E are just a "cross-section" (cutting through the circuit at the 2nd layer), not a superset of all components

Predicting AGI/ASI timelines is highly speculative and unviable. Ultimately, there are too many unknowns and complex variables at play. Any timeline must deal with systems and consequences multiple steps out, where tiny initial errors compound dramatically. A range can be somewhat reasonable, a more specific figure less so, and accurately predicting the consequences of the final event when it comes to pass even further improbable. It is simply impractical to come up with an accurate timeline with the knowledge we currently have.

Despite this, timelines are popular – both with the general AI hype crowd and those more informed. People don't seem to penalise incorrect timelines – as evidenced by the many predicted dates we've seen pass without event. Thus, there's little downside to proposing a timeline, even an outrageous one. If it's wrong, it's largely forgotten. If it's right, you're lauded a prophet. The nebulous definitions of "AGI" and "ASI" also offer an out. One can always argue the achieved system doesn't meet their specific definition or point to the AI Effect.

I suppose fantastic @gwern [LW · GW]'s work on The Scaling Hypothesis is evidence of how an accurate prediction can significantly boost credibility and personal notoriety. Proposing timelines gets attention. Anyone noteworthy with a timeline becomes the centre of discussion, especially if their proposal is on the extremes of the spectrum.

The incentives for making timeline predictions seem heavily weighted towards upside, regardless of the actual predictive power or accuracy. Plenty to gain; not much to lose.

Ah that’s not actually what I meant but that’s clarifies it fully, thanks

Thanks for your support! I really think you're making a great impact here.

I'm a bit out of the loop, I used to think Anthropic was quite different from the other labs and quite in sync with the AI x-risk community.

Do you consider them relatively better? How would you quantify the current AI labs (Anthropic, OpenAI, Google DeepMind, DeepSeek, xAI, Meta AI)?

Suppose that the worst lab has a -100 influence on the future, for each $1 they spend. A lab half as bad, has a -50 influence on the future for each $1 they spend. A lab that's actually good (by half as much) might have a +50 influence for each $1.

What numbers would you give to these labs?^[1]

EDIT: I'm asking this question to anyone willing to answer!

1. ^{^}

   It's possible this rating is biased against smaller labs since spending a tiny bit increases "the number of labs" by 1 which is a somewhat fixed cost. Maybe pretend each lab was scaled to the same size to avoid this bias against smaller labs.

I think Anthropic did tests like this, e.g. in Alignment Faking in Large Language Models [LW · GW].

But I guess that's more of a "test how they behave in adversarial situations" study. If you're talking about a "test how to fight against them" study, that consists of "red teams" trying to hack various systems to make sure they are secure.

I'm not sure if the red teams used AI, but they are smart people and if AI improved their hacking ability I'm sure they would use them. So they're already stronger than AI.

See also: 

https://borretti.me/article/how-i-use-claude

I was planning on putting a whole list here but alas I am drawing a blank. 

There's a lot of dispersed wisdom in @Zvi [LW · GW]'s stack too but I can't remember any sufficiently discriminatory key words to find them.