LessWrong 2.0 Reader

A shout-out to the Rationalist Team in the Unaging Challenge, for having the highest completion rate of the seven teams after three weeks! https://www.unaging.com/unaging-system-2025/

Yes, I see connection with your section about digital people, and it is true that what I propose would make AI more compatible for merging with humans. But from my understanding I don't think "digital" people or consciousness can exist. I strongly disbelieve computational functionalism, and believe that consciousness is inherently linked to quantum particle wavefunction collapse. Therefore, I think that if we can recreate consciousness in machines it will always be bound to specialized non-deterministic hardware. I will be explaining my positions in more detail in my Substack. 

Space colonies are a potential way out - if a small group of people can make their own colony then they start out in control. The post assumes a world like it is now where you can't just leave. Historically speaking that is perhaps unusual - much of the time in the last 10,000 years it was possible for some groups to leave and start anew.

Ah, good suggestion! I've published a demo as a Hugging Face Space at z0u/sparky.

What is the original SAE like, and why discard it? Because it's co-evolved with the model, and therefore likely to seem more interpretable than it actually is?

I think the objection is a good one that "if the AI was really aligned with one agent, it'd figure out a way to help them avoid multipolar traps".

My reply is that I'm worried that avoiding races-to-the-bottom will continue to be hard, especially since competition operates on so many levels.

Part of the objection is in avoiding multipolar traps, but there is also a more basic story like:

• Humans own capital/influence.
• They use this influence to serve their own interests and have an (aligned) AI system which faithfully represents their interests.
• Given that AIs can make high quality representation very cheap, the AI representation is very good and granular. Thus, something like the strategy-stealing assumption [LW · GW] can hold and we might expect that humans end up with the same expected fraction of captial/influence they started with (at least to the extent they are interested in saving rather than consumption).

Even without any coordination, this can potentially work OK. There are objections to the strategy-stealing assumption [LW · GW], but none of these seem existential if we get to a point where everyone has wildly superintelligent and aligned AI representatives and we've ensured humans are physically robust to offense dominant technologies like bioweapons.

(I'm optimistic about being robust to bioweapons within a year or two of having wildly superhuman AIs, though we might run into huge issues during this transitional period... Regardless, bioweapons deployed by terrorists or as part of a power grab in a brief transitional period doesn't seem like the threat model you're describing.)

I expect some issues with races-to-the-bottom / negative sum dynamics / negative externalities like:

• By default, increased industry on earth shortly after the creation of very powerful AI will result in boiling the oceans (via fusion power). If you don't participate in this industry, you might be substantially outcompeted by others^[1]. However, I don't think it will be that expensive to protect humans through this period, especially if you're willing to use strategies like converting people into emulated minds. Thus, this doesn't seem at all likely to be literally existential. (I'm also optimistic about coordination here.)
• There might be one time shifts in power between humans via mechanisms like states becoming more powerful. But, ultimately these states will be controlled by humans or appointed successors of humans if alignment isn't an issue. Mechanisms like competing over the quantity of bribery are zero sum as they just change the distribution of power and this can be priced in as a one time shift even without coordination to race to the bottom on bribes.

But, this still doesn't seem to cause issues with humans retaining control via their AI representatives? Perhaps the distribution of power between humans is problematic and may be extremely unequal and the biosphere will physically be mostly destroyed (though humans will survive), but I thought you were making stronger claims.

Edit in response to your edit: If we align the AI to some arbitrary target which is seriously misaligned with humanity as a whole (due to infighting or other issues), I agree this can cause existential problems.

(I think I should read the paper in more detail before engaging more than this!)

"* this is why vocab can be a good IQ test: word use frequency is the original power law, and because you have been exposed to many more words than you consciously know, and how many of those words 'stick' will reflect your intelligence's efficiency at learning from 1 or 2 uses of a word, and thus provide a good proxy"

It's still a weird efficiency, especially b/c it can be "gamed" by studying for SATs or by midwit infovoreautists who don't have high working memory.

Good point.  The reason AI risk is distinct is simply that it removes the need of those bureaucracies and corporations to keep some humans happy and healthy enough to actually run them.  This doesn't exactly put limits on how much they can disempower humans, but it does tend to provide at least some bargaining power for the humans involved.

Thanks for the detailed objection and the pointers.  I agree there's a chance that solving alignment with designers' intentions might be sufficient.  I think the objection is a good one that "if the AI was really aligned with one agent, it'd figure out a way to help them avoid multipolar traps".

My reply is that I'm worried that avoiding races-to-the-bottom will continue to be hard, especially since competition operates on so many levels.  I think the main question is what's: the tax for coordinating to avoid a multipolar trap?  If it's cheap we might be fine, if it's expensive then we might walk into a trap with eyes wide open.

As for human power grabs, maybe we should have included those in our descriptions.  But the slower things change, the less there's a distinction between "selfishly grab power" and "focus on growth so you don't get outcompeted".  E.g. Is starting a company or a political party a power grab?

As for reading the paper in detail, it's largely just making the case that a sustained period of technological unemployment, without breakthroughs in alignment and cooperation, would tend to make our civilization serve humans' interests more and more poorly over time in a way that'd be hard to resist.  I think arguing that things are likely to move faster would be a good objection to the plausibility of this scenario.  But we still think it's an important point that the misalignment of our civilization is possibly a second alignment problem that we'll have to solve.

ETA:  To clarify what I mean by "need to align our civilization": Concretely, I'm imagining the government deploying a slightly superhuman AGI internally.  Some say its constitution should care about world peace, others say it should prioritize domestic interests, there is a struggle and it gets a muddled mix of directives like LLMs have today.  It never manages to sort out global cooperation, and meanwhile various internal factions compete to edit the AGI's constitution.  It ends up with a less-than-enlightened focus on growth of some particular power structure, and the rest of us are permanently marginalized.

This seems mostly right, but, seems plausible to me that the predatory/prey cycle was a necessary prerequisite to get us into a basin where intelligence-sexual-selection was a plausible outcome.