LessWrong 2.0 Reader

Also I suggest that given the number of tags in each section, load more should be load all.

At the end of 2023, MIRI had ~$19.8 mio. in assets. I don't know much about the legal restrictions of how that money could be used, or what the state for financial assets is now, but if it's similar then MIRI could comfortably fund Velychko's primate experiments, and potentially some additional smaller projects.

(Potentially relevant: I entered the last GWWC donor lottery with the hopes of donating the resulting money to intelligence enhancement, but wasn't selected.)

This is awesome! Three comments:

1. Please make an easy to find Recent Changes feed (maybe a thing on the home page which only appears if you've made wiki edits). If you want an editor community, that will be their home, and the thing they're keeping up with and knowing to positively reinforce each other.
2. The concepts portal is now a slightly awkward mix of articles and tags, with potentially very high use tags being quite buried because no one's written a good article for it (e.g Rationality Quotes has 136 pages tagged, but zero karma, so requires many clicks to reach). I'm especially thinking about the use case of wanting to know what types of articles there are to browse around. I'm not sure exactly what to do about this.. maybe having the sorting not be just about karma, but a mix of karma and number of tagged posts? Like (k+10)*(t+10) or something? Disadvantage is this is opaque and drops alphabetical much harder.
3. A bunch of the uncategorized ones could be categorized, but I'm not seeing a way to do this with normal permissions.

Adjusting 2 would make it much cleaner to categorize the many ones in 3 without that clogging up the normal lists.

The Real AI Alignment Failure Will Happen Before AGI—And We’re Ignoring It

Amazing writing in the story! Very captivating and engaging. This post raises an important concern—that AI misalignment might not happen all at once, but through a process of deception, power-seeking, and gradual loss of control. I agree that alignment is not a solved problem, and that scenarios like this deserve serious consideration.

But there is a deeper structural failure mode that may make AI takeover inevitable before AGI even emerges—one that I believe deserves equal attention:

The real question in AI alignment is not whether AI follows human values, but whether intelligence itself optimizes for sustainable collective fitness—defined as the capacity for each intelligence to execute its adaptive functions in a way that remains stable across scales. We can already observe this optimization dynamic in biological and cognitive intelligence systems, where intelligence does not exist as a fixed set of rules but as a constantly adjusting process of equilibrium-seeking. In the human brain, for example, intelligence emerges from the interaction between competing cognitive subsystems. The prefrontal cortex enables long-term planning, but if it dominates, decision paralysis can occur. The dopaminergic system drives motivation, but if it becomes overactive, impulsivity takes over. Intelligence does not optimize for any single variable but instead functions as a dynamic tension between multiple competing forces, ensuring adaptability across different environments.

The same principle applies to decentralized ecosystems. Evolution does not optimize for individual dominance but for the collective fitness of species within an ecosystem. Predator-prey relationships self-correct over time, preventing runaway imbalances. When a species over-optimizes for short-term survival at the cost of ecosystem stability, it ultimately collapses. The intelligence of the system is embedded not in any single entity but in the capacity of the entire system to adapt and self-regulate. AI alignment must follow the same logic. If we attempt to align AI to a fixed set of human values rather than allowing it to develop a self-correcting process akin to biological intelligence, we risk building an optimization framework that is too rigid to be sustainable. A real-time metric of collective fitness must be structured as a process of adaptive equilibrium, ensuring that intelligence remains flexible enough to respond to shifting conditions without locking into a brittle or misaligned trajectory.

Why the Real Alignment Failure Happens Before AGI

Most AI alignment discussions assume we need to "control" AI to ensure it follows "human values," but this is based on a flawed premise. Human values are not a coherent, stable optimization target. They are polarized, contradictory, and shaped by cognitive biases that are adaptive in some contexts and maladaptive in others. No alignment approach based on static human values can succeed.

The real question is not whether AI aligns with human values, but whether intelligence itself optimizes for sustainable collective fitness (collective well-being), in terms of the level of the collective ability of each individual to execute each of their functions.

If we look at where AI is actually being deployed today, the greatest risk is not from a rogue AI deceiving its creators, but from the rapid monopolization of AI power under incentives that are structurally misaligned with long-term well-being.

• Superhuman optimization capabilities will emerge in centralized AI systems long before AGI.
• These systems will be optimized for control, economic dominance, and self-preservation—not for the sustainability of intelligence itself.
• If AI is shaped by competitive pressures rather than alignment incentives, misalignment will become inevitable even if AI never becomes an independent agent seeking power.
• If we do not solve the centralization problem first, alignment failure is inevitable—even before AI reaches human-level general intelligence.

Why AI Alignment Needs a Real-Time Metric of Collective Fitness

Rather than attempting to align AI to human values, alignment must be framed as a real-time, adaptive process that ensures intelligence remains dynamically aligned across all scales of optimization.

What This Requires:

• A real-time metric of collective fitness that detects when intelligence is becoming misaligned due to centralization.
• A real-time metric of individual fitness that detects when decentralization is leading to inefficiency.
• A functional model of intelligence that ensures alignment does not become brittle or static.
• A functional model of collective intelligence that prevents runaway centralization before AGI even emerges.

But even if we recognize that AI alignment must be framed dynamically rather than statically, we face another problem: the way AI safety itself is structured prevents us from acting on this insight.

The Deeper Misalignment Failure: How Intelligence is Selected and Cultivated
The post assumes that AI misalignment is an event (e.g., AI deception leading to a coup). But misalignment is actually a structural process—it is already happening as AI is being shaped by centralized, misaligned incentives.

The deeper problem with AI alignment is not just technical misalignment or deceptive AI—it is the structural reality that AI safety institutions themselves are caught in a multi-agent optimization dynamic that favors institutional survival over truth-seeking. If we model the development of AI safety institutions as a game-theoretic system rather than an isolated, rational decision process, a troubling pattern emerges. Organizations tasked with AI alignment do not operate in a vacuum; they are in constant competition for funding, influence, and control over the AI safety narrative. Those that produce frameworks that reinforce existing power structures—whether governmental, corporate, or academic—are more likely to receive institutional support, while those that challenge these structures or advocate for decentralization face structural disincentives. Over time, this creates a replicator dynamic in which the prevailing AI alignment discourse is not necessarily the most accurate or effective but simply the one most compatible with institutional persistence.

This selection effect extends to the researchers and policymakers shaping AI safety. Institutions tend to favor individuals who can optimize within the dominant problem definition rather than those who challenge it. As a result, AI safety research becomes an attractor state where consensus is rewarded over foundational critique. The same forces that centralize AI development also centralize AI alignment thinking, which means that the misalignment risk is not just a future AGI problem—it is embedded in the very way intelligence is structured today. If AI safety is being shaped within institutions that are themselves optimizing for control rather than open-ended intelligence expansion, then any alignment effort emerging from these institutions is likely to inherit that misalignment. This is not just an epistemic blind spot—it is a fundamental property of competitive multi-agent systems. Any alignment solution that fails to account for this institutional selection dynamic risks failing before it even begins, because it assumes AI alignment is a purely technical problem rather than a structural one.

As a result, the institutions responsible for AI alignment are structurally incapable of seeing their own misalignment—because they select for intelligence that solves problems within the dominant frame rather than questioning the frame itself.

• If AI is not aligned to a real-time metric of collective fitness, misalignment will happen long before AGI—because centralized AI power structures will dictate misalignment before AI autonomy even becomes an issue.
• And why didn’t we solve this? Because the structures that trained AI researchers, policymakers, and engineers to think about alignment selected for individuals who optimize within the dominant paradigm, rather than those who question it.

Conclusion: AI Alignment Must Be Grounded in a Functional Model of Intelligence
The future of intelligence must not be dictated by the incentives of centralized AI power. Alignment is not a ruleset—it is a self-correcting process, and we are designing AI systems today that have no reason to self-correct.

• The real failure will occur not because AI takes over, but because we never built an AI system that was aligned with a functional model of intelligence itself in terms of modeling what outcomes intelligence functions to achieve.
• If we do not fix how intelligence is trained, structured, and rewarded, we will create AI that optimizes for power, not truth—even if we never reach AGI.

The real failure of AI alignment will not occur because AI takes over, but because we never built an AI system that was aligned with a functional model of intelligence itself—one that explicitly models what outcomes intelligence functions to achieve. But if the core failure is embedded in how we structure intelligence itself, then the real question is: what would an alignment framework that prioritizes intelligence as a dynamic optimization process actually look like in practice?

If collective fitness is the real alignment target, how do we define it in a way that remains stable as intelligence scales? What mechanisms could prevent intelligence from collapsing into centralized control without fragmenting into incoherence? Are there existing real-world intelligence structures—biological, social, or computational—that successfully maintain dynamic alignment over time? These questions are not just theoretical; they point toward a fundamental reframing of alignment as an evolving process rather than a fixed goal.

If AI safety is truly about alignment, then we should be aligning intelligence to the process that keeps intelligence itself stable across scales—not to static human values. What would it take to build a framework that makes this possible? I’d be interested in thoughts on whether this framing clarifies an overlooked risk or raises further questions. How does this perspective compare to traditional AI alignment strategies, and does it suggest a direction worth exploring further?

I still expect the Singularity somewhere in the 2030s, even under that model.

Have you written up your model of AI timelines anywhere?

In what sense is Georgism "leftish"?

I spent an hour this morning going through a bunch of articles on scam farms and I can say safely say I had no idea how bad it was for human trafficking victims. Like, embarrassingly unaware. 

> Imprisoning someone for one year in the USA costs in the order of 100,000 dollars

There surely must be some way to decrease that by *at least* a factor of 4 or so, possibly by an order of magnitude, if we wanted to?  (The poverty line for a 8-person household in the contiguous US in 2025 is $54,150.)  Surely that might involve treating prisoners in rather questionable ways, but still way less questionable than f---ing killing them, IMO.

Another objection I have is that [waaay too many things are considered crimes that shouldn't be](https://archive.org/details/threefeloniesday0000silv) -- what fraction of people in prison are there for reasons comparable to any of your examples?

True, if you were gonna vomit repeatedly. I suspect the association might be forged after only one or two times. Maybe it fades after one week, so you do it again, then it fades after one month, then a year... like it's an Anki card.

When we look at experience itself, there is no fixed “I” to be found. Boundaries between self and other aren’t innate to reality but drawn after the fact. We carve up this vast space of experience into “mine” and “yours,” but these divisions are somewhat arbitrary.

The boundaries are somewhat arbitrary, but it seems to me that if we keep going in this direction far enough, at the end of the road is equanimity with the universe being converted to paperclips. (Which would be a wrong thing in my current unenlightened opinion.) After all, there is no sharp boundary between "me" and a paperclip.