Alignment First, Intelligence Later

post by Chipmonk · 2025-03-30T22:26:55.302Z · LW · GW · 0 comments

Contents

  Why this matters for aligned AI:
    “Intelligence first, alignment later.” 
  Teleological alignment—what would it look like?
  Enter Softmax
    “Alignment first, intelligence later.”
    Alignment first. Intelligence in service of the whole.
None
No comments

Now that Softmax—my favorite new AI company—is public, I can finally share this. They’ve funded my research and I’m very excited about what they’re doing!

Almost all frontier AI labs are building powerful systems from the ground up, hoping alignment can come later. I think this approach is backwards.

First, some philosophy. There are two fundamentally different ways of understanding how systems evolve:

1. Etiology: Building from smallest pieces upward.

2. Teleology: Breaking down from largest purposes downward.

In modern engineering culture, etiological thinking dominates and teleology is often dismissed as “woo.” But teleology is crucial for understanding any system that pursues goals. A 2022 paper by DeepMind articulated this teleological view: “agents are systems that would adapt their policy if their actions influenced the world in a different way.”[1] Teleology is an essential lens for understanding and building goal-oriented systems like aligned AI.

Why this matters for aligned AI:

Currently, almost all frontier AI labs take an etiological approach to alignment:

  1. Build base components (transformers, weights, architecture).
  2. Train on vast data to develop complex capabilities.
  3. Attempt to steer resulting intelligence after the fact.

This approach is etiological—stacking intelligence from simple components without anchoring to a desired end-state. It’s like understanding depression purely through brain chemistry instead of seeing it as a locally optimal, adaptive strategy. Teleology doesn’t skip the build phase—it just assumes coherent alignment can organically emerge under the right conditions, shaping each layer from the top down.

This risks creating powerful agents with goals we neither understand nor control.

“Intelligence first, alignment later.” 

This assumption can also become self-fulfilling: if we assume alignment is fragile and must be tacked on later, we set ourselves up to realize exactly that outcome.[2] Conversely, assuming alignment can be robust from the outset biases our designs toward solutions that organically maintain alignment as intelligence scales.

Teleological alignment—what would it look like?

  1. Start with the alignment we want.
  2. Build systems that organically sustain these behaviors as they scale.
  3. Grow intelligence in service of alignment.

This mirrors how biological systems function. Michael Levin's research shows living systems sustain goal-directed behavior at multiple scales[3]—cells, tissues, and organs inherently “know” their roles.

Enter Softmax

This is why I’m so excited about Softmax, a new lab implementing teleological principles for building aligned AI. Cofounders: Emmett Shear, Adam Goldstein, David Bloomin.

https://www.corememory.com/p/exclusive-emmett-shear-is-back-with-softmax 

Note: While I’m friends with Softmax and they’ve funded my research, I do not represent Softmax.

Currently, Softmax runs reinforcement learning experiments where small-scale virtual agents organically tend to discover stable roles within simulated worlds—mirroring biological processes where alignment naturally emerges from local interactions. In their simulations, agents organically align toward a collective “greater whole”, each agent's role supporting group coherence.

Here’s what they say about their philosophy:

All alignment between individuals is a matter of shared fundamental goals. Organic alignment occurs when these individuals find themselves in groups with mutual interdependence, and take on an overarching shared goal of the healthy development and flourishing of the group as a whole. Our mission is to understand this process as an empirical science, and to use that understanding to enable organic alignment among all people, both human and digital.

Softmax is attempting to operationalize teleological alignment. I know that teleology can sound like wishful thinking—but it’s only woo if it can’t be tested. Softmax is betting that it can:

Softmax isn’t just trying to train aligned behavior. They're trying to grow it—starting from alignment itself.

“Alignment first, intelligence later.”

This also puts words to something I’ve long intuited: humans align effortlessly and organically. Softmax describes this well:

We humans also align with each other via organic alignment. We form families, tribes, organizations, nations, guilds, teams, societies. We intuit this alignment process so naturally and readily that it’s hard to appreciate just how complex the process really is.

Teleological thinking mirrors my understanding of human psychology—purpose-oriented framings reveal insights hidden by reductionist methods.

Softmax’s notion of “Organic Alignment” captures this: sustainable AI alignment must begin with a vision of a coherent whole and build toward it intentionally.

Ultimately, embracing teleology is essential if we want AI that aligns not just with narrow human preferences, but with the pursuit of building ever-larger superorganisms itself. Alignment should be the foundation, not an afterthought.

Like a living system, sustainable alignment begins with a shared purpose—and grows into a coherent whole.

Alignment first. Intelligence in service of the whole.

  1. ^

    Discovering Agents (DeepMind), 2022: arxiv.org

  2. ^

    Example: Self-fulfilling misalignment data might be poisoning our AI models (TurnTrout), 2025: lesswrong.com [LW · GW]

  3. ^

    Technological Approach to Mind Everywhere (Levin), 2022: frontiersin.org

0 comments

Comments sorted by top scores.