LessWrong 2.0 Reader

I was assuming very strongly superhumanly intelligent AI, but yeah no promises of optimality were made here.

That said, I suspect a crux is that optimality ends up with multipolarity, assuming a one world government hasn't happened by then, because I think the offense-defense balance moderately favors defense even at optimality, assuming optimal defense and offense.

Wow, thanks for doing the legwork on this - seems like quite possibly I'm analyzing fiction? Annoying if true.

Sigh. Ok. I'm giving an upvote for good-faith effort to think this through and come up with a plan, but I just disagree with your world-model and its projections about training costs and associated danger levels so strongly that it seems hard to figure out how to even begin a discussion.

I'll just leave a link here [LW(p) · GW(p)] to a different comment talking about the same problem.

but it still says "it's easy for others to get their own superintelligences with different values", with 'superintelligence' referring to the 'superhuman' AI of 2035?

still confused about this btw. in my second reply to you i wrote:

(i wonder if you're using the term 'superintelligence' in a different way though, e.g. to mean "merely super-human"?)

and you did not say you were, but it looks like you are here?

far too many people tend to deny that you do in fact have to make other values lose out

i don't know where that might be true, but at least on lesswrong i imagine it's an uncommon belief. a core premise of alignment being important is value orthogonality implying that an unaligned agent with max-level-intelligence would compete for the same resources whose configurations it values (the universe). most of the reason for collaborating on alignment despite orthogonality is that our values tend to overlap to a large degree, e.g. most people (and maybe especially most alignment researchers?) think hells are bad.

also on the "lose out" phrasing: even if someone "wants at least some people to have tormentful lives", they don't "lose out" overall if they also positively value other things / still negatively value any of the vast majority of beings having tormentful lives.

My Current Metacognitive Engine

Someday I might work this into a nicer top-level post, but for now, here's the summary of the cognitive habits I try to maintain (and mostly succeed at maintaining). Some of these are simple TAPs, some of them are more like mindsets.

• Twice a day, asking “what is the most important thing I could be working on and why aren’t I on track to deal with it?”
  • you probably want a more specific question (“important thing” is too vague). Three example specific questions (but, don’t be a slave to any specific operationalization)
    • what is the most important uncertainty I could be reducing, and how can I reduce it fastest?
    • what’s the most important resource bottleneck I can gain, or contribute to the ecosystem, and would gain me that resource the fastest?
    • what’s the most important goal I’m backchaining from?
• Have a mechanism to iterate on your habits that you use every day, and frequently update in response to new information
  • for me, this is daily prompts and weekly prompts, which are:
    • optimized for being the efficient metacognition I obviously want to do each day
    • include one skill that I want to level up in, that I can do in the morning as part of the meta-orienting (such as operationalizing predictions, or “think it faster”, or whatever specific thing I want to learn to attend to or execute better right now)
• The five requirements each fortnight:
  • be backchaining 
    • from the most important goals
  • be forward chaining 
    • through tractable things that compound
  • ship something 
    • to users every fortnight
  • be wholesome 
    • (that is, do not minmax in a way that will predictably fail later)
  • spend 10% on meta (more if you’re Ray in particular but not during working hours. During working hours on workdays, meta should pay for itself within a week)
• Correlates:
  • have a clear, written model of what you’re backchaining from
  • have a clear, written model of how you’re compounding
• The general problem solving approach:
  • breadth first
  • identify cruxes
  • connect inner-sim to cruxes / predictions
  • follow your heart
  • see how your predictions went
• Random ass skills
  • napping
  • managing working memory, innovating and applying on working memory tools
  • grieving
  • Generalizing

Skill I’m working on that hasn’t paid off yet but I think you should try anyway:

• At least once a day or so, when you notice a mistake or surprise, spent a couple minutes asking “how could I have thought that faster” (and periodically do deeper dives)
• each day/week, figure out what you’re confused or predictably going to tackle in a dumb way, and think in advance about how to be smart about it the first time

I want to note something about how your position seems to have evolved through this discussion. Initially, you argued that societal pressure often reflects genuine wisdom, using examples where a 'society who aggressively shames overconsumption of sweets' might be wiser than a child's raw preferences. You suggested that what I was calling 'intrinsic preferences' might just be 'shallow preferences' that hadn't yet been trained to reflect reality.

Now you're making a different and more sophisticated argument - that the whole framework of 'intrinsic' versus 'external' preferences is problematic because preferences necessarily develop within and respond to reality, including social reality. While this is an interesting perspective that deserves consideration, it seems to contradict rather than support your initial defense of social restrictions as transmitting wisdom.

There's also an important point about my own position that I should clarify. When I said 'generally, upon reflection, people would prefer to satisfy their and others' preferences as calculated prior to such influences,' I wasn't making a claim about how often admonitions reflect preference inversions. Rather, I was suggesting that if people were to reflect explicitly on cases of preference inversion, they typically wouldn't want those inverted preferences to count; they would recognize these as preferences shaped by forces systematically opposed to their interests.

This connects to what I see as the core distinction: I'm not just talking about external influences or errors in the transmission of wisdom. I'm specifically pointing to cases where restrictions are moralized for the purpose of restriction itself - where the system is systematically deprecating the evolutionarily fit preferences of the person being restricted. This isn't just clumsy teaching or social pressure - it's adversarial. The system works by first making people feel guilty about their natural inclinations, then betting that they won't fully succeed at suppressing those inclinations despite earnestly trying to adopt the system's restrictions.

Consider the survival of variants of Christianity that 'do poorly' at helping people develop healthy attitudes toward sexuality. Their persistence suggests this poor performance is actually functional - they are able to exploit their members precisely because they create a system where most people must be 'bad' by design, where hypocrisy isn't a bug but a feature. When dessert companies can successfully market their products as 'sinfully delicious,' they're exploiting a system of moral restrictions that creates the very compulsive relationship to sweets it claims to prevent.

Prompt: write a micro play that is both disturbing and comforting
--

Title: "The Silly Child"

Scene: A mother is putting to bed her six-year-old child 

CHILD: Mommy, how many universes are there?

MOTHER: As many as are possible.

CHILD (smiling): Can we make another one?

MOTHER (smiling): Sure. And while we're at it, let's delete the number 374? I've never liked that one. 

CHILD (excited): Oh! And let's make a new Fischer-Griess group element too! Can we do that Mommy?

MOTHER (bops nose) That's enough stalling. You need to get your sleep. Sweet dreams, little one. (kisses forehead)

End

Einstein was not an experimentalist, yet was perfectly capable of physics; his successors have largely not touched his unfinished work, and not for lack of data.

While it is interesting at first glance, some caveats are called for here.

One, Einstein's achievements were sort of overrated, see these comments for details:

https://www.lesswrong.com/posts/GSBCw94DsxLgDat6r/interpreting-yudkowsky-on-deep-vs-shallow-knowledge#6HPjxMvTnP9JeibXZ [LW(p) · GW(p)]

https://www.lesswrong.com/posts/GSBCw94DsxLgDat6r/interpreting-yudkowsky-on-deep-vs-shallow-knowledge#icmCewLmXnxgtmANP [LW(p) · GW(p)]

Two, the EPR paradox is resolvable in modern physics by allowing non-locality in entanglement, but having a no-communication theorem that prevents exploiting it to break special relativity.

Societies aren't the issue; they're mindless aggregates that don't experience anything and don't actually even have desires in anything like the way a human, or or even an animal or an AI, has desires. Individuals are the issue. Do individuals get to choose which of these societies they live in?