LessWrong 2.0 Reader

Sigh. Ok. I'm giving an upvote for good-faith effort to think this through and come up with a plan, but I just disagree with your world-model and its projections about training costs and associated danger levels so strongly that it seems hard to figure out how to even begin a discussion.

I'll just leave a link here [LW(p) · GW(p)] to a different comment talking about the same problem.

but it still says "it's easy for others to get their own superintelligences with different values", with 'superintelligence' referring to the 'superhuman' AI of 2035?

still confused about this btw. in my second reply to you i wrote:

(i wonder if you're using the term 'superintelligence' in a different way though, e.g. to mean "merely super-human"?)

and you did not say you were, but it looks like you are here?

far too many people tend to deny that you do in fact have to make other values lose out

i don't know where that might be true, but at least on lesswrong i imagine it's an uncommon belief. a core premise of alignment being important is value orthogonality implying that an unaligned agent with max-level-intelligence would compete for the same resources whose configurations it values (the universe). most of the reason for collaborating on alignment despite orthogonality is that our values tend to overlap to a large degree, e.g. most people (and maybe especially most alignment researchers?) think hells are bad.

also on the "lose out" phrasing: even if someone "wants at least some people to have tormentful lives", they don't "lose out" overall if they also positively value other things / still negatively value any of the vast majority of beings having tormentful lives.

My Current Metacognitive Engine

Someday I might work this into a nicer top-level post, but for now, here's the summary of the cognitive habits I try to maintain (and mostly succeed at maintaining). Some of these are simple TAPs, some of them are more like mindsets.

• Twice a day, asking “what is the most important thing I could be working on and why aren’t I on track to deal with it?”
  • you probably want a more specific question (“important thing” is too vague). Three example specific questions (but, don’t be a slave to any specific operationalization)
    • what is the most important uncertainty I could be reducing, and how can I reduce it fastest?
    • what’s the most important resource bottleneck I can gain, or contribute to the ecosystem, and would gain me that resource the fastest?
    • what’s the most important goal I’m backchaining from?
• Have a mechanism to iterate on your habits that you use every day, and frequently update in response to new information
  • for me, this is daily prompts and weekly prompts, which are:
    • optimized for being the efficient metacognition I obviously want to do each day
    • include one skill that I want to level up in, that I can do in the morning as part of the meta-orienting (such as operationalizing predictions, or “think it faster”, or whatever specific thing I want to learn to attend to or execute better right now)
• The five requirements each fortnight:
  • be backchaining 
    • from the most important goals
  • be forward chaining 
    • through tractable things that compound
  • ship something 
    • to users every fortnight
  • be wholesome 
    • (that is, do not minmax in a way that will predictably fail later)
  • spend 10% on meta (more if you’re Ray in particular but not during working hours. During working hours on workdays, meta should pay for itself within a week)
• Correlates:
  • have a clear, written model of what you’re backchaining from
  • have a clear, written model of how you’re compounding
• The general problem solving approach:
  • breadth first
  • identify cruxes
  • connect inner-sim to cruxes / predictions
  • follow your heart
  • see how your predictions went
• Random ass skills
  • napping
  • managing working memory, innovating and applying on working memory tools
  • grieving
  • Generalizing

Skill I’m working on that hasn’t paid off yet but I think you should try anyway:

• At least once a day or so, when you notice a mistake or surprise, spent a couple minutes asking “how could I have thought that faster” (and periodically do deeper dives)
• each day/week, figure out what you’re confused or predictably going to tackle in a dumb way, and think in advance about how to be smart about it the first time

I want to note something about how your position seems to have evolved through this discussion. Initially, you argued that societal pressure often reflects genuine wisdom, using examples where a 'society who aggressively shames overconsumption of sweets' might be wiser than a child's raw preferences. You suggested that what I was calling 'intrinsic preferences' might just be 'shallow preferences' that hadn't yet been trained to reflect reality.

Now you're making a different and more sophisticated argument - that the whole framework of 'intrinsic' versus 'external' preferences is problematic because preferences necessarily develop within and respond to reality, including social reality. While this is an interesting perspective that deserves consideration, it seems to contradict rather than support your initial defense of social restrictions as transmitting wisdom.

There's also an important point about my own position that I should clarify. When I said 'generally, upon reflection, people would prefer to satisfy their and others' preferences as calculated prior to such influences,' I wasn't making a claim about how often admonitions reflect preference inversions. Rather, I was suggesting that if people were to reflect explicitly on cases of preference inversion, they typically wouldn't want those inverted preferences to count; they would recognize these as preferences shaped by forces systematically opposed to their interests.

This connects to what I see as the core distinction: I'm not just talking about external influences or errors in the transmission of wisdom. I'm specifically pointing to cases where restrictions are moralized for the purpose of restriction itself - where the system is systematically deprecating the evolutionarily fit preferences of the person being restricted. This isn't just clumsy teaching or social pressure - it's adversarial. The system works by first making people feel guilty about their natural inclinations, then betting that they won't fully succeed at suppressing those inclinations despite earnestly trying to adopt the system's restrictions.

Consider the survival of variants of Christianity that 'do poorly' at helping people develop healthy attitudes toward sexuality. Their persistence suggests this poor performance is actually functional - they are able to exploit their members precisely because they create a system where most people must be 'bad' by design, where hypocrisy isn't a bug but a feature. When dessert companies can successfully market their products as 'sinfully delicious,' they're exploiting a system of moral restrictions that creates the very compulsive relationship to sweets it claims to prevent.

Prompt: write a micro play that is both disturbing and comforting
--

Title: "The Silly Child"

Scene: A mother is putting to bed her six-year-old child 

CHILD: Mommy, how many universes are there?

MOTHER: As many as are possible.

CHILD (smiling): Can we make another one?

MOTHER (smiling): Sure. And while we're at it, let's delete the number 374? I've never liked that one. 

CHILD (excited): Oh! And let's make a new Fischer-Griess group element too! Can we do that Mommy?

MOTHER (bops nose) That's enough stalling. You need to get your sleep. Sweet dreams, little one. (kisses forehead)

End

Einstein was not an experimentalist, yet was perfectly capable of physics; his successors have largely not touched his unfinished work, and not for lack of data.

While it is interesting at first glance, some caveats are called for here.

One, Einstein's achievements were sort of overrated, see these comments for details:

https://www.lesswrong.com/posts/GSBCw94DsxLgDat6r/interpreting-yudkowsky-on-deep-vs-shallow-knowledge#6HPjxMvTnP9JeibXZ [LW(p) · GW(p)]

https://www.lesswrong.com/posts/GSBCw94DsxLgDat6r/interpreting-yudkowsky-on-deep-vs-shallow-knowledge#icmCewLmXnxgtmANP [LW(p) · GW(p)]

Two, the EPR paradox is resolvable in modern physics by allowing non-locality in entanglement, but having a no-communication theorem that prevents exploiting it to break special relativity.

Societies aren't the issue; they're mindless aggregates that don't experience anything and don't actually even have desires in anything like the way a human, or or even an animal or an AI, has desires. Individuals are the issue. Do individuals get to choose which of these societies they live in?

I’m pretty sure he doesn’t buy the Christian Paradise of "having no job, only leisure is good actually" either.

This (a) doesn't have anything in particular to do with Christianity, (b) has been the most widely held view among people in general since forever, and (c) seems obviously correct. If you want to rely on the contrary supposition, I'm afraid you're going to have to argue for it.

You can still have hobbies.

I also kinda notice that there are no meaningful place left for humans in that society.

There's that word "meaningful" that I keep hearing everywhere. I claim it's a meaningless word (or at least that it's being used here in a meaningless sense). Please define it in a succinct, relevant, and unambiguous way.

If you believe that the democratic consensus made mostly of normal people will allow you that [Glorious Transhumanist Future], I have a bridge to sell to you.

The democratic consensus also won't allow a Butlerian Jihad, and I don't think you're claiming that it will.

So apparently nobody arguing for either can claim to represent either the democratic consensus or the only alternative to it. What's your point?

If you don’t have a plan then don’t build AGI, pretty please ?

I agree there.

This is obviously wrong. I won’t argue for why it is wrong — too long post, and so on.

I'm actually not sure what you're arguing for or against in this whole section.

Obviously you're not going to "solve human values". Equally obviously, any future, AI or non-AI, is going to be better for some people's values than others. Some values have always won, and some values have always lost, and that will not change. What that has to do with justice destroying the world, I have absolutely no clue.

I think you're trying to take the view that any major change in the "human condition", or in what's "human", is equivalent to the destruction of the world, no matter what benefits it may have. This is obviously wrong. I won't argue for why it's wrong, but now that I've said those magic words, you're bound to accept all my conclusions.

I still can’t believe some of you would sided with the super-happies !

So you're siding with the guy who killed 15 billion non-consenting people because he personally couldn't handle the idea of giving up suffering?

Wrong answers will disempower humans forever at best, reducing them to passive leafs in the wind.

Just like they are now and always have been. The Heat Death of the Universe (TM) is gonna eat ya, regardless of what you do.

Slightly wrong answers won’t go as far as that, but will result in the permanent loss of vast chunks of Human Values — the parts we will decide to discard, consciously or not.

Human Values have been changing, for individuals and in the "average", for as long as there've been humans, including being discarded consciously or unconsciously. Mostly in a pretty aimless, drifting way. This is not new and neither AI nor anything else will fundamentally change it. At least not while the "humans" involved are recognizably like the humans we have now... and changing away from that would be a pretty big break in itself, no?

You build your ASI. You have that big Diverse Plural Assembly that is apparently plan A

I haven't actually heard many people suggesting that.

I think its this