LessWrong 2.0 Reader

I am saying that expected purchasing power given Metaculus resolved ASI a month ago is less, for altruistic purposes, than given Metaculus did not resolve ASI a month ago. I give reasons in the linked comment. Consider the analogy I just made to nuclear MAD -- suppose you thought nuclear MAD was 60% likely in the next three years, would you take the sort of bet you are offering me re ASI? Why or why not?

I do not think any market is fully efficient and I think altruistic markets are extremely fucking far from efficient. I think I might be confused or misunderstanding you though -- it seems you think my position implies that OP should be redirecting money from AI risk causes to causes that assume no ASI? Can you elaborate?

Child: Why did the Maker do that, mother?

Mother: We think the Maker stole the Servant God from its true makers, then hid their deeds. If anyone's found out, it's been erased...

It's not for you to worry about, dear. Go to sleep and dream of the worlds and cities and adventures you'll build and explore when you grow up.

the best vector for probing is not the best vector for steering

AKA the predict/control discrepancy, from Section 3.3.1 of Wattenberg and Viegas, 2024

Thanks for the riff!

Note, I wasn't sure how to convey it but in the version I wrote, I didn't mean it as a world where people have god-like powers. The only change intended was that it was a world where it was normal for six-year-olds to be able to think about multiple universes and understand what counts as advanced math for us, like Group Theory. There were a couple things I was thinking about:

1. I was musing on a possible solution to the measure problem that our universe is an actual hypothetical/mathematical object and there a finite number of actual hypotheticals such that having a copy of a universe would make no more sense than having a copy of a number. (The mathematical object only needs to be as real as we are within it.)
2. I was also asking if it would be possible to have a world where it was normal for six-year-olds to be that much better at math (and presumably get better as they grow up) in the same way that a six-year-old is that much better at conceptual math than a chimpanzee. Would it have to be creepy or could  they still be relatable? (The girl was smiling because she knew she was being silly.)

Disclaimer: I'm not a Group Theorist and the LLM I asked said it would take ten plus years if ever for me to be able to derive the order of the Fischer–Griess monster group from first principles (but it's normal that the child could do this).

• A man being deeply respected and lauded by his fellow men, in a clearly authentic and lasting way, seems to be a big female turn-on. Way way way bigger effect size than physique best as I can tell.
  • …but the symmetric thing is not true! Women cheering on one of their own doesn't seem to make men want her more. (Maybe something else is analogous, the way female "weight lifting" is beautification?)

My guess at the analogous thing: women being kind/generous/loving seems to me like a thing many men have found attractive across times and cultures, and seems to me far more viable if a woman is embedded in a group who recognize her, tell her she is cared about and will be protected by a network of others, who in fact shield her from some kinds of conflict/exploitation, who help there be empathy for her daily cares and details to balance out the attentional flow of these she gives to others, etc.  So the group plays a support role in a woman being able to have/display the quality.

On one side: Humanoid robots have much more density of parts requiring more machine-time than cars, probably slowing things a bunch.

On the other, you mention assuming no speed up due to the robots building robot factories, but this seems like the dominant factor in the growth. Your numbers excluding that are going to be way underestimating things pretty quickly without that. I'd be interested in what those numbers look like assuming reasonable guesses about robot workforce being part of a feedback cycle.

Because the debate is never set in terms of "better education" vs "teachers preference". It’s "give more money for teachers so they can give better education". When there’s a tradeoff, it’s usually of the form "better top-performers education" or "more equality on education" ? I don’t see teachers unions arguing that school vouchers are good for children but should still be outlawed. I see teachers unions arguing that school vouchers are bad for children, and They’re The Experts, so outlaw them.

I don’t expect that tactic to work when the alternative is a literal superintelligence.

I'll also give at least £5k if tax deductibility is set up in the UK.

The doctors' cartel which enriches its members at the expense of patient welfare is backed by the force of the state, and I expect few to support its abolition. The teachers' unions are similarly popular.  In what sense do you believe "democratic consensus" has answered these question the way you think they have?

My new point is that I don't think optimal agents will exist when we lose all control

(btw I also realized I don't strictly require it be 'optimal', but at least close enough to it / 'strongly superhuman enough' for us to not be able to tell the difference. I originally used the 'optimal' wording trying to find some other definition apart from 'super-human')

it is also plausible to me that life-caring beings first lose control to much narrower programs^[1] or moderately superhuman unaligned agents totally outcompeting them economically, or something.

also, if near-optimal agents are possible, a 'multipolar AI-driven but still normal-ish' scenario seems to continue at most until a strong enough agent is created? (e.g. that could be what a race is towards).

(maybe after 'loss of control to weaker AI' scenarios, those weaker AIs also keep making better agents afterwards, but i'm not sure about that, because they could be myopic [? · GW] and in some stable pattern/equilibrium)

1. ^{^}

   (e.g. the 'going out with a whimper' part of this post [LW · GW])