LessWrong 2.0 Reader

Thanks for the riff!

Note, I wasn't sure how to convey it but in the version I wrote, I didn't mean it as a world where people have god-like powers. The only change intended was that it was a world where it was normal for six-year-olds to be able to think about multiple universes and understand what counts as advanced math for us, like Group Theory. There were a couple things I was thinking about:

1. I was musing on a possible solution to the measure problem that our universe is an actual hypothetical/mathematical object and there a finite number of actual hypotheticals such that having a copy of a universe would make no more sense than having a copy of a number. (The mathematical object only needs to be as real as we are within it.)
2. I was also asking if it would be possible to have a world where it was normal for six-year-olds to be that much better at math (and presumably get better as they grow up) in the same way that a six-year-old is that much better at conceptual math than a chimpanzee. Would it have to be creepy or could  they still be relatable?

Disclaimer: I'm not a Group Theorist and the LLM I asked said it would take ten plus years if ever for me to be able to derive the order of the Fischer–Griess monster group from first principles (but it's normal that the child could do this).

• A man being deeply respected and lauded by his fellow men, in a clearly authentic and lasting way, seems to be a big female turn-on. Way way way bigger effect size than physique best as I can tell.
  • …but the symmetric thing is not true! Women cheering on one of their own doesn't seem to make men want her more. (Maybe something else is analogous, the way female "weight lifting" is beautification?)

My guess at the analogous thing: women being kind/generous/loving seems to me like a thing many men have found attractive across times and cultures, and seems to me far more viable if a woman is embedded in a group who recognize her, tell her she is cared about and will be protected by a network of others, who in fact shield her from some kinds of conflict/exploitation, who help there be empathy for her daily cares and details to balance out the attentional flow of these she gives to others, etc.  So the group plays a support role in a woman being able to have/display the quality.

On one side: Humanoid robots have much more density of parts requiring more machine-time than cars, probably slowing things a bunch.

On the other, you mention assuming no speed up due to the robots building robot factories, but this seems like the dominant factor in the growth. Your numbers excluding that are going to be way underestimating things pretty quickly without that. I'd be interested in what those numbers look like assuming reasonable guesses about robot workforce being part of a feedback cycle.

Because the debate is never set in terms of "better education" vs "teachers preference". It’s "give more money for teachers so they can give better education". When there’s a tradeoff, it’s usually of the form "better top-performers education" or "more equality on education" ? I don’t see teachers unions arguing that school vouchers are good for children but should still be outlawed. I see teachers unions arguing that school vouchers are bad for children, and They’re The Experts, so outlaw them.

I don’t expect that tactic to work when the alternative is a literal superintelligence.

I'll also give at least £5k if tax deductibility is set up in the UK.

The doctors' cartel which enriches its members at the expense of patient welfare is backed by the force of the state, and I expect few to support its abolition. The teachers' unions are similarly popular.  In what sense do you believe "democratic consensus" has answered these question the way you think they have?

My new point is that I don't think optimal agents will exist when we lose all control

(btw I also realized I don't strictly require it be 'optimal', but at least close enough to it / 'strongly superhuman enough' for us to not be able to tell the difference. I originally used the 'optimal' wording trying to find some other definition apart from 'super-human')

it is also plausible to me that life-caring beings first lose control to much narrower programs^[1] or moderately superhuman unaligned agents totally outcompeting them economically, or something.

also, if near-optimal agents are possible, a 'multipolar AI-driven but still normal-ish' scenario seems to continue at most until a strong enough agent is created? (e.g. that could be what a race is towards).

(maybe after 'loss of control to weaker AI' scenarios, those weaker AIs also keep making better agents afterwards, but i'm not sure about that, because they could be myopic [? · GW] and in some stable pattern/equilibrium)

1. ^{^}

   (e.g. the 'going out with a whimper' part of this post [LW · GW])

I think the economic value of long-term agents will be greater than that of agents without such skills, similarly to how an employee who can effectively work towards long-term goals coherently is more economically valuable than an employe that can’t.

I think short-term agentic skills are a subset of long-term agentic skills.

I also object to putting long-term agentic coherence at odds with controllability and direct usefulness. As a thought experiment, imagine two employees: one known to be good at short-term work and one known to be good at long-term work. Who would be more useful? Who would you expect to stay on-task better (1)?

I fear the financial incentives are such that long-term coherent agents are inevitable and will be worked towards at speed.

(1) using “staying on task” as a stand in for controllability.

Sorry, I want to ask a question that seems out of topic.

In this post, you mention a paper named Towards Safe and Honest AI Agents with Neural Self-Other Overlap.

This paper submitted to ICLR 2025 at 27 Sept 2024 and NeurIPS 2024 Workshop SafeGenAi at 13 Oct 2024.

Does this behaviour count as dual submissions or is it normal?

While this post didn't yield a comprehensive theory of how fact finding works in neural networks, it's filled with small experimental results that I find useful for building out my own intuitions [LW · GW] around neural network computation.

I think that's speaks to how well these experiments are scoped out that even a set of not-globally-coherent findings yield useful information.