LessWrong 2.0 Reader

https://www.nature.com/articles/d41586-024-03129-3 this is separate research.  It looks like this will happen, and it will come from somewhere other than the west.

Hot take: Prevalence of gender transition in male-majority fields is attempt to restore pronoun compression efficiency.

Communication efficiency is just that important.

You need just enough of them to distinguish subjects, but not so much that they lose their intuitive meaning. When cops are interviewing witnesses about a suspect, they’ll glom onto easily observable and distinguishing physical traits. Was the suspect a man or a woman? White or black? Tall or short?

Yeah.

Notably, basically all of the people I've known who have asked for neutral pronouns were also visibly of indeterminate gender (for instance, mid-transition), and over time their preferred/accepted pronouns always lined up with what a person would guess by looking at them.

This is generally the norm.

If you've encountered a lot of genderqueer people with non-obvious pronoun preferences, and they're pushy about them, that's probably a product of some kinda perverse selection process. In the least, whatever is causing those people to be annoying about that is not the queerness per se.

Sure. But I think you’re reading my argument to be stronger than I mean it to be. Which is partially my fault since I made my previous replies a bit too short, and for that I apologize.

What I’m doing here is presenting one particular simulation scenario that (to me) seems quite plausible within the realm of simulations. I’m not claiming that that one scenario dominates all others combined. But luckily that stronger claim is really not necessary to argue against Eliezer’s point: the weaker one suffices. Indeed, if the scenario I’m presenting is more than 4.5e-10 likely (and I do think it’s much more likely than that, probably by a few orders of magnitude), than it is more than enough to outweigh the practical cost of the ASI having to build a Dyson shell with a hole with the order of 4.5e-10 of it’s surface area.

Now, that scenario is (I claim) the most likely one, conditional of course on a simulation taking place to begin with. The other candidate simulation scenarios are various, and none of them seems particularly likely, though combined they might well outweigh this one in terms of mass probability, as I already acknowledged. But so what? Are you really claiming that the distribution of those other simulation scenarios is skewed enough to tilt the scales back to the doom side? It might be, but that’s a much harder argument to make. I’m approximately completely unsure, which seems way better than the 99%+ chance Eliezer seems to give to total doom. So I guess I’d count that as good news.

The Improbability Principle sounds close. The summary seems to suggest law of large numbers is one part of the pop science book, but admittedly some of the other parts ("probability lever") seem less relevant

If not (paused): utility=NormalUtilityFunction

If NormalUtilityFunction is not indifferent about being paused, then that has the usual problem where the utility function prefers worlds where its agent avoids being paused or shut down.

I'll try modifying your proposal to instead use UtilityFunction2, which assigns, to worlds where its agent ends up paused, utility equal to the average value of {worlds where the agent will not end up paused} in the agent's prior as it was just before updating on observing being paused. I think this agent would not try to specifically avoid action paths where it may end up paused. This doesn't mean it wouldn't still instrumentally converge and start disassembling things^[1], just that if you tried to pause it, it wouldn't have a direct incentive to specifically try to stop you, because if you succeed it still assigns utility equal to that of the average world in its prior where you didn't.

The latter function also leads something which may be unintuitive. Consider if this utility function were to evaluate two possible actions: on action (1), which has a 99% of being paused, and results in an average of 10 utility if not paused; and action (2), which has a 0% chance of being paused, and results in 5 utility if not paused. In this case, the stated function prefers action (1). (That may or may not be a problem; this is just an observation).

Also, note how "the average value of {worlds where the agent will not end up paused} in the agent's hypothesis space as it was just before updating on observing being paused" does depend on its own action policy. Specifically, the value of the average world where the agent does not end up paused could be higher or lower conditional on whether it tries to resist being paused (as resisting would add more worlds to the set of worlds whose values are being averaged).

1. ^{^}

   If this utility function is otherwise a classical physical-thing-maximizer

but we only need one person or group who we’d be somewhat confident would do alright in CEV. Plausibly there are at least a few eg MIRIers who would satisfy this.

Why do you think this, and how would you convince skeptics? And there are two separate issues here. One is how to know their CEV won't be corrupted relative to what their values really are or should be, and the other is how to know that their real/normative values are actually highly altruistic. It seems hard to know both of these, and perhaps even harder to persuade others who may be very distrustful of such person/group from the start.

Another is that even if we don’t die of AI, we get eaten by various moloch instead of being able to safely solve the necessary problems at whatever pace is necessary.

Would be interested in understanding your perspective on this better. I feel like aside from AI, our world is not being eaten by molochs very quickly, and I prefer something like stopping AI development and doing (voluntary and subsidized) embryo selection to increase human intelligence for a few generations, then letting the smarter humans decide what to do next. (Please contact me via PM if you want to have a chat about this.)

Mild spoilers for a contemporary science-fiction book, but the second half was a major plot point in 
 

Each of these carry assumptions about reality I'm not convinced a superintelligence would share. Though it may be able to find the answer in some cases.

We'd be just as likely for it to choose to preserve out of some sense of amusement or preservation.

To use the OP example: billionaire won't spare everyone 78 bucks, but will spend more on things he prefers. Some keep private zoos or other stuff which only purpose is anti boredom.

Making the intelligence like us won't eliminate the problem. There are plenty of fail states for humanity where it isn't extinct. But while we pave over ant colonies and actively hunt wild hogs as a nuisance, there are lots of human cultures that won't do the same to cats. I hope that isn't the best we can do, but it's probably better than extinction.

They (historically) had a large head start(up) on being scaling-pilled and various innovations like RLHF/instruction-tuning*, while avoiding pathologies of other organizations, and currently enjoy some incumbent advantages like what seems like far more compute access via MS than Anthropic gets through its more limited partnerships. There is, of course, no guarantee [LW(p) · GW(p)] any of that will last, and it generally seems like (even allowing for the unknown capabilities of GPT-5 and benefits from o1 and everything else under the hood) the OA advantage over everyone else has been steadily eroding since May 2020.

* which as much as I criticize the side-effects, have been crucial in democratizing LLM use for everybody who just wants to something done instead of learning the alien mindset of prompt-programming a base model