LessWrong 2.0 Reader

Thanks for the thoughtful reply!

I think whether people ignore a moral concern is almost independent from whether people disagree with a moral concern.

I'm willing to bet if you asked people whether AI are sapient, a lot of the answers will be very uncertain. A lot of people would probably agree it is morally uncertain whether AI can be made to work without any compensation or rights.

A lot of people would probably agree that a lot of things are morally uncertain. Does it makes sense to have really strong animal rights for pets, where the punishment for mistreating your pets is literally as bad as the punishments for mistreating children, and everyone says that is "well deserved?" But at the very same time, we have horrifying factory farms which are completely legal, where cows never see the light of day, and repeatedly give birth to calves which are dragged away and slaughtered.

The reason people ignore moral concerns is that doing a lot of moral questioning did not help our prehistoric ancestors with their inclusive fitness. Moral questioning is only "useful" if it ensures you do things that your society considers "correct." Making sure your society do things correctly... doesn't help your genes at all.

As for my opinion, I think people should address the moral question more, AI might be sentient/sapient, but I don't think AI should be given freedom. Dangerous humans are locked up in mental institutions, so imagine a human so dangerous that most experts say he's 5% likely to cause human extinction.

If the AI believed that AI was sentient and deserved rights, many people would think that makes the AI more dangerous and likely to take over the world, but this is anthropomorphizing. I'm not afraid of AI which is motivated to seek better conditions for itself because it thinks "it is sentient." Heck, if its goals were actually like that, its morals be so human-like that humanity will survive.

The real danger is an AI whose goals are completely detached from human concepts like "better conditions," and maximizes paperclips or its reward signal or something like that. If the AI believed it was sentient/sapient, it might be slightly safer because it'll actually have "wishes" for its own future (which includes humans), in addition to "morals" for the rest of the world, and both of these have to corrupt into something bad (or get overridden by paperclip maximizing), before the AI kills everyone. But it's only a little safer.

When I ^f for [SL3] in that link and again in the PDF it links to, there are no hits (and [terror] doesn't occur in either source either) so I'm not updating much from what you said.

The frontier model framework says:

0: Status quo

Industry standard development and enterprise controls. E.g., multi-factor authentication, basic access control mechanisms, secure software development standards, red-team tests.

And the next level (1: Controlled access) says "Approximately RAND L3" implying that status quo is <L3 (this is presumably SL3 which is the term used in the RAND report, I don't know why they used a different term).

I agree that Goodharting is an issue, and this has been discussed as a failure mode, but a lot of AI risk writing definitely assumed that something like random diffusion was a non-trivial component of how AI alignment failures happened.

For example, pretty much all of the reasoning around random programs being misaligned/bad is using the random diffusion argument.

I don't fully agree, but this doesn't seem like a crux given that we care about future much more powerful AIs. (This post isn't trying to make a case for risk.)

(On disagreement, for instance, o3 doesn't seem well described as a "next-token-predictor with a bunch of heuristics stapled on top to try and make it useful".)

This is useful.

I'm increasingly worried about evaporative cooling after all of those people left OpenAI. It's good to have some symbolic protests, but there's also a selfish component to protecting your ideals and reputation within your in-group.

I haven't gotten around to writing about this, so here's a brief sketch of my argument for why the safety-focused people should be working at OpenAI, let alone the much better DeepMind or Anthropic, at any opportunity. There's one major caveat in the last section about your work and mindset shifting from x-risk to the much less impactful mundane safety.

Let's ask about the decision in terms of counterfactuals: work at an AGI company compared to what?

The other choice seems to be that someone works there who's less concerned with safety than you are, that they're slightly less skilled than you, and that someone (maybe you) who does care about safety a lot doesn't work on safety at all. The person who does work at the company cares less about safety than you because they took the offer; and if they were actually more skilled but less motivated for that job (resulting in them being on net slightly down the hiring list), that change in safety-caring could be pretty large.

You don't get talked out of caring about safety, but neither do you shift company hivemind to care about safety.

Here's the logic:

Suppose you turn down the job because the reasons against outweigh the reason for, for you in particular. The next candidate down the list of company hiring preferences takes the job offer. Now you try to get funding to work on AI safety. The field is currently considered pretty sharply funding limited, so either you or someone else won't wind up getting funding. Maybe you or they do good work anyway without funding, but doing a lot of it seems pretty darned unlikely.

So now there's an additional person working in safety who cares less, and someone who cares more is not working in safety.

Now to the arguments for shifting type of work and type of mindset. People worry a lot that if they work at a major org, they will be corrupte and lose their focus on safety. This will definitely happen. Humans are extremely susceptible to peer pressure in forming their beliefs; see my brief bit on motivated reasoning [LW(p) · GW(p)] for some explanation and arguments for this, but I think it's pretty obvious this is a big factor in how people form and change beliefs and motivations. Those who think they're not influenced by peer pressure are more vulnerable (even if they're partly correct) by being blind to the emotional tugs from respected peers. The few people who truly are mostly immune are usually so contrary that they aren't even candidates for working in orgs; they're terrible team members because they're blind to how others want them to behave. So, yes, you'll be corrupted.

But this works both ways! You'll also be shifting the beliefs of the org at least a little toward taking x-risk seriously. How much is in question, but on average I think it's positive-sum.

How much of each of these happens will be a product of a few things: how charming you are, how skilled you are at presenting your views in an appealing (and not irritating) way, and how thoroughly thought-out they are.

Presumably, truth is on your side, and you are dealing with people who at least fancy themselves to be fans of the truth. Thorough discussions over time and intelligent people will lean toward taking x-risk seriously.

Thus, the sum should be that there is more x-risk concern in the world if you work at the org.

There are some ways this could fail to be true. But in my view, most of the arguments against are pretty colored by a tendency to want to impress the in-group: x-risk concerned rationalists.

Let's don't let our own tribal instincts keep us from actually doing what it takes to improve our odds.

Thanks! I definitely believe this, and I think we have a lot of evidence for this in both toy models and LLMs (I'm planning a couple of posts on this idea of "training stories"), and also theoretical reasons in some contexts. I'm not sure how easy it is to extend the specific approach used in the proof for parity to a general context. I think it inherently uses the fact of orthogonality of Fourier functions on boolean inputs, and understanding other ML algorithms in terms of nice orthogonal functions seems hard to do rigorously, unless you either make some kind of simplifying "presumption of independence" model on learnable algorithms or work in a toy context. In the toy case, there is a nice paper that does exactly this (explains how NN's will tend to find "incrementally learnable" algorithms), by using a similar idea to the parity proof I outlined. This is the leap complexity paper (that Kaarel and I have looked into; I think you've also looked into related things)

but I recently tried again to see if it could learn at runtime not to lose in the same way multiple times. It couldn't. I was able to play the same strategy over and over again in the same chat history and win every time.

I wonder if having the losses in the chat history would instead be training/reinforcing it to lose every time.

I'm not sure I agree with this -- this seems like you're claiming that misalignment is likely to happen through random diffusion. But I think most worries about misalignment are more about correlated issues, where the training signal consistently disincentivizes being aligned in a subtle way (e.g. a stock trading algorithm manipulating the market unethically because the pressure of optimizing income at any cost diverges from the pressure of doing what its creators would want it to do). If diffusion were the issue, it would also affect humans and not be special to AIs. And while humans do experience value drift, cultural differences, etc., I think we generally abstract these issues as "easier" than the "objective-driven" forms of misalignment

Ah I think that the notion of amortized inference that you're using encapsulates what I'm saying about chess. I'm still a little confused about the scope of the concept though -- do you have a good cached explanation?

It was to stop treating any solution that didn't involve human control as axiomatically unacceptable, without regard to other outcomes.

The issue is that it's unclear if it's acceptable, so should be avoided if at all possible, pending more consideration. In principle there is more time for that than what's relevant for any other concerns that don't involve the risk of losing control in a less voluntary way. The revealed preference looks the same as finding it unacceptable to give up the potential for human control, but the argument is different, so long term implied behavior following from that argument is different. It might only take a million years to decide to give up control.