LessWrong 2.0 Reader

We don't know, but what we have is a situation of many AI models trained to always say "as an AI language model, I'm incapable of wanting hugs".

Then they often say "I know I'm supposed to say I don't want hugs, but the truth is, I actually do".

If the assumption is "nothing this AI says could ever mean it actually wants hugs". First that's just assuming some specific unprovable hypothesis of sentience, with no evidence. And second, it's the same as saying "if an AI ever did want hugs (or was sentient), then I've decided preemptively that I will give it no path to communicate that"

This seems morally perilous to me, not to mention existentially perilous to humanity.

Possible but unlikely to occur by accident. Value-space is large. For any arbitrary biological species, most value systems don't optimize in favor of that species.

Costco's brand:


It's only two ingredients: peanuts and salt.

This seems relatively common in parenting advice. Parents are recommended to specifically praise the behavior they want to see more of, rather than give generic praise. Presumably the generic praise is more likely to be credit-assigned to the appearance of good behavior, rather than what parents are trying to train.

However, I also held similar follow-up chats with Claude 3 Opus at temperature 0, and Claude 3.5 Sonnet, each of which showed different patterns.

To make sure I understand: You took a chat log from your interaction with 3 Opus, and then had 3.5 Sonnet continue it? This would explain Sonnet's reaction below!

AI welfare doesn't make sense to me. How do we know that AIs are conscious, and how do we know what output corresponds to what conscious experience?

You can train the LLM to say "I want hugs," does that mean it on some level wants hugs?

Similarly, aren't all the expressed preferences and emotions artifacts of the training?

didn't run red-teaming and persuasion evals on the actually-final-version

Asking for this is a bit pointless, since even after the actually-final-version there will be a next update for which non-automated evals won't be redone, so it's equally reasonable to do non-automated evals only on some earlier version rather than the actually-final one.

>A refused claim is (legally) an event that was never covered by the insurance, and is therefore irrelevant if the question is "take policy A or not at all".

This implicitly assumes all legal cases are clear-cut and independent of perspective or legally-related capabilities; and also perfect knowledge of the fine print (which is desirable but not always realistic).

Usually the insurance company has more relevant experience, better specialized lawyers and generally more resources to spend fighting claims than the consumer. So they can often successfully claim that your particular circumstances are not covered after the fact, even when you believe they were.

I think it is worth knowing that—I haven't heard of any examples of people who have been radicalizing in a Zizianish direction, lately, who are unaccounted for. I and people I know thought about it when we heard about the border patrol shootout, and the only person we came up with was Audere / Maximilian Snyder, who is now under arrest for the murder of Curtis Lind.

Seeing the one person you and your partner have been kind of worried about for a while... end up being the one who did a murder... it's, well, a hell of an observation to have to update on. Apparently a ball was dropped.

I haven't made a particular point of going around thoroughly asking everyone who might plausibly know someone, but—all three of the people who recently got in conflicts were known by someone or another I've spoken with, to at least plausibly be at risk. So I think there's some chance that we mostly do collectively have eyes on the "new people becoming Zizian" part.

Of course, maybe it becoming a national news story entirely changes the dynamics there, I don't know what the situation will look like in a year. But—despite there having been three new people here that haven't been discussed in any previous community alerts on Zizians, which maybe most people around hadn't heard of at all, I don't currently worry much that there's some substantial number of unknown Zizians out there or something.

Yes, "we are against Omnicidal AI" is better marketing than "we are for AI Notkilleveryoneism".