LessWrong 2.0 Reader

Claude-powered version of Alexa

Does this mean Amazon is going to be Anthropic's big brother, the way that Microsoft is for OpenAI? 

Without verifiable details it's impossible to be sure if his claim is factually correct, or even grounded in consensus reality. He may have been pointing at a genuine potential conflict where he feels like he de-escalated something, or he may have been storytelling to flex his power (real or imagined) as a peacemaker. The countries could be real or imagined, the de-escalation could be real or imagined.

From the context, my guess is that regardless of the reality of these "two smaller nations", he was merely employing a plot device to tell Zelenskyy, "War is always bad (and you might be implicitly bad for allowing Ukraine to be involved in one). You should just give Russia what it wants so Russia stops hurting Ukraine with war." He doesn't need real nations for that persuasion tactic, and speculating on the identity of these possibly-fictional nations might be a distraction.

Interesting that he would suggest Russia's invasion of Ukraine should be resolved the same way as when a mugger demands your wallet. But that seems (to me) to be what he's suggesting here.

Maybe? We might still have consistent results within this narrow experimental setup, but it's not clear to me that it would generalize outside that setup [LW(p) · GW(p)].

Forced choice settings are commonly used in utility elicitation, e.g. in behavioral economics and related fields.

True. It's also a standard criticism of those studies that answers to those questions measure what a person would say in response to being asked those questions (or what they'd do within the context of whatever behavioral experiment has been set up), but not necessarily what they'd do in real life when there are many more contextual factors. Likewise, these questions might answer what an LLM with the default persona and system prompt might answer when prompted with only these questions, but don't necessarily tell us what it'd do when prompted to adopt a different persona, when its context window had been filled with a lot of other information, etc.. 

The paper does control a bit for framing effects by varying the order of the questions, and notes that different LLMs converge to the same kinds of answers in that kind of neutral default setup, but that doesn't control for things like "how would 10 000 tokens worth of discussion about this topic with an intellectually sophisticated user affect the answers", or "how would an LLM value things once a user had given it a system prompt making it act agentically in the pursuit of the user's goals and it had had some discussions with the user to clarify the interpretation of some of the goals". 

Some models like Claude 3.6 are a bit infamous for very quickly flipping all of their views into agreement with what it thinks the user's views are, for instance. Results like in the paper could reflect something like "given no other data, the models predict that the median person in their training data would have/prefer views like this" (where 'training data' is some combination of the base-model predictions and whatever RLHF etc. has been applied on top of that; it's a bit tricky to articulate who exactly the "median person" being predicted is, given that they're reacting to some combination of the person they're talking with, the people they've seen on the Internet, and the behavioral training they've gotten).

This reasoning would justify violating any social convention whatsoever. "Refusing to say 'please' and 'thank you' signals confidence and self-esteem".

Wrong. I distinguished between conventions that people have a reason to respond negatively to if you violate them (e.g. wearing a suit to the gym or swimming pool which is stupid since it will ruin both your exercise and your suit), and behaviors that just happen to be unusual but not intrinsically negative. Refusing to say 'please' and 'thank you' would fall squarely in the category that people would have a reason to feel negative about.

My understanding is that fedoras became a sign of cluelessness because they got associated with groups like pick-up artists, which is also an explicit reason to have a negative reaction to them.

Interesting outlook!

I might be thinking of it too much in terms of classical Bayesianism, so let me know if my question is confused and doesn't make sense:

Is this just a way to count other agents' predictions as evidence, or is there also a differnce in how a single agent predicts and behaves in a world with no other agents?

I'm familiar, in classical bayesianism, with the fact that other agents' predictions can influence my prediction. There's Aumann agreement theorem and so on. Seeing another agent predict X will increase the probability I assign X at least a little.

Is this framework expanding on that, or proposing something else?

See also here: https://www.lesswrong.com/posts/AcTEiu5wYDgrbmXow/open-problems-in-emergent-misalignment [LW · GW]

Yes, that's why it's compromise - nobody will totally like it. But if Earth is going to exist for trillions of years, it will radically change too.

Wouldn’t the Sun also change color in that scenario?

$\frac{\partial F\left(e_{\alpha }\right)}{\partial \alpha }=\frac{\partial F\left(e_{\alpha }\right)}{\partial e_{\alpha }}\frac{\partial e_{\alpha }}{\partial \alpha }$

$=\frac{\partial F\left(e_{\alpha }\right)}{\partial e_{\alpha }}\frac{\partial [e_{\text{clean}}+\alpha \times (e_{\text{corr}}-e_{\text{clean}}\left)\right]}{\partial \alpha }$

$=\frac{\partial F\left(e_{\alpha }\right)}{\partial e_{\alpha }}{e}_{\text{corr}}-e_{\text{clean}}$

Set $\alpha =0$, ie. $e_{\alpha }=e_{clean}$

$=e_{\text{corr}}-e_{\text{clean}}\frac{\partial F\left(e_{\text{clean}}\right)}{\partial e_{\text{clean}}}$

It looks like some parentheses are missing here?