LessWrong 2.0 Reader

'Local volume' should also give a kind of upper bound on the LLC defined at finite noise though, right? Since as I understand it, what you're referring to as the volume of a behavioral region here is the same thing we define via the behavioural LLC at finite noise scale in this paper? And that's always going to be bigger or equal to the LLC taken at the same point at the same finite noise scale.

Thank you, daijin, you have interesting ideas!

The library metaphor is a versatile tool it seems, the way I understand it:

My motivation is safety, static non-agentic AIs are by definition safe. We can expose the library to people instead of keeping it “in the head” of the librarian. Basically this way we can play around in the librarian’s “head”. Right now mostly AI interpretability researchers do it, not the whole humanity, not the casual users. 

I see a few ways AIs can work:

1. The current only way: “The librarian visits your brain.” Sounds spooky but this is what is essentially happening right now to a small extent when you prompt it and read the output (the output enters your brain). 
   “The librarian visits your brain.”

2. “The librarian visits and changes our world.” This is where we are heading with agentic AIs.

   3. New safe way: Let the user visit the librarian’s “brain” instead, make this “brain” more place-like. So instead of the agentic librarians intruding and changing our world/brains, we’ll intrude and change theirs, seeing the whole content of it and taking into our world and brain only what we want.
   
   I wrote more about this in the first half of this comment [EA(p) · GW(p)], if you’re interested

   Have a nice day!

I don't know what your views on self-driving cars are, but if you are like me you look at what Waymo is doing and you think "Yep, it's working decently well now, and they are scaling up fast, seems plausible that in a few years it'll be working even better and scaled to every major city. The dream of robotaxis will be a reality, at least in the cities of America."

The example of self-driving cars is actually the biggest one that anchors me to timelines of decades or more. A lot of people's impression after the 2007 DARPA Grand Challenge seemed to be something like "oh, we seem to know how to solve the problem in principle, now we just need a bit more engineering work to make it reliable and agentic in the real world". Then actually getting things to be as reliable as required for real agents took a lot longer. So past experience would imply that going from "we know in principle how to make something act intelligently and agentically" to "this is actually a reliable real-world agent" can easily take over a decade.

Another example is that going from the first in-principle demonstration of chain-of-thought to o1 took two years. That's much shorter than a decade but also a much simpler capability.

For general AI, I would expect the "we know how to solve things in principle" stage to at least be something like "can solve easy puzzles that a normal human can that the AI hasn't been explicitly trained on". Whereas with AI,  we're not even there yet. E.g. I tried giving GPT-4.5, DeepSeek R1, o3-mini, and Claude 3.7 with extended thinking a simple sliding square problem, and they all committed an illegal move at one stage or another.

And that's to say nothing about all the other capabilities that a truly general agent - say one capable of running a startup - would need, like better long-term memory, ability to formulate its own goals and prioritize between them in domains with no objective rules you could follow to guarantee success, etc.. Not only are we lacking convincing in-principle demonstrations of general intelligence within puzzle-like domains, we're also lacking in-principle demonstrations of these other key abilities.

How does the performance of this compare to the SGLD sampling approach used by Timaeus, or to bounding the volume by just calculating the low-lying parts of the Hessian eigenspectrum [LW · GW]? Or, to go even hackier and cheaper, just guessing the Hessian eigenspectrum with kfac-approximation by doing a PCA of the activations and gradients at every layer and counting the zero eigenvalues of those?

(For all of those approaches, I'd use the loss landscape/Hessian of the behavioural loss defined in section 2.2 of that last link, since you want to measure the volume of a behavioural region.)

Claude-powered version of Alexa

Does this mean Amazon is going to be Anthropic's big brother, the way that Microsoft is for OpenAI? 

Without verifiable details it's impossible to be sure if his claim is factually correct, or even grounded in consensus reality. He may have been pointing at a genuine potential conflict where he feels like he de-escalated something, or he may have been storytelling to flex his power (real or imagined) as a peacemaker. The countries could be real or imagined, the de-escalation could be real or imagined.

From the context, my guess is that regardless of the reality of these "two smaller nations", he was merely employing a plot device to tell Zelenskyy, "War is always bad (and you might be implicitly bad for allowing Ukraine to be involved in one). You should just give Russia what it wants so Russia stops hurting Ukraine with war." He doesn't need real nations for that persuasion tactic, and speculating on the identity of these possibly-fictional nations might be a distraction.

Interesting that he would suggest Russia's invasion of Ukraine should be resolved the same way as when a mugger demands your wallet. But that seems (to me) to be what he's suggesting here.

Maybe? We might still have consistent results within this narrow experimental setup, but it's not clear to me that it would generalize outside that setup [LW(p) · GW(p)].

Forced choice settings are commonly used in utility elicitation, e.g. in behavioral economics and related fields.

True. It's also a standard criticism of those studies that answers to those questions measure what a person would say in response to being asked those questions (or what they'd do within the context of whatever behavioral experiment has been set up), but not necessarily what they'd do in real life when there are many more contextual factors. Likewise, these questions might answer what an LLM with the default persona and system prompt might answer when prompted with only these questions, but don't necessarily tell us what it'd do when prompted to adopt a different persona, when its context window had been filled with a lot of other information, etc.. 

The paper does control a bit for framing effects by varying the order of the questions, and notes that different LLMs converge to the same kinds of answers in that kind of neutral default setup, but that doesn't control for things like "how would 10 000 tokens worth of discussion about this topic with an intellectually sophisticated user affect the answers", or "how would an LLM value things once a user had given it a system prompt making it act agentically in the pursuit of the user's goals and it had had some discussions with the user to clarify the interpretation of some of the goals". 

Some models like Claude 3.6 are a bit infamous for very quickly flipping all of their views into agreement with what it thinks the user's views are, for instance. Results like in the paper could reflect something like "given no other data, the models predict that the median person in their training data would have/prefer views like this" (where 'training data' is some combination of the base-model predictions and whatever RLHF etc. has been applied on top of that; it's a bit tricky to articulate who exactly the "median person" being predicted is, given that they're reacting to some combination of the person they're talking with, the people they've seen on the Internet, and the behavioral training they've gotten).

This reasoning would justify violating any social convention whatsoever. "Refusing to say 'please' and 'thank you' signals confidence and self-esteem".

Wrong. I distinguished between conventions that people have a reason to respond negatively to if you violate them (e.g. wearing a suit to the gym or swimming pool which is stupid since it will ruin both your exercise and your suit), and behaviors that just happen to be unusual but not intrinsically negative. Refusing to say 'please' and 'thank you' would fall squarely in the category that people would have a reason to feel negative about.

My understanding is that fedoras became a sign of cluelessness because they got associated with groups like pick-up artists, which is also an explicit reason to have a negative reaction to them.

Interesting outlook!

I might be thinking of it too much in terms of classical Bayesianism, so let me know if my question is confused and doesn't make sense:

Is this just a way to count other agents' predictions as evidence, or is there also a differnce in how a single agent predicts and behaves in a world with no other agents?

I'm familiar, in classical bayesianism, with the fact that other agents' predictions can influence my prediction. There's Aumann agreement theorem and so on. Seeing another agent predict X will increase the probability I assign X at least a little.

Is this framework expanding on that, or proposing something else?