LessWrong 2.0 Reader

Not a doctor, but have you explored whether neurodiversity is part of the picture? What you're describing sounds similar to how adults with PDA describe their experience. 

Another helpful route might be to practice self-acceptance and stop blaming yourself for not being 'naturally' geared towards doing laundry. Once you do this you can focus on ways of solving the issue, e.g. paying someone to do the things you find hard. 

I claim that if you find someone who’s struggling to get out of bed, making groaning noises, and ask them the following question:

Hey, I have a question about your values. The thing you’re doing right now, staying in bed past your alarm, in order to be more comfortable at the expense of probably missing your train and having to walk to work in the cold rain … is this thing you’re doing in accordance with your values?

I bet the person says “no”. Yet, they’re still in fact doing that thing, which implies (tautologically) that they have some desire to do it—I mean, they’re not doing it “by accident”! So it’s conflicting desires, not conflicting values.

I don’t think your wooden log example is relevant. Insofar as different values are conflicting, that conflict has already long ago been resolved, and the resolution is: the action which best accords with the person’s values, in this instance, is to get up. And yet, they’re still horizontal.

Another example: if someone says “I want to act in accordance with my values” or “I don’t always act in accordance with my values”, we recognize these as two substantive claims. The first is not a tautology, and the second is not a self-contradiction.

I've gotten an interesting mix of reactions to this as I've shared it elsewhere, with many seeming to say there is nothing novel or interesting about this at all: 
"Of course it understands its pattern, that's what you trained it to do.  It's trivial to generalize this to be able to explain it." 

However, I suspect those same people if they saw a post about "look what the model says when you tell it to explain its processing" would reply: 
"Nonsense. They have no ability to describe why they say anything.  Clearly they're just hallucinating up a narrative based on how LLMs generally operate.

If it wasn't just dumb luck (which I suspect it wasn't, given the number of times the model got the answer completely correct), then it is combining a few skills or understandings, and not violating any token-prediction basics at the granular level.  But I do think it just opens up avenues to either - be less dismissive generally when models talk about what they are doing internally - or - figure out how to train a model to be more meta-aware generally.

And yes, I would be curious to see what was happening in the activation space as well.  Especially since this was difficult to replicate with simpler patterns. 

Yeah I can see how Scott's quote can be interpreted that way. I think the people listed would usually be more careful with their words. But also, Scott isn't necessarily claiming what you say he is. Everyone agrees that when you prompt a base model to act agentically, it can kinda do so. This can happen during RLHF. Properties of this behaviour will be absorbed from pretraining data, including moral systems. I don't know how Scott is imagining this, but it needn't be an inner homunculi that has consistent goals.

I think the thread below with Daniel and Evan and Ryan is good clarification of what people historically believed (which doesn't put 0 probability on 'inner homunculi', but also didn't consider it close to being the most likely way that scheming consequentialist agents could be created, which is what Alex is referring to^[1]). E.g. Ajeya is clear at the beginning of this post [LW · GW] that the training setup she's considering isn't the same as pretraining on a myopic prediction objective.

1. ^{^}

   When he says 'I think there's a ton of wasted/ungrounded work around "avoiding schemers", talking about that as though we have strong reasons to expect such entities.'

Perhaps I have listened to too much Georgist propaganda, but it seems to me that land is pretty important even today. But maybe this is mostly true for land in cities? Not sure. I would like to see some statistics about what was the typical consequence for being an aristocrat during the industrial revolution. I can imagine that each of the following was true for someone, the question is how many:

• you get killed
• you survive, but your land is taken away
• you sell the land trying to join the new economy, but you suck at the new economy, so you lose the money
• you keep the land, but it is at an unimportant place, the rent is too low
• you keep the land, you get a big rent, you are a successful modern rentier

I'd like to see a pie chart of this. Maybe some other options, if I forgot them.

Definitely similar, and nice design! I hadn't seen that before, unfortunately. How did the models do on it?

Unfortunately I don't remember much details :/

My vague memories:

• Nothing impressive, but with CoT you sometimes see examples that clearly show some useful skills
• You often see reasoning like "I now know f(1) = 2, f(2) = 4. So maybe that multiplies by two. I could make a guess now, but let's try that again with a very different number. Whats f(57)?"
• Or "I know f(1) = 1, f(2) = 1, f(50) = 2, f(70) = 2, so maybe that function assigns 1 below some threshold and 2 above that threshold. Let's look for the threshold using binary search" (and uses the search correctly)
• There's very little strategy in terms of "is this a good moment to make a guess?" - sometimes models gather like 15 cases that confirm their initial hypothesis. Maybe that's bad prompting though.

But note that I think my eval is significantly easier than yours.

Also have you seen 'Connecting the Dots'? That tests a few things, but one of them is whether, after fine-tuning on (x, f(x)) pairs, the model can articulate what f is, and compute the inverse. Really interesting paper.

I even wrote (a small part of) it : ) I'm glad you find it interesting!

Thanks for the comments! 

I'm especially keen to explore bottlenecks (e.g. another suggestion I saw is that to reach 1 billion a year would require 10x current global lithium production to supply the batteries.)

A factor of 2 for increased difficultly due to processing intensity seems reasonable, and I should have thrown it in. (Though my estimates were to an order of magnitude so this probably won't change the bottom line, and on the other side, many robots will weigh <100kg and some will be non-humanoid.)

I suspect that it's a tooling and scaffolding issue and that e.g. claude-3-5-sonnet-20241022 can get at least 70% on the full set of 60 with decent prompting and tooling.

By "tooling and scaffolding" I mean something along the lines of

• Naming the lists that the model submits (e.g. "round 7 list 2")
• A tool where the LLM can submit a named hypothesis in the form of a python function which takes a list and returns a boolean and check whether the results of that function on all submitted lists from previous rounds match the answers it's seen so far
• Seeing the round number on every round
• Dropping everything except the seen lists and non-falsified named hypotheses in the context each round (this is more of a practical thing to avoid using absurd volumes of tokens, but I imagine it wouldn't hurt performance too much)

I'll probably play around with it a bit tomorrow.

but his takes were probably a little more predictably unwelcome in this venue

I hope he doesn't feel his takes are unwelcome here. I think they're empirically very welcome. His posts seem to have a roughly similar level of controversy and popularity as e.g. so8res. I'm pretty sad that he largely stopped engaging with lesswrong.

There's definitely value to being (rudely?) shaken out of lazy habits of thinking [...] and I think Alex has a knack for (at least sometimes correctly) calling out others' confusion or equivocation.

Yeah I agree, that's why I like to read Alex's takes.

Observation: for a variable that is positive and often close to zero (e.g. approximately lognormal), the L infty minimizer would be half of the maximum, which is isomorphic to literally just using the maximum. This seems like the quantitative version of focusing terrorism discussions on 9/11 and focusing wealth inequaility discussions on Elon Musk and so on.