LessWrong 2.0 Reader

"Game theoretic strengthen-the-tribe perspective" is a completely unpersuasive argument to me. The psychological unity of humankind [LW · GW] OTOH is persuasive when combined with the observation that this unitary psychology changes slowly enough that the human mind's capacity to understand the other human minds in the environment and manage the risks posed by them can keep up.

Creativity is RL, converting work into closing the generation-discrimination gap wherever it's found (or laboriously created by developing good taste). The resulting generations can be novelty-worthy.

Why doesn't lesswrong have a library, perhaps one that is curated by AI?

Well, arguably it does: https://www.lesswrong.com/library [? · GW]

There's an alignment-related problem, the problem of defining real objects. Relevant topics: environmental goals; task identification problem; "look where I'm pointing, not at my finger"; Eliciting Latent Knowledge [? · GW].

Another highly relevant post: The Pointers Problem [LW · GW].

A reasoning model depends on starting from a sufficient base model that captures the relevant considerations. Solving AIME is like winning at chess, except the rules of chess are trivial, and the rules of AIME are much harder. But the rules of AIME are still not that hard, it's using them to win that is hard.

In the real world, the rules get much harder than that, so it's unclear how far o1 can go if the base model doesn't get sufficiently better (at knowing the rules), and it's unclear how much better it needs to get. Plausibly it needs to get so good that o1-like post-training won't be needed for it to pursue long chains of reasoning on its own, as an emergent capability. (This includes the possibility that RL is still necessary in some other way, as an engine of optimization to get better at rules of the real world, that is to get better reward models.)

It sounds to me like you're saying that the intelligent agent will just disregard optimization of its utility function and instead investigate the possibility of an objective goal.

Yes, exactly.

The logic is similar to Pascal's wager. If objective goal exists, it is better to find and pursue it, than a fake goal. If objective goal does not exist, it is still better to make sure it does not exist before pursuing a fake goal. Do you see?

Thanks for having a read!

Do you expect abductive reasoning to be significantly different from deductive reasoning? If not, (and I put quite high weight on this,) then it seems like (Berglund, 2023) already tells us a lot about the cross-context abductive reasoning capabilities of LLMs. I.e. replicating their methodology wouldn't be very exciting. 

Berglund et. al. (2023) utilise a prompt containing a trigger keyword (the chatbot's name) for their experiments,

$\begin{array}{cc}{A}_1\to B_1& \\ A_2\to B_2& \\ A_1& \\ & \beta \end{array}$

where $\beta$ corresponds much more strongly to $B_1$ than $B_2$. In our experiments the As are the chatbot names, Bs are behaviour descriptions and $\beta$s are observed behaviours in line with the descriptions. My setup is:

$\begin{array}{cc}{A}_1\to B_1& \\ A_2\to B_2& \\ & \beta \\ A_1& \end{array}$

The key difference here is that  $\beta$ is much less specific/narrow since it corresponds with many Bs, some more strongly than others. This is therefore a (Bayesian) inference problem. I think it's intuitively much easier to forward reason given a narrow trigger to your knowledge base (eg. someone telling you to use the compound interest equation) than to figure out which parts of your knowledge base are relevant (you observe numbers 100, 105, 110.25 and asking which equation would allow you to compute the numbers).

Similarly, for the RL experiments (experiment 3) they use chatbot names as triggers and therefore there results are relevant to narrow backdoor triggers rather than more general reward hacking leveraging declarative facts in training data. 

I am slightly confused whether their reliance on narrow triggers is central to the difference between abductive and deductive reasoning. Part of the confusion is because (Beysian) inference itself seems like an abductive reasoning process. Popper and Stengel (2024) discuss this a little bit under page 3 and footnote 4 (process of realising relevance). 

One difference that I note here is that abductive reasoning is uncertain / ambiguous; maybe you could test whether the model also reduces its belief of competing hypotheses (c.f. 'explaining away'). 

Figures 1 and 4 attempt to show that. If the tendency to self-identify as the incorrect chatbot falls with increasing k and iteration than the models are doing this inference process right. In Figures 1 and 4, you can see that the tendency to incorrectly identify as a pangolin falls in all non-pangolin tasks (except for tiny spikes sometimes at iteration 1). 

Yes, I know; the following tab was already present at that time, is what I meant to communicate.

Twitter did use an algorithmic timeline before (e.g. tweets you might be interested in, tweets people you followed liked), it was just less algorithmic than the "for you" tab currently. The time when it was completely like the current "following" tab was many years ago.

This is also my interpretation of the rumors, assuming they are true, which I don't put much probability on.