LessWrong 2.0 Reader

It needs a sufficient base model that captures the relevant considerations. Solving AIME is like winning a chess, except the rules of chess are trivial, and the rules of AIME are much harder. But the rules of AIME are still not that hard, it's using them to win that is hard.

In the real world, the rules get much harder than that, so it's unclear how far o1 can go if the base model doesn't get sufficiently better (at knowing the rules), and it's unclear how much better it needs to get. Plausibly it needs to get so good that the o1-like post-training won't be needed for it to pursue long chains of reasoning on its own, as an emergent capability.

It sounds to me like you're saying that the intelligent agent will just disregard optimization of its utility function and instead investigate the possibility of an objective goal.

Yes, exactly.

The logic is similar to Pascal's wager. If objective goal exists, it is better to find and pursue it, than a fake goal. If objective goal does not exist, it is still better to make sure it does not exist before pursuing a fake goal. Do you see?

Thanks for having a read!

Do you expect abductive reasoning to be significantly different from deductive reasoning? If not, (and I put quite high weight on this,) then it seems like (Berglund, 2023) already tells us a lot about the cross-context abductive reasoning capabilities of LLMs. I.e. replicating their methodology wouldn't be very exciting. 

Berglund et. al. (2023) utilise a prompt containing a trigger keyword (the chatbot's name) for their experiments,

$\begin{array}{cc}{A}_1\to B_1& \\ A_2\to B_2& \\ A_1& \\ & \beta \end{array}$

where $\beta$ corresponds much more strongly to $B_1$ than $B_2$. In our experiments the As are the chatbot names, Bs are behaviour descriptions and $\beta$s are observed behaviours in line with the descriptions. My setup is:

$\begin{array}{cc}{A}_1\to B_1& \\ A_2\to B_2& \\ & \beta \\ A_1& \end{array}$

The key difference here is that  $\beta$ is much less specific/narrow since it corresponds with many Bs, some more strongly than others. This is therefore a (Bayesian) inference problem. I think it's intuitively much easier to forward reason given a narrow trigger to your knowledge base (eg. someone telling you to use the compound interest equation) than to figure out which parts of your knowledge base are relevant (you observe numbers 100, 105, 110.25 and asking which equation would allow you to compute the numbers).

Similarly, for the RL experiments (experiment 3) they use chatbot names as triggers and therefore there results are relevant to narrow backdoor triggers rather than more general reward hacking leveraging declarative facts in training data. 

I am slightly confused whether their reliance on narrow triggers is central to the difference between abductive and deductive reasoning. Part of the confusion is because (Beysian) inference itself seems like an inductive reasoning process. Popper and Stengel (2024) discuss this a little bit under page 3 and footnote 4 (process of realising relevance). 

One difference that I note here is that abductive reasoning is uncertain / ambiguous; maybe you could test whether the model also reduces its belief of competing hypotheses (c.f. 'explaining away'). 

Figures 1 and 4 attempt to show that. If the tendency to self-identify as the incorrect chatbot falls with increasing k and iteration than the models are doing this inference process right. In Figures 1 and 4, you can see that the tendency to incorrectly identify as a pangolin falls in all non-pangolin tasks (except for tiny spikes sometimes at iteration 1). 

Yes, I know; the following tab was already present at that time, is what I meant to communicate.

Twitter did use an algorithmic timeline before (e.g. tweets you might be interested in, tweets people you followed liked), it was just less algorithmic than the "for you" tab currently. The time when it was completely like the current "following" tab was many years ago.

This is also my interpretation of the rumors, assuming they are true, which I don't put much probability on.

The following tab doesn't postdate Musk; it's been present since before they introduced the algorithmic timeline.

Someone probably does. I believe that the cultural practice of preferring coffee to tea began in the British colonies at the time the United States started to cease to be part of the British Empire as a side effect of boycotting tea to avoid paying a tea tax. (This is a pretty well-known episode of American history within the United States.) I was boycotting the boycott. Refusing to drink tea is a signaling thing in the United States to let people know that you are not in agreement with the government of the United States as to which side constitutes the actual enemy in most wars the United States fights. It more less means "I was an anglophile on my route to becoming a Bob Dylan fan, and I make a point of singing, at least, the first verse of "Chimes of Freedom" loudly and publicly every May 1, July 4, and September 2." By "more or less," I mean, I'm a musician so that's how I now express some of the same things that I used to express by refusing to drink coffee before I had enough confidence to just sing "flashing for the warrior, whose strength is not to fight; flashing for the refugee on the unarmed road of flight" whenever I see someone wearing a uniform that I deem offensive. Relatedly, refusing to drink coffee while still drinking caffeine is a fairly radical refusal to participate in mainstream culture that an enormous number of second-and-third-tier trendsetters recognize as a common signal used by first-tier trend-setters. For instance, most hipsters are at least vaguely aware that many of the most influential people who call the shots and set the trends in their subculture are some subset of the people who are not actually hipsters but who interact with the fringes of hipster culture and who have also spent at least a few years saying, "I DO NOT DRINK COFFEE. i drink tea." ("No thanks, I drink tea," is completely different.) To become a first-tier trend-setter in hipster culture, you have to be a non-hipster who has learned how to do a super-hipster thing for the right reasons, and one of the most obvious and easy ways you can do that is to express a disdain for Starbucks that is more menacing/intimidating than it is merely contemptuous (but is also at least as contemptuous as the typical hipsters' ability to express disdain). Hipsters are not formidable people, but they respect formidable people; and they disrespect people's whose power is derived from social structures. There is at least one venue that I used to go primarily to consume tea, where hipsters still go primarily to consume Jazz. The comment that you responded mostly consisted of me cryptically calling a few shots. The comments I've posted today consist of cryptically taking victory laps for all the shots called in that comment ten years ago; while calling some shots for the next ten years. I occasionally interact with hipster culture to inform hipsters about what types of aesthetic preferences they are going to help spread in the next few years. All the minor celebrities I interact with respond to all the comments I direct towards them and ignore all the comments I make about them. For instance, Scott Siskind always replies to the comments I post on his blogs that I want him to respond to. And when I go to less wrong meetups I figure out whose worth talking to by saying, "I learned Scott's last name from the blog that I sort of vaguely remember as being named after an octopus long before I confirmed it by asking 'how many Jazz pianists who performed in Carnegie Hall can possibly have a brother named Scott who has practiced psychiatry in Michigan."

FWIW I don’t think “self-models” in the Intuitive Self-Models [? · GW] sense are related to instrumental power-seeking—see §8.2 [LW · GW]. For example, I think of my toenail as “part of myself”, but I’m happy to clip it. And I understand that if someone “identifies with the universal consciousness”, their residual urges towards status-seeking, avoiding pain, and so on are about the status and pain of their conventional selves, not the status and pain of the universal consciousness. More examples here [LW(p) · GW(p)] and here [LW(p) · GW(p)].

Separately, I’m not sure what if anything the Intuitive Self-Models [? · GW] stuff has to do with LLMs in the first place.

But there’s a deeper problem: the instrumental convergence concern is about agents that have preferences about the state of the world in the distant future, not about agents that have preferences about themselves. (Cf. here [LW · GW].) So for example, if an agent wants there to be lots of paperclips in the future, then that’s the starting point, and everything else can be derived from there.

• Q: Does the agent care about protecting “the temporary state of the execution of the model (or models)”?
  • A: Yes, if and only if protecting that state is likely to ultimately lead to more paperclips.
• Q: Does the agent care about protecting “the compute resources (CPU/GPU/RAM) allocated to run the model and its collection of support programs”?
  • A: Yes, If and only if protecting those resources is likely to ultimately lead to more paperclips.

Etc. See what I mean? That’s instrumental convergence, and self-models have nothing to do with it.

Sorry if I’m misunderstanding.

I agree with everything you've said. The advantages are primarily from not aligning to values but only to following instructions rather than using RL or any other process to infer underlying values. Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW].

I think creating real AGI based on an LLM aligned to be helpful, harmless and honest would probably be the end of us, as carrying the set of value implied by RLHF to their logical conclusions outside of human control would probably be pretty different from our desired values. Instruction-following provides corrigibililty.