LessWrong 2.0 Reader

Shutting Down all Competing AI Projects is not Actually a Pivotal Act

This seems like an excellent title to me.

shane legg had 2028 median back in 2008, see e.g. https://e-discoveryteam.com/2023/11/17/shane-leggs-vision-agi-is-likely-by-2028-as-soon-as-we-overcome-ais-senior-moments/

Appreciate actually the overall take (although not sure how many would not have found most of it simply common sense anyway), but: A bit more caution with the stats would have been great

• Just-about-significant $\ne$ 'insignificant and basta'. While you say the paper shows up to incl. 27 there's no 'effect' (and concluding on causality is anyway problematic here, see below), all data provided in the graph you show and in the table of the paper suggest BMI 27 has a significant or nearly significant (on 95%..) association with death even in this study. You may instead want to say the factor is not huge (or small compared to much larger BMI variations), although the all-cause point-estimate mortality factor of roughly 1.06 for already that BMI is arguably not trivial at all: give me something that, as central-albeit-imprecise estimate, increases my all-cause mortality by 6%, and I hope you'd accept if I politely refused, explaining you propose something that seems quite harmful, maybe even in those outcomes where I don't exactly die from it.
• Non-significance $\ne$ No-Effect. Even abstracting from the fact that the BMI 27 data is actually significant or just about so: "not significant" reduction in deaths on BMI 18-27 in the study wouldn't mean as you claim "will not extend your life". It means, the study was too imprecise to be exactly 95% or more sure that there's a relationship. Without strong prior to the contrary, the point estimate, or even any value to the upper CI bound, cannot be excluded at all as describing the 'real' relationship.
• Stats lesson 0: Association $\ne$ Causality. The paper seems to purposely talk about association, mentioning some major potential issues with interfering unobserved factors already in the Abstract, and there are certainly a ton of confounding factors that may well bias the results (it would seem rather unnatural to expect people who work towards having a supposedly-healthy BMI to behave not differently on average in any other health-releveant way than people who may be working less towards such BMI).

I agree that if overhang is a relevant consideration for pauses, then it's also a relevant consideration for RSPs. My previous question was: Do you see the same people invoking overhang as an argument against pauses and also talking about RSPs as though they are not also impacted?

Maybe you're not saying that there are people taking that position, but rather that those who invoke overhang as an argument against pauses don't seem to be equally vocal against RSPs (if not necessarily in favor of them either). I can think of a couple of separate reasons this could be the case:

1. To the extent I think a pause is bad (for example, because of overhang), I might still be more motivated to prioritize arguing against "unconditional pause" than "maybe pause in the future", even if the argument could apply to both. This is especially true if I consider the prospect of an unconditional pause a legitimate, near-term threat. 
2. If I think a pause introduces a high, additional risk, and I think the base level of risk is low, it seems clear that I should not introduce that high risk. But if I get new evidence that there is an immediate, even-higher risk, which a pause could help mitigate, I should be willing to roll the dice on the pause, which now comes with a net reduction in risk.

(2) isn't a very reassuring position, but it does suggest that "immediate pause bad because overhang" and "RSPs good [in spite of overhang]" are logically compatible.

So what we have is not "the CoT remains constant but the answers vary". Instead, the finding is: "a CoT created in response to a biased prompt changes in order to match the bias, without mentioning the bias."

Thanks for bringing this up.

I think I was trying to shove this under the rug by saying "approximately constant" and "~constant," but that doesn't really make sense, since of course the CoTs actually vary dramatically in response to the biasing features. (They have to, in order to justify different final answers.)

To be honest, I wrote the account of Turpin et al in the post very hastily, because I was really mainly interested in talking about the other paper.  My main reaction to Turpin et al was (and still is) "I don't know what you expected, but this behavior seems totally unsurprising, given its ubiquity among humans (and hence in the pretraining distribution), and the fact that you didn't indicate to the model that it wasn't supposed to do it in this case (e.g. by spelling that out in the prompt)."

But yeah, that summary I wrote of Turpin et al is pretty confused – when I get a chance I'll edit the post to add a note about this.

Thinking about it more now, I don't think it makes sense to say the two papers discussed in the post were both "testing the causal diagram (question -> CoT -> answer)" – at least not in the same sense.

As presented, that diagram is ambiguous, because it's not clear whether nodes like "CoT" are referring to literal strings of text in the context window, or to something involving the semantic meaning of those strings of text, like "the aspects of the problem that the CoT explicitly mentions."

With Lanham et al, if we take the "literal strings of text" reading, then there's a precise sense in which the paper is testing the casual diagram.

In the "literal strings" reading, only arrows going from left-to-right in the context window are possible (because of the LLM's causal masking). This rules out e.g. "answer -> CoT," and indeed almost uniquely identifies the diagram: the only non-trivial question remaining is whether there's an additional arrow "question -> answer," or whether the "question"-"answer" relationship is mediated wholly through "CoT."  Testing whether this arrow is present is exactly what Lanham et al did. (And they found that it was present, and thus rejected the diagram shown in the post, as I said originally.)

By contrast, Turpin et al are not really testing the literal-strings reading of the diagram at all.  Their question is not "which parts of the context window affect which others?" but "which pieces of information affects which others?", where the "information" we're talking about can include things like "whatever was explicitly mentioned in the CoT."

I think there is perhaps a sense in which Turpin et al are testing a version of the diagram where the nodes are read more "intuitively," so that "answer" means "the value that the answer takes on, irrespective of when in the context window the LLM settles upon that value," and "CoT" means "the considerations presented in the CoT text, and the act of writing/thinking-through those considerations."  That is, they are testing a sort of (idealized, naive?) picture where the model starts out the CoT not having any idea of the answer, and then brings up all the considerations it can think of that might affect the answer as it writes the CoT, with the value of the answer arising entirely from this process.

But I don't want to push this too far – perhaps the papers really are "doing the same thing" in some sense, but even if so, this observation probably confuses matters more than it clarifies them.

As for the more important higher-level questions about the kind of faithfulness we want and/or expect from powerful models... I find stuff like Turpin et al less worrying than you do.

First, as I noted earlier: the kinds of biased reasoning explored in Turpin et al are ubiquitous among humans (and thus the pretraining distribution), and when humans do them, they basically never mention factors analogous to the biasing factors.

When a human produces an argument in writing – even a good argument – the process that happened was very often something like:

1. (Half-consciously at best, and usually not verbalized even in one's inner monologue) I need to make a convincing argument that P is true. This is emotionally important for some particular reason (personal, political, etc.)
2. (More consciously now, verbalized internally) Hmm, what sorts of arguments could be evidenced for P?  [Thinks through several of them and considers them critically, eventually finding one that seems to work well.]
3. (Out loud) P is true because [here they provide a cleaned-up version of the "argument that seemed to work well," crafted to be clearer than it was in their mind at the moment they first hit upon it, perhaps with some extraneous complications pruned away or the like].

Witness the way that long internet arguments tend to go, for example.  How both sides keep coming back, again and again, bearing fresh new arguments for P (on one side) and arguments against P (on the other).  How the dispute, taken as a whole, might provide the reader with many interesting observations and ideas about object-level truth-value of P, and yet never touch on the curious fact that these observations/ideas are parceled out to the disputants in a very particular way, with all the stuff that weighs in favor P spoken by one of the two voices, and all the stuff that weighs against P spoken by the other.

And how it would, in fact, be very weird to mention that stuff explicitly.  Like, imagine someone in an internet argument starting out a comment with the literal words: "Yeah, so, reading your reply, I'm now afraid that people will think you've not only proven that ~P, but proven it in a clever way that makes me look dumb. I can't let that happen. So, I must argue for P, in such a way that evades your clever critique, and which is itself very clever, dispelling any impression that you are the smarter of the two. Hmm, what sorts of arguments fit that description? Let's think step by step..."

Indeed, you can see an example of this earlier in this very comment!  Consider how hard I tried to rescue the notion that Turpin et al were "testing the causal diagram" in some sense, consider the contortions I twisted myself into trying to get there.  Even if the things I said there were correct, I would probably not have produced them if I hadn't felt a need to make my original post seem less confused than it might otherwise seem in light of your comment.  And yet I didn't say this outright, at the time, above; of course I didn't; no one ever does^[1].

So, it's not surprising that LLMs do this by default.  (What would be surprising is we found, somehow, that they didn't.)

They are producing text that is natural, in a human sense, and that text will inherit qualities that are typical of humans except as otherwise specified in the prompt and/or in the HHH finetuning process.  If we don't specify what we want, we get the human default^[2], and the human default is "unfaithful" in the sense of Turpin et al.

But we... can just specify what we want?  Or try to?  This is what I'm most curious about as an easy follow-up to work like Turpin et al: to what extent can we get LLM assistants to spell out the unspoken drivers of their decisions if we just ask them to, in the prompt?

(The devil is in the details, of course: "just ask" could take various forms, and things might get complicated if few-shots are needed, and we might worry about whether we're just playing whack-a-mole with the hidden drivers that we just so happen to already know about. But one could work through all of these complications, in a research project on the topic, if one had decided to undertake such a project.)

A second, related reason I'm not too worried involves the sort of argumentation that happens in CoTs, and how we're seeing this evolve over time.

What one might call "classic CoT" typically involves the model producing a relatively brief, straight-to-the-point argument, the sort of pared-down object for public consumption that a human might produce in "step 3" of the 1-2-3- process listed above.  (All the CoTs in Turpin et al look like this.)

And all else being equal, we'd expect such CoTs to look like the products of all-too-human 1-2-3 motivated reasoning.

But if you look at o1 CoTs, they don't look like this. They verbalize much more of the "step 2" and even "step 1" stuff, the stuff that a human would ordinarily keep inside their own head and not say out loud.

And if we view o1 as an indication of what the pressure to increase capabilities is doing to CoT^[3], that seems like an encouraging sign.  It would mean that models are going to talk more explicitly about the underlying drivers of their behavior than humans naturally do when communicating in writing, simply because this helps them perform better. (Which makes sense – humans benefit from their own interior monologues, after all.)

(Last note: I'm curious how the voice modality interacts with all this, since humans speaking out loud in the moment often do not have time to do careful "step 2" preparation, and this makes naturally-occurring speech data importantly different from naturally-occurring text data. I don't have any particular thoughts about this, just wanted to mention it.)

1. ^{^}

   In case you're curious, I didn't contrive that earlier stuff about the causal diagram for the sake of making this meta point later. I wrote it all out "naively," and only realized after the fact that it could be put to an amusing use in this later section.

2. ^{^}

   Some of the Turpin et al experiments involved few-shots with their own CoTs, which "specifies what we want" in the CoT to some extent, and hence complicates the picture. However, the authors also ran zero-shot versions of these, and found broadly similar trends there IIRC.

3. ^{^}

   It might not be, of course. Maybe OpenAI actively tried to get o1 to verbalize more of the step 1/2 stuff for interpretability/safety reasons.

How about this? Edited the post:

I wanted to run a fun “Make 100% eye contact or get sent to jail” party. But I didn’t want to force people to make eye contact — I wanted everyone to become genuinely comfortable. How?

Consider: Eye contact is effortless when you’re not emotionally blocked. If you have trouble making eye contact, you are probably held back by emotional blocks. These blocks are probably to help you stay safe, so forcing eye contact could even be harmful!

So what if I helped attendees notice and integrate their blocks?

I'm intrigued. Let me know if you'd like to chat

I think we’re in agreement on everything.

Excellent. Sorry for thinking you were saying something you weren't!

still not have an answer to whether it’s spinning clockwise or counterclockwise

More simply (and quite possibly true), Nobuyuki Kayahara rendered it spinning either clockwise or counterclockwise, lost the source, and has since forgotten which way it was going.

Huh, are there examples of right leaning stuff they stopped funding? That's new to me

The big issue for the MNM effect isn't that it won't exist or not cause a strong response, but rather the issue that plagues pause-type policies will apply 100-fold to the MNM effect:

The issue is that algorithmic advances could make the policy quite useless fast, and in the MNM effect, there will plausibly be only 1 or 2 OOMs left to superintelligent AI through primarily scaling.

People have debated on LW about the value of algorithmic advances and whether they are necessary for AGI and ASI, but the debate is a bit irrelevant for pause/ban purposes, because there definitely have been useful algorithmic progress before, and under a pause scenario, there will be a lot of incentives to get more use out of algorithmic progress like AI search, so I consider the policies generated by the MNM effect to only last several years at most, and also we will almost certainly have super-persuasive AIs that blunt the strength of any laws passed on AI, so I don't expect the MNM effect to be helpful at all, and plausibly see it as a very harmful effect for AI safety.

Link below:

https://yellow-apartment-148.notion.site/AI-Search-The-Bitter-er-Lesson-44c11acd27294f4495c3de778cd09c8d