LessWrong 2.0 Reader

Suppose you had an identical twin with identical genes and, until very recently, an identical history.  From the perspective of anyone else, you're similar enough to be interchangeable with each other.  But from your perspective, the twin would be a different person.  

The brain is you, full stop.  It isn't running a computer program; its hardware and software are inseparable and developed together over the course of your life.  In other words, the hardware/software distinction doesn't apply to brains.

Whether one is an accelerationist, Pauser, or an advocate of some nuanced middle path, the prospects/goals of everyone are harmed if the discourse-landscape becomes politicized/polarized.
...
I just continue to think that any mention, literally at all, of ideology or party is courting discourse-disaster for all, again no matter what specific policy one is advocating for.
...
Like a bug stuck in a glue trap, it places yet another limb into the glue in a vain attempt to push itself free.

I would agree in a world where the proverbial bug hasn't already made any contact with the glue trap, but this very thing has clearly already been happening for almost a year in a troubling direction. The political left has been fairly casually 'Everything-Bagel-izing' AI safety, largely in smuggling in social progressivism that has little to do with the core existential risks, and the right, as a result, is increasingly coming to view AI safety as something approximating 'woke BS stifling rapid innovation.' The fly is already a bit stuck. 

The point we are trying to drive home here is precisely what you're also pointing at: avoiding an AI-induced catastrophe is obviously not a partisan goal [LW · GW]. We are watching people in DC slowly lose sight of this critical fact. This is why we're attempting to explain here why basic AI x-risk concerns are genuinely important regardless of one's ideological leanings. ie, genuinely important to left-leaning and right-leaning people alike. Seems like very few people have explicitly spelled out the latter case, though, which is why we thought it would be worthwhile to do so here.

By "control", I mean AI Control [? · GW]: approaches aiming to ensure safety and benefit from AI systems, even if they are goal-directed and are actively trying to subvert your control measures.

AI control stops working once AIs are sufficiently capable (and likely don't work for all possible deployments that might eventually be otherwise desirable), but there could be other approaches that work at that point. In particular aligning systems.

The main hope I think about is something like:

• Use control until AIs are capable enough that if we trusted them, we could obsolete top human scientists and experts.
• Use our controlled AI labor to do the work needed to make systems which are capable enough, trustworthy enough (via alignment), and philosophically competent enough that we can safely hand things off to them. (There might be some intermediate states to get to here.)
• Have these systems which totally obsolete us figure out what to do, including figuring out how to aligning more powerful systems as needed.

We discuss our hopes more in this post [LW · GW].

(Discussion continues here on EA Forum. [EA(p) · GW(p)])

Embarrassingly, that was a semi-unintended reaction — I would bet a small amount against that statement if someone gave me a resolution method, but am not motivated to figure one out, and realized this a second after making it — that I hadn't figured out how to remove by the time you made that comment. Sorry.

You can still inspect the underlying CoT to know if you have big issues.

It is worth noting that the baseline proposal was already incentivizing the Face to be deceptive to the extent to which this proposal is important (the whole hope was to allow for hiding deception). So, I agree that my proposed change trades off "less deceptive CoT" for "more deceptive Face" relative to the baseline proposal, but this feels like this is mostly a quantitative difference?

(My guess is that you have this concern with the baseline protocol and thus don't like moving further in this direction.)

It could be that the baseline proposal doesn't make sense because shaping with process based feedback on the CoT is so useful that it is worth taking the risk of incentivizing unfaithful CoT.

make the system overall much harder to shape in ways you want (because it is actively hiding things from you so your RL is less accurate)

I thought the point was that we wouldn't train against the CoT basically at all with the baseline proposal? (Similar to o1.)

Oh I see, you mean that the observation is weak evidence for the median model relative to a model in which the most competent researchers mostly determine memeticity, because higher median usually means higher tails. I think you're right, good catch.

@Nick_Tarleton [LW · GW] How much do you want to bet, and what resolution method do you have in mind?

Use of repeated data was first demonstrated in the 2022 Galactica paper (Figure 6 and Section 5.1), at 2e23 FLOPs but without a scaling law analysis that compares with unique data or checks what happens for different numbers of repeats that add up to the same number of tokens-with-repetition. The May 2023 paper does systematic experiments with up to 1e22 FLOPs datapoints (Figure 4).

So that's what I called "tiny experiments". When I say that it wasn't demonstrated at scale, I mean 1e25+ FLOPs, which is true for essentially all research literature^[1]. Anchoring to this kind of scale (and being properly suspicious of results several orders of magnitude lower) is relevant because we are discussing the fate of 4e27 FLOPs runs.

If the means/medians are higher, the tails are also higher as well (usually). 

Norm(μ=115, σ=15) distribution will have a much lower proportion of data points above 150 than Norm(μ=130, σ=15). Same argument for other realistic distributions. So if all I know about fields A and B are that B has a much lower mean than A, by default I'll also assume B has a much lower 99th percentile than A, and much lower percentage of people above some "genius" cutoff.