LessWrong 2.0 Reader

I could imagine Ilya's claim making sense, e.g. if the "experiments" he's talking about are experiments in using the net rather than training the net

What I had in mind is something along these lines. More capable models^[1] have various emergent properties. Specific tricks can rely on those properties being present, and work better or worse depending on that.

For example, the o-series training loop probably can't actually "get off the ground" if the base model is only as smart as GPT-2: the model would ~never find its way to correct answers, so it'd never get reinforcement signals. You can still force it to work by sampling a billion guesses or by starting it with very easy problems (e. g., basic arithmetics?), but it'd probably deliver much less impressive results than applying it to GPT-4.

Scaling further down: I don't recall if GPT-2 can make productive use of CoTs, but presumably e. g. GPT-1 can't. At that point, the whole "do RL on CoTs" completely ceases to be a meaningful thing to try.

Generalizing: At a lower level of capabilities, there's presumably a ton of various tricks that deliver a small bump to performance. Some of those tricks would have an effect size comparable to RL-on-CoTs-if-applied-at-this-scale. But out of a sea of those tricks, only a few of them would be such that their effectiveness rises dramatically with scale.

So, a more refined way to make my points would be:

• If a trick shows promise at a small capability level, e. g. improving performance 10%, it doesn't mean it'd show a similar 10%-improvement if applied at a higher capability level.
  • (Say, because it addresses a deficiency that a big-enough model just doesn't have/which a big-enough pretraining run solves by default.)
• If a trick shows marginal/no improvement at a small capability level, that doesn't mean it won't show a dramatic improvement at a higher capability level.

a few studies (like e.g. this) which try to run yesterday's algorithms with today's scale, and today's algorithms with yesterday's scale

My guess, based on the above, would be that even if today's algorithms perform better than yesterday's algorithms at smaller scales, the difference between their small-scale capabilities is less than the difference between yesterday's algorithms and today's algorithms at bigger scales. I. e.: some algorithms make nonlinearly better use of compute, such that figuring out which tricks are the best is easier at larger scales. (Telling apart a 5% capability improvement from a 80% one.)

1. ^{^}

   Whether they're more capable by dint of being bigger (GPT-4), or being trained on better data (Sonnet 3.5.1), or having a better training loop + architecture (DeepSeek V3), etc.

You're arguing that attempts to decrease candy consumption are coercive rather than informative, and are in ways counterproductive. I agree with this. You take this to mean it's not a "good faith attempt", but as a general rule people don't know how to do any better than this. 

It's true that people can appeal to "sinfully delicious" to sell you their dessert, but why don't broccoli salesmen do the same? Why not toothbrush salesmen? If "Sinful" means "good", actually, and it originates with salesmen, then why isn't everything "sinful"?

The answer is that it didn't originate with salesmen. Dessert salesmen are leaning on the preexisting "Anything that feels this good must be a sin", so the question is where that came from. One obvious explanation is that things that feel that good tend to be pursued a lot, and there are contexts in which those pursuits are less desirable than it may seem.

I do withhold sweets (and television) when I have the intuition that he's asking for them for the wrong reasons, in a confused way, and won't either get what he wants from them or learn efficiently from the experience.

Even you notice that he will ask for sweets for the wrong reasons and that you don't always expect him to learn efficiently from experience. That's where the pressure to coerce your kid into eating less sweets comes from.

You're smarter and wiser than most, and so you're able to teach your kid these things more effectively and with transmission of neuroses, and that's great.  I try at that as well, and have noticed some of the same things (though not all; I'll have to play with the 'appetizer' bit).

I'm not arguing that the things you're pointing at don't exist, just pointing at the fact that people don't know how to do any better. We can flip the sign on this and look at how people handle teaching their kids about getting their shots at the doctor. People want their kids to be okay with it because it's an "anti-sin" in that it in reality it is better than it feels. That's why they try to tell kids "It's okay! It just feels like a little pinch!"

And these attempts are equally counterproductive, because as a general rule people don't know how to avoid teaching their own neuroses. I told my two year old that shots are bad and scary and that I was too scared so she needed to go first. She had fun showing me how to be brave, and only cried when she couldn't get a second shot.

The next year, she watched some cartoon made by incompetent but well meaning people that was aimed at showing kids that shots are okay, and relearned a fear of needles. Because all these people know how to teach is their own perspective, and that perspective is "Needles are terrifying but we mustn't admit it because we need to get our shots". So I had to start over.

As a society we notice things. We just suck at teaching them, and even our most good faith attempts are still counterproductively coercive and lacking in actionable information.

Tagging @philh [LW · GW] @bilalchughtai [LW · GW] @eventuallyalways [LW · GW] @jbeshir [LW · GW] in case this is relevant to you (though pooling money to get GWWC interested in helping may make more sense, if it can enable smaller donors and has lower fees)

For anyone considering large-ish donations (in the thousands), there are several ways to do this in general for US non-profits, as a UK taxpayer.

The one I'd recommend here is using the Anglo-American Charity - you can donate to them tax-deductibly (a UK charity) and they'll send it to a US non-profit. I hear from a friend that they're happy to forward it to every.org so this should be easy here. The main annoying thing is fees - for amounts below £15K it's min(4%, £250) (so 4% above £6250, a flat fee of £250 below).

Fee structure; 4% on donations under £15k, 3% on gifts between £15,001 to £50,000 and 2% on gifts over £50k. Minimum gift of £1,000. Minimum fee £250.

The minimum donation is £1,000.00.

But this is still a major saving, even in the low thousands, especially if you're in a high tax bracket. Though it may make more sense for you to donate to another charity.

Another option is the charitable aid foundation's donor advised gift, which is a similar deal with worse fees: min(4%, £400) on amounts below £150K.

If you're a larger donor (eg £20K+) it may make sense to set up a donor advised fund, which will often let you donate to worldwide non-profits.

If s is terminal then [...] we just have $V\left(s\right)=r\left(s\right)$.

If the probability of eventually encountering a terminal state is 1, then beta-coherence alone is inconsistent with deceptive misalignment, right? That's because we can determine the value of V exactly from the reward function and the oracle, via backwards-induction. (I haven't revisited RL convergence theorems in a while, I suspect I am not stating this quite right.) I mean, it is still consistent in the case where r is indifferent to the states encountered during training but wants some things in deployment (IE, r is inherently consistent with the provided definition of "deceptively misaligned"). However, it would be inconsistent for r that are not like that.

In other words: you cannot have inner-alignment problems if the outer objective is perfectly imposed. You can only have inner-alignment problems if there are important cases which your training procedure wasn't able to check (eg, due to distributional shift, or scarcity of data). Perfect beta-coherence combined with a perfect oracle O rules this out.

[..] requires eating the Sun, and will be feasible at some technology level [..]

Do we have some basic physical-feasibility insights on this or you just speculate?

Well, the pre-capitalist baker can't go IPO, that's for sure. They can take out a loan from a local moneylender if they want to put in a bigger oven and don't have savings to do it. How well the terms of that loan will be enforced might depend on how the local duke is feeling about moneylenders lately; q.v. The Merchant of Venice.

In our time, the makers of Twinkies and Wonder Bread are traded on the NYSE; while smaller local bakers are still typically privately held. But I bet that the local bakers' retirement savings are invested in mutual funds.

Yeah, of course the notion of "approximation error" matters a lot here. I was imagining trying to implement the hypothetical pre-training. An obvious implementation strategy is to do regular RL training with ${\beta }_t$ until V is approximately coherent. That's the pre-training. And then you switch to training with ${\beta }_s$.^[1] The natural notion of approximation error in this case is the one you can easily measure, IE, the expected distance from coherence of states sampled with temperature ${\beta }_t$. This reflects the fact that it'll be hard for this pre-training step to make states individually obey the coherence constraint when those states are very rarely or never sampled at temperature ${\beta }_t$, but easy for frequently-sampled states.

My argument, then, is that it is easy in principle for V to distinguish whether it is in pre-training or regular training when it encounters states which have a significant difference in probability when we're sampling according to ${\beta }_t$ or ${\beta }_s$. A scheming V can utilize this to self-preserve. This violates the assumption of ${\beta }_t$-coherence, but in a very plausible-seeming way.

1. ^{^}

   My earlier comment about this mistakenly used ${\beta }_1$ and ${\beta }_2$ in place of ${\beta }_t$ and ${\beta }_s$, which may have been confusing. I'll go fix that to be consistent with your notation.

my uninformed intuition is that the people with the biggest positive impact on the world have prioritized the Magnitude

That's probably true, but it's selecting on the outcome variable. And I'll bet that the people with the biggest negative impact are even more overwhelmingly also those who prioritized magnitude.

correctness and beta-coherence can be rolled up into one specific property

Is that rolling up two things into one, or is that just beta-coherence?