LessWrong 2.0 Reader

"Seems mistaken to think that the way you use a model is what determines whether or not it’s an agent. It’s surely determined by how you train it?" 
---> Nah, pre training, fine tuning, scaffolding and especially RL seem like they all affect it. Currently scaffolding only gets you shitty agents, but it at least sorta works 

a worthy goal for alignment work: help superintelligence develop a sense of self that is substrate independent.

a physically bounded self will be competitive with humans and clear us away. an interconnected substrate independent self may be loving AND be evolutionarily selected for, i.e. superintelligence to survive long-term

(vague memory from the in person discussions we had last year, might be inaccurate):

jeremy!2023: If you're expecting AI to be capable enough to "accelerate alignment research" significantly, it'll need to be a full-blown agent that learns stuff. And that'll be enough to create alignment problems because data-efficient long-horizon generalization is not something we can do.

joshc!2023: No way, all you need is AI with stereotyped skills. Imagine how fast we could do interp experiments if we had AIs that were good at writing code but dumb in other ways!

...

joshc!now:

Training AI agents so they can improve their beliefs (e.g. do research) as well as the best humans can.

Seems like the reasoning behind your conclusions has changed a lot since we talked, but the conclusions haven't changed much? 

If you were an AI: Negative reward, probably a bad belief updating process.

Literally macroeconomics 101. Trade surpluses aren't shipping goods for free. There is a whole balance of payments to consider. I'm shocked EY could get that so wrong, surprised that lsusr is so ready to agree, and confused because surely I missed something huge here, right?

The example is really helpful for me getting a concrete understanding of what it looks like to satisfy Trust without Reflection, and why that goes along with deferring to someone else for decisions - but I don't see what this example of Alice has to do with locality. It looks like the only relevant propositions are whether it rains tomorrow, and what Alice's credences are, and there don't seem to be any propositions we don't defer to her on.

The original tweets seem at least partially tongue-in-cheek? Trade has lots of benefits [LW · GW] that don't depend on the net balance. If Country A buys $10B of goods from Country B and sells $9B of other goods to country B, that is $19B of positive-sum transactions between individual entities in each country, presumably with all sorts of positive externalities and implications about your economy.

The fact that the net flow is $1B in one direction or the other just doesn't matter too much. Having a large trade surplus (or large trade deficit) is a proxy for generally doing lots of trading and industry, which will tend to correlate with a lot of other things that made or will make you wealthy. But it would be weird if a country could get rich by solely by running a trade surplus, while somehow avoiding reaping any of the other usual benefits of trading. "Paying other countries to discern your peoples' ability to produce" is plausibly a benefit that you get from a trade surplus even if you try hard to avoid all the others, though.

You should show your calculation or your code, including all the data and parameter choices. Otherwise I can't evaluate this.

The code is pretty complicated and not something I'd expect a non-expert (even a very smart one) to be able to quickly check over; it's not just a 100 line python script. (Or even a very smart expert for that matter, more like anyone who wasn't already familiar with our particular codebase.) We'll likely open source it at some point in the future, possibly soon, but that's not decided yet. Our finemapping (inferring causal effects) procedure produces ~identical results to the software from the paper I linked above when run on the same test data (though we handle some additional things like variable per-SNP sample sizes and missing SNPs which that finemapper doesn't handle, which is why we didn't just use it).

The parameter choices which determine the prior over SNP effects are the number of causal SNPs (which we set to 20,000) and the SNP heritability of the phenotype (which we set to 0.19, as per the GWAS we used). The erroneous effect size adjustment was done at the end to convert from the effect sizes of the GWAS phenotype (low reliability IQ test) to the effect sizes corresponding to the phenotype we care about (high reliability IQ test).

We want to publish a more detailed write up of our methods soon(ish), but it's going to be a fair bit of work so don't expect it overnight.

It's natural in your position to scrutinize low estimates but not high ones.

Yep, fair enough. I've noticed myself doing this sometimes and I want to cut it out. That said, I don't think small-ish predictable overestimates to the effect sizes are going to change the qualitative picture, since with good enough data and a few hundred to a thousand edits we can boost predicted IQ by >6 SD even with much more pessimistic assumptions, which probably isn't even safe to do (I'm not sure I expect additivity to hold that far). I'm much more worried about basic problems with our modelling assumptions, e.g. the assumption of sparse causal SNPs with additive effects and no interactions (e.g. what if rare haplotypes are deleterious due to interactions that don't show up in GWAS since those combinations are rare?).

It's just very hard for me to believe there aren't huge gains possible from genetic engineering. It goes against everything we've seen from a millenia of animal breeding. It goes against the estimates we have for the fraction of variance that's linear for all these highly polygenic traits. It goes against data we've seen from statisitcal outliers like Shawn Bradley, who shows up as a 4.6 standard deviation outlier in graphs of height:

Do I buy that things will get noisier around the tails, and that we might not be able to push very far outside the +5 SD mark or so? Sure. That seems unlikely, but plausible.

But the idea that you're only going to be able to push traits by 2-3 standard deviations with gene editing before your predictor breaks down seems quite unlikely.

Maybe you've seen some evidence I haven't in which case I would like to know why I should be more skeptical. But I haven't seen such evidence so far.

Oh yeah my bad, I didn't notice that the second date was Sep instead of Feb

Our hiring this round is a small fraction of our overall team size, so this is really just correcting a minor imbalance, and shouldn't be taken as reflective of some big strategy. I'm guessing we'll go back to hiring a mix of the two around mid-2025.