LessWrong 2.0 Reader

Ari's work is on Arxiv here

For what it's worth I think this accurately conveys "Zac endorses the Lightcone fundraiser and has non-trivially donated", and dropping the word "unusually" would leave the sentence unobjectionable; alternatively maybe you could have dropped me from the list instead.

I just posted this because I didn't want people to assume that I'd donated >10% of my income when I hadn't :-)

It's going to depend on the "hacks". I think o3 is plausibly better described as "vast amounts of rl with an llm init" than "an llm with some rl applied".

(The idealized utility maximizer question mostly seems like a distraction that isn't a crux for the risk argument. Note that the expected utility you quoted is our utility, not the AI's.)

I'm not exactly sure about "what you want". It is not the case that you can exactly reconstruct most probability distributions you'll encounter in real life from their moments/ cumulants (hence the expansion is perturbative, not exact).

But in the interpretability/ field-theoretic model of wide NN's point of view, this is what you want (specifically, the fourth-order correction)

Yes, I actually thought about this a bit. It is definitely the case that the LC (or RLCT) in the SLT context is also exactly a (singular) stationary phase expansion. Unfortunately, the Fourier transform of a random variable, including a higher-dimensional one, really does have an isolated nondegenerate maximum at 0 (unless the support of your random variable is contained in a union of linear subspaces, which is kinda boring/ reducible to simpler contexts). Maybe if you think about some kind of small perturbation of a lower-dimensional system, you can get some components of the singular free energy expansion, but the expansion relevant here is really nonsingular. This is also the type signature of the expansion you see in most physical QFT systems, at least if they have a perturbative form (in which case, the free theory will in general be nondegenerate).

Continuing the example with sweets, I estimate my terminal goals to include both "not be ill e.g. with diabetes" and "eat tasty things".

That sounds basically right to me, which is why I put effort into learning (and teaching) to enjoy the right things. I'm pretty proud of the fact that both my little girls like "liver treats".

Technology and other progress has two general directions: a) more power for those who are able to wield it; b) increasing forgiveness, distance to failure. For some reason, I thought that b) was a given at least on average.

I think that's right, but also "more distance to failure" doesn't help so much if you use your newfangled automobile to cover that distance more quickly. It's easier to avoid failure, but also easier to fail. A gun makes it easier to defend yourself, and also requires you to grow up until you can make those calls correctly one hundred percent of the time. With great power comes great responsibility, and all that.

I'll take the car, and the gun, and the society that trusts people with cars and guns and other technologically enabled freedoms. But only because I think we can aspire to such responsibilities, and notice when they're not met. All the enabling with none of the sobering fear of recklessness isn't a combination I'm a fan of.

With respect to the "why do you believe this" question on my previous comment about promiscuity being statistically linked with marital dissatisfaction, I'm not very good at keeping citations on hand so I can't tell you which studies I've seen, but here's what chatgpt found for me when I asked for studies on the correlation.

https://www.jstor.org/stable/3600089
https://unews.utah.edu/u-researcher-more-sex-partners-before-marriage-doesnt-necessarily-lead-to-divorce/
https://ifstudies.org/blog/testing-common-theories-on-the-relationship-between-premarital-sex-and-marital-stability
https://www.proquest.com/openview/46b66af73b830380aca0e6fbc3b597e3/1

I don't actually lean that hard on the empirical regularity though, because such things are complicated and messy (e.g. the example I gave of a man with a relatively high partner count succeeding because he took an anti-promiscuous stance). The main reason I believe that pills don't remove all the costs of promiscuity is that I can see some of the causal factors at work and have experience actually working with them to help women land happy stable relationships.

Thanks for writing this! I've participated in some similar conversations and on balance, think that working in a lab is probably net good for most people assuming you have a reasonable amount of intellectual freedom (I've been consistently impressed by some papers coming out of Anthropic).

Still, one point made by Kaarel in a recent conversation seemed like an important update against working in a lab (and working on "close-to-the-metal" interpretability in general). Namely, I tend to not buy arguments by MIRI-adjacent people that "if we share our AI insights with the world then AGI will be developed significantly sooner". These were more reasonable when they were the only ones thinking seriously about AGI, but now it mostly seems that a capabilities researcher will (on the margin, and at the same skill level) contribute more to making AGI come soon than a safety researcher. But a counterpoint is that serious safety researchers "are trying to actually understand AI", which has a global orientation towards producing valuable new research results (something like people at the Manhattan project or Apollo program at the hight of these programs' quality), whereas a capabilities researcher is more driven by local market incentives. So there may be a real sense in which interpretability research, particularly of more practical types, is more dangerous, conditional on "globally new ideas" (like deep learning, transformers etc.) being needed for AGI. This was so far the most convincing argument for me against working on technical interpretability in general, and it might be complicated further by working in a big lab (as I said, it hasn't been enough to flip my opinion, but seems worth sharing)

Isn't it the other way around ?

If inner alignment is hard then general is bad because applying less selection pressure, i.e. more generally, more simplicity prior, means more daemons/gremlins

Let's focus on inner alignment. By instill you presumably mean train. What values get trained is ultimately a learning problem which in many cases (as long as one can formulate approximately a boltzmann distribution) comes down to a simplicity-accuracy tradeoff.

I guess im mostly thinking about the regime where AIs are more capable and general than humans.

It seems at first glance that the latter failure mode is more of a capability failure. Something one would expect to go away as AI truly surpasses humans. It doesnt seem core to the alignment problem to me.