LessWrong 2.0 Reader

Probably for me, the main thing that helped was Yoda Timers. Then again, that was probably just a function of getting to practice it much more than anything else. Next up is probably TAPs.

I agree that even if the book turned out to be entirely accurate we should not assume that this case is representative of the average development project, but we could still learn from it. Many hours from highly trained and well-paid people are allocated to planning and evaluating such projects, which expenditure is ostensibly to ensure quality. Even looking at worst cases helps us see what sort of quality is or is not being ensured.

I agree that if an AI is incapable of competently scheming (i.e., alignment faking and sabotaging safety work without being caught), but is capable of massively accelerating safety work, then doing huge amounts of safety work with this AI is very promising.

(I put this aside in this post as I was trying to have a more narrow focus on how we'll update about scheming independent of how easily scheming will be handled and without talking about methods that don't currently exist.)

(The specific directions you mentioned of "fancy behavioral red teaming and interp" may not be that promising, but I think there are a moderate number of relatively empirical bets that look decently promising.)

It seems like the first AIs capable of massively accelerating safety work might also scheme pretty competently (it will depend on the architecture). However, we might be able to compensate with sufficient control measures such that the AI is forced to be very helpful (or is caught).

I'm reasonably optimistic about bootstrapping if the relevant AI company could afford several years of delay due to misalignment, was generally competent, and considered mitigating risk from scheming to be a top priority. You might be able to get away with less delay (especially if you heavily prep in advance). I don't really expect any of these to hold, at least across all the relevant AI companies and in short timelines.

I think it affects both, since alignment difficulty determines both the probability that the AI will have values that cause it to take over, as well as the expected badness of those values conditional on it taking over.

Adding onto this, I would broadly say that the Lightcone team did not update that in-person infrastructure was unimportant, even while our first attempt was an investment into an ecosystem we later came to regret investing in.

Also here's a quote of mine from the OP:

If I came up with an idea right now for what abstraction I'd prefer, it'd be something like an ongoing festival with lots of events and workshops and retreats for different audiences and different sorts of goals, with perhaps a small office for independent alignment researchers, rather than an office space that has a medium-size set of people you're committed to supporting long-term.

I'd say that this is a pretty close description of a key change that we made, that changes my models of the value of the space quite a lot.

That, incidentally, implies that human labor will retain a well-paying niche—just as less-skilled labor today can still get jobs despite more-skilled labor also existing.

Less skilled labor has a well-paying niche today?

For the record, all of Lightcone's community posts and updates from 2023 do not seem to me to be at all good fits for the review, as they're mostly not trying to teach general lessons, and are kinda inside-baseball / navel-gazing, which is not what the annual review is about.

Okay. So from what I understand, you want to use a magnetic effect observed in plasma as a primary energy source.

Generally, a source of energy works by taking a fuel which contains energy and turning it into a less energetic waste product. For example, carbon and oxygen can be burned to form CO2. Or one can split some uranium nucleus into two fragments which are more stable and reap the energy difference as heat.

Likewise, a wind turbine will consume some of the kinetic energy of the air, and a solar panel will take energy from photons. For a fusion reactor, you gain energy because you turn lighter nuclei (hydrogen isotopes or helium-3) into helium-4, which is extraordinarily stable.

Thus, my simple question: where does the energy for your invention come from? "The plasma" is not a sufficient answer, because on Earth we generally encounter plasma rarely for us to exploit, in fusion reactor designs, it is generated painstakingly at a huge energy cost.

Something goes into your reactor, and something comes out of it. If it is the same, then it can hardly have expended energy in your reaction.

I just donated $3000, for reasons based on Ryan's comment [LW(p) · GW(p)]. 

But humans don't seem to optimize for reward all that often!

You might be interested in an earlier discussion on whether "humans are a hot mess": https://www.lesswrong.com/posts/SQfcNuzPWscEj4X5E/the-hot-mess-theory-of-ai-misalignment-more-intelligent [LW · GW] https://www.lesswrong.com/posts/izSwxS4p53JgJpEZa/notes-on-the-hot-mess-theory-of-ai-misalignment [LW · GW]