LessWrong 2.0 Reader

I was just thinking about writing a post that overlaps with this, inspired by a recent Drexler post. I'll turn it into a comment.

Leopold Aschenbrenner's framing of a <i>drop-in remote worker</i> anthropomorphizes AI in a way that risks causing AI labs to make AIs more agenty than is optimal.

Anthropomorphizing AI is often productive. I use that framing a fair amount to convince myself to treat AIs as more capable than I'd expect if I thought of them as mere tools. I collaborate better when I think of the AI as a semi-equal entity.

But it feels important to be able to switch back and forth between the tool framing and the worker framing. Both framings have advantages and disadvantages. The ideal framing is likely somewhere in between that seems harder to articulate.

I see some risk that AI labs turning AIs into agents, when if they were less focused on replacing humans they might lean more toward Drexler's (safer) services model.

Please, AI labs, don't anthropomorphize AIs without carefully considering when that's an appropriate framing.

i.e. the question "what sort of community institutions are good to build?" is a timeless question. Why should we artificially limit our ability to reflect on that sort of thing during the Review, given that we set the Review up in an openended way that allows us to do that on the margin? 

Fwiw I disagree, I think the Review is deliberately openended. 

Yes there's a specific goal of find the top 50 posts, and to identify important timeless intellectual contributions. But, part of the whole point of the review (as I originally envisioned it) is also to help reflect in a more general sense on "what happened on LessWrong and what can we learn from it?".

I think rather than trying to say "no, don't reflect on particular things that don't fit the most central use case of the Review", it seems actively good to me to take advantage of the openended nature of it to think about less central things. We can learn timeless lessons from posts that weren't, themselves, particularly timeless. 

The point I’m trying to make here is a really obvious one. Like, suppose that Bob is a really great, top-percentile employee. But suppose that Bob’s roommate Alice is an obviously better employee than Bob along every possible axis. Clearly, Bob will still be able to get a well-paying job—the existence of Alice doesn’t prevent that, because the local economy can use more than one employee.

My most recent post on LessWrong (https://www.lesswrong.com/posts/yj2hyrcGMwpPooqfZ/a-proposal-for-iterated-interpretability-with-known), which is also my first post proposing a novel avenue for AI alignment research, took me a total of 30 minutes.

I think I agree with Feynman being a straight talker, but I just want to caution on inferences on Feynman from books about Feynman. See:

TLDW: Feynman didn't actually write any of the books that use his name, and his influence over them is pretty tenuous. (e.g. Surely Your Joking was written by a young friend of Feynman's, and the book wasn't written until like at least 10 years after the stories were originally told, allegedly)

You've probably thought of this and have reasons for and against it, but maybe some hotels (bedside) and restaurants (on tables) would be willing to take copies too? Seems much less likely that libraries though.

Probably for me, the main thing that helped was Yoda Timers. Then again, that was probably just a function of getting to practice it much more than anything else. Next up is probably TAPs.

I agree that even if the book turned out to be entirely accurate we should not assume that this case is representative of the average development project, but we could still learn from it. Many hours from highly trained and well-paid people are allocated to planning and evaluating such projects, which expenditure is ostensibly to ensure quality. Even looking at worst cases helps us see what sort of quality is or is not being ensured.

I agree that if an AI is incapable of competently scheming (i.e., alignment faking and sabotaging safety work without being caught), but is capable of massively accelerating safety work, then doing huge amounts of safety work with this AI is very promising.

(I put this aside in this post as I was trying to have a more narrow focus on how we'll update about scheming independent of how easily scheming will be handled and without talking about methods that don't currently exist.)

(The specific directions you mentioned of "fancy behavioral red teaming and interp" may not be that promising, but I think there are a moderate number of relatively empirical bets that look decently promising.)

It seems like the first AIs capable of massively accelerating safety work might also scheme pretty competently (it will depend on the architecture). However, we might be able to compensate with sufficient control measures such that the AI is forced to be very helpful (or is caught). Correspondingly, I'm excited about AI control.

(More generally, rapid takeoff might mean that we have to control AIs that are capable of competent scheming without having already obsoleted prior work.)

I'm reasonably optimistic about bootstrapping if the relevant AI company could afford several years of delay due to misalignment, was generally competent, and considered mitigating risk from scheming to be a top priority. You might be able to get away with less delay (especially if you heavily prep in advance). I don't really expect any of these to hold, at least across all the relevant AI companies and in short timelines.