LessWrong 2.0 Reader

Made a slightly improved version.

Thanks for the feedback. I've added some headers to break it up.

Fundamentally, OP is making the case that Biorisk is an extremely helpful (but not exact) analogy for AI risk, in the sense that we can gain understanding by looking the ways it's analogous, and then get an even more nuanced understanding by analyzing the differences.

The point made seems to be more about it's place in the discourse than about the value of the analogy itself? (E.g. "The Biorisk analogy is over-used" would be less misleading then)

Would you please expand on how ai-plans.com addresses the question from the post above ... ?

Maybe let's try to make a smart counter-move and accelerate the development of for-profit AI Safety projects [...] ? With the obvious idea to pull some VC money, which is a different pool than AI safety philanthropic funds.

I took a look at ai-plans, but I have yet to find information about:

1. How does it work?
2. Who created it?
3. What is the motivation for building it?
4. What problem(s) will ai-plans help solve?
5. Who controls / curates / moderates it?
6. What is the process/algorithm for: curation? moderation? ranking?

I would suggest (i) answering these questions on the ai-plans website itself then (ii) adding links here.

Thanks for the question! This is not something we have included in our distribution, so I think our patching experiments aren't answering that question. If I would speculate though, I'd suggest

• The Prev Tok head 1.4 might "check" for a signature of "I am inside a function definition" (maybe a L0 head that attends to the def keyword. This would make it work only on B_def not B_dec
• Duplicate Tok head 1.2 might help the mover heads by suppressing their attention to repeated tokens. We observed this ("Duplicate Token Head 1.2 is helping Argument Movers"), but were not confident whether it is important. When doing ACDC we felt 1.2 wasn't actually too important (IIRC) but again this would depend on the distribution

In summary, I can think of a range of possible mechanisms how the model could achieve that, but our experiments don't test for that (because copying the 2nd token after B_dec would be equally bad for the clean and corrupted prompts).

I read an article about McDonalds a while ago. One of the things that powered their early success and growth was an extremely small menu. The slim menu meant that the kitchen had little variety enabling everything to be done faster and cheaper.

(https://en.wikipedia.org/wiki/History_of_McDonald%27s)

A giant company with 50 products could plausibly have 50 teams, each of which is focussed on making its product the best it can be. So the "focus" doesn't have to be lost at scale, its more like a bunch of different organisations under one umbrella.

I think it is pretty conclusive that sunscreen wearing will improve your skin complexion and smoothness in the long term (presumably the effect is not worth having if you never go outside in sunshine, obv). 

Whether it has enough material effect to do anything to your risk of dying from malignant melonoma (which is pretty rare anyway), is more of an open question. It might not be a worthwhile trade-off for everyone. 

I think the hidden motives are basically rationalisation, I have found myself singlethinking those motives in the past nowadays I just bring those reasons to the centre-stage and try to actually find whether they align with my commitments instead of motivated stopping. Sometimes I just corner my motivated reasoning (bottom line) so bad (since it's not that hard to just do expected consequentialist reasoning properly for day to day stuff) that instead of my brain trying to come up with better reasoning it just makes the idea of the impulsive action more salient, some urge along the lines of "think less and intuit/act more".  

Also I have personally used this concept of "intellectual masturbation" to divert discussions away from potentially philosophical bomb to more relevant topics it's much better to reduce the philosophical jargon in day to day conversations lol.

Reflecting on this more, I wrote in a discord server (then edited to post here):

I wasn't aware the concept of pivotal acts was entangled with the frame of formal inner+outer alignment as the only (or most feasible?) way to cause safe ASI. 

I don't expect any current human is capable of solving formal inner alignment,^[1] i.e. of devising a method to create ASI with any specified output selection policy. This is why I've been focusing on other approaches which I believe are more likely to succeed. I suspect that by default, we might mutually believe each others agendas to be doomed, which also makes discussion possibly more valuable (as at least one of us should make a drastic update).

[The post [LW · GW] linked above] introduced a different frame. For anyone interested in trying that discussion, I'd be curious what you think of it.

As one of the comments says,
I found myself coming back to this now, years later, and feeling like it is massively underrated. Idk, it seems like the concept of training stories is great and much better than e.g. "we have to solve inner alignment and also outer alignment" or "we just have to make sure it isn't scheming."

1. ^{^}

   Though I encourage anyone who understands the problem and thinks they can solve it to try to prove me wrong! I can sure see some directions and think it's possible for a very creative human to achieve in principle, I just don't currently think it's probable that it will happen, and my best guess about the progress of the humans focused on it is that they've made approximately none. 

   A frame which doesn't require a method which accepts any defined policy, ie which allows the method to be specialized for a particular policy or space of policies that seems (1) easiest to attain while (2) not implying physical world optimization, seems more promising to me.

I agree with you that analogies are needed, but they are also inevitably limited. So I'm fine with saying "AI is concerning because its progress is exponential, and we have seen from COVID-19 that we need to intervene early," or "AI is concerning because it can proliferate as a technology like nuclear weapons," or "AI is like biological weapons in that countries will pursue and use these because they seem powerful, without appreciating the dangers they create if they escape control." But what I am concerned that you are suggesting is that we should make the general claim "AI poses uncontrollable risks like pathogens do," or "AI needs to be regulated the way biological pathogens are," and that's something I strongly oppose. By ignoring all of the specifics, the analogy fails.

In other words, "while I think the disanalogies are compelling, comparison can still be useful as an analytic tool - while keeping in mind that the ability to directly learn lessons from biorisk to apply to AI is limited by the vast array of other disanalogies."