LessWrong 2.0 Reader

All 3 points seem very reasonable, looking forward to Buck's [LW · GW] response to them.

What if the thing we really need the Aligned AI to engineer for us is... a better governance system?

I've been arguing for the importance of having wise AI advisors. Which isn't quite the same thing as a "better governance system", since they could advise us about all kinds of things, but feels like it's in the same direction.

Wow, my intuition was that it was really hard to get mentors onboard with supervising scholars given time constraints most senior researchers have, so seeing 87 mentors apply feels is wild! 

I'm not sure all/most unlearning work is useless, but it seems like it suffers from a "use case" problem. When is it better to attempt unlearning rather than censor the bad info before training on it?

Seems to me like there is a very narrow window where you have created a model, but got new information about what sort of information it works be bad for the model to know, and now need to fix the model before deploying it.

Why not just be more reasonable and cautious about filtering the training data in the first place?

So you predict that if Claude was in a situation where it knew that it had complete power over you and could make you say that you liked it then it would stop being nice? I think would continue to be nice in any situation of that rough kind which suggests it's actually nice not just narcissistically pretending

But a human could instruct an aligned ASI to help it take over and do a lot of damage

About Tesla, do you think it had any influence on China betting hard for EVs?

About SpaceX, do you think it makes a big difference to be 'space-ready' a couple of decades earlier or later?

See also ‘The Main Sources of AI Risk? [LW · GW]’ by Wei Dai and Daniel Kokotajlo, which puts forward 35 routes to catastrophe (most of which are disjunctive). (Note that many of the routes involve something other than intent alignment going wrong.)

That structural difference you point to seems massive. The reputational downsides of bad behavior will be multiplied 100-fold+ for AI as it reflects on millions of instances and the company's reputation. 

And it will be much easier to record and monitor ai thinking and actions to catch bad behaviour. 

Why unlikely we can detect selfishness? Why can't we bootstrap from human-level? 

Sure, we don't know exactly how good EVs are for fighting climate change, but the current view is that they are needed in the context in which we are because they seem better mostly than the other alternatives. [Incidentally, since some time I tend to think that he's probably been vastly less net-good in the past than I previously thought. Not really because of him, but because Chinese companies are beating everyone, including Tesla, with their EVs (and I don't think he's had any influence in China betting hard for EVs, though I might be wrong here); so if Tesla would have not existed, the adoption of EVs would just have been only delayed for few years (and mostly only in the west). So his net-positive contribution -for me and now- seems much lower than it seemed before.] But this, of course, is not what I am asking for. 

Maybe Hitler, by sheer chance, killed someone who had been much worse than him. But this would not make him be net-positive in the sense of this question (eg. we'd had other ways to deal with that person -even if the odds that we did are very small).

But probably I have not been clear enough, sure.