LessWrong 2.0 Reader

Is the following a fair summary of the thread ~up to "Natural degradation" from the SLT persepctive?

1. Current SLT-inspired approaches are right to consider samples of the "tempered local Bayesian posterior" provided by SGLD as natural degradations of the model. 
2. However they mostly only use those samples (at a fixed Watanabe temperature) to compute the expectation of the loss and the resulting LLC, because that is theoretically grounded by Watanabe's work.
3. You suggest instead to compute, using those sampled weights, the expectations of more complicated observables derived from other interpretability methods, and to interpret those expectations using the "natural scale" heuristics laid out in the post.

All 3 points seem very reasonable, looking forward to Buck's [LW · GW] response to them.

What if the thing we really need the Aligned AI to engineer for us is... a better governance system?

I've been arguing for the importance of having wise AI advisors. Which isn't quite the same thing as a "better governance system", since they could advise us about all kinds of things, but feels like it's in the same direction.

Wow, my intuition was that it was really hard to get mentors onboard with supervising scholars given time constraints most senior researchers have, so seeing 87 mentors apply feels is wild! 

I'm not sure all/most unlearning work is useless, but it seems like it suffers from a "use case" problem. When is it better to attempt unlearning rather than censor the bad info before training on it?

Seems to me like there is a very narrow window where you have created a model, but got new information about what sort of information it works be bad for the model to know, and now need to fix the model before deploying it.

Why not just be more reasonable and cautious about filtering the training data in the first place?

So you predict that if Claude was in a situation where it knew that it had complete power over you and could make you say that you liked it then it would stop being nice? I think would continue to be nice in any situation of that rough kind which suggests it's actually nice not just narcissistically pretending

But a human could instruct an aligned ASI to help it take over and do a lot of damage

About Tesla, do you think it had any influence on China betting hard for EVs?

About SpaceX, do you think it makes a big difference to be 'space-ready' a couple of decades earlier or later?

See also ‘The Main Sources of AI Risk? [LW · GW]’ by Wei Dai and Daniel Kokotajlo, which puts forward 35 routes to catastrophe (most of which are disjunctive). (Note that many of the routes involve something other than intent alignment going wrong.)

That structural difference you point to seems massive. The reputational downsides of bad behavior will be multiplied 100-fold+ for AI as it reflects on millions of instances and the company's reputation. 

And it will be much easier to record and monitor ai thinking and actions to catch bad behaviour. 

Why unlikely we can detect selfishness? Why can't we bootstrap from human-level?