LessWrong 2.0 Reader

Is your question directed at me, or the person I was replying to? I agree with the point "Sun is big, but..." makes. Here's a link to a recent summary of my view on a plausible plan for the world to handle surviving AI. Please feel free to share your thoughts on it. https://www.lesswrong.com/posts/NRZfxAJztvx2ES5LG/a-path-to-human-autonomy [LW · GW] 

Wouldn't that destroy the whole idea? Anyone could tell that an opinion voiced that's not on the list must have been the person's true opinion.

In fact, I'd hope that several people composed the list, and didn't tell each other what items they added, so no one can say for sure that an opinion expressed wasn't one of the "hot takes".

I think all the same arguments that it doesn't change decisions also apply to why it doesn't change virtue evaluations.  It still all adds up to normality.  It's still unimaginably big.  Our actions as well as our beliefs and evaluations are irrelevant at most scales of measurement.

Because model has residual connections.

I guess one way of framing it is that I find the shoggoth/face idea great as a science experiment; it gives us useful evidence! However, it doesn't make very much sense to me as a safety method intended for deployment.

Sadly, gathering evidence of misalignment in deployment seems likely to me to be one of the most effective strategies for gathering legible evidence (at least for early systems) given likely constraints. (E.g., because people won't believe results in text beds and because RL might be too expensive to run twice.)

Sure, I mean that I expect that for the critical regime for TAI, we can literally have the Face be 3.5 sonnet.

Who generated the hot takes? I'd love to see the full list.

Thanks for the interesting comments.

You're right, I didn't discuss the possibility of infinite numbers of branches, though as you suggest this leads to essentially the same conclusion as I reach in the case of finite branches, which is that it causes problems for consequentialist ethics (Joe Carlsmith's Infinite Ethics is good on this). If what you mean by 'normalize everything' is to only consider the quantum weights (which are finite as mathematical measures) and not the number of worlds, then that seems more a case of ignoring those problems rather than addressing them.

I hope it was clear that I was suggesting a third approach (the number of worlds is neither finite, nor infinite, but indefinite) which does I think address the ethical problems better, since if there is no definite number of worlds then we have a reason to ignore the number of worlds and focus on the weights. 

This third approach is based on the idea that 'worlds' are macroscopic, emergent phenomena created through decoherence (Wallace's book contains a full mathematical treatment of this). This supports both the claim that the number of worlds is indefinite (since it depends on ultimately arbitrary mappings of macroscopic to microscopic states) and the claim that worlds are created through quantum processes (since they are macroscopically indistinguishable before decoherence occurs). My point in the post was that these two claims in combination can avoid the repugnant conclusion via the approach of focusing on the weights.

(And when it comes to the virtue-theoretic implications, I've again tried to follow a weight-based approach, and not make assumptions about whether worlds are created or revealed.)

Thanks for the Egan suggestion, yea I love his work though need to read Quarantine more fully. It seems like the most philosophically relevant bit might be the ending which of course is the source of Egan's Law (it all adds up to normality). I also need to read his short story 'Singleton', which I gather is very relevant too.

Here was the combined effect

Where do the numbers come from?

Having read something about self-driving cars actually being a thing now, I wonder how the trolley-problem thing (and whatever other ethics problems come up) was solved in the relevant regulation?