Metaphilosophical competence can't be disentangled from alignment

My current intuition is that there are under 100 people whom, if 1,000,000,000,000,000x'd, would end up avoiding irreversible catastrophes with > 50% probability. (I haven't thought too much about this question, and wouldn't be surprised if I update to thinking there are fewer than 10 such people, or even 0 such people.)

I've asked this before but don't feel like I got a solid answer: (a) do you think that giving the 100th person a lot of power is a lot worse than the status quo (w.r.t. catastrophic risk), and (b) why?

If you think it's a lot worse, the explanations I can imagine are along the lines of: "the ideas that win in the marketplace of ideas are systematically good," or maybe "if people are forced to reflect by thinking some, growing older, being replaced by their children, etc., that's way better than having them reflect in the way that they'd choose to given unlimited power.", or something like that.

But those seem inconsistent with your position in at least two ways:

• If this is the case, then people don't need metaphilosophical competence to be fine, they just need a healthy respect for business as usual and whatever magic it is that causes the status quo to arrive at good answers. Indeed, there seem to be many people (>> 100) who would effectively abdicate their power after being greatly empowered, or who would use it in a narrow way to avoid catastrophes but not to change the basic course of social deliberation.
• The implicit claim about the magic of the status quo is itself a strong metaphilosphical claim, and I don't see why you would have so much confidence in this position while thinking that we should have no confidence in other metaphilosphical conclusions.

If you think that the status quo is even worse, then I don't quite understand what you mean by a statement like:

Once humanity makes enough metaphilosophical progress (which might require first solving agent foundations), I might feel comfortable 1,000,000,000,000,000x'ing the most metaphilosophically competent person alive, though it's possible I'll decide I wouldn't want to 1,000,000,000,000,000x anyone running on current biological hardware. I'd also feel good 1,000,000,000,000,000x'ing someone if we're in the endgame and the default outcome is clearly self-annihilation.

Other questions: why can we solve agent foundations, but the superintelligent person can't? What are you imagining happening after you empower this person? Why are you able to foresee so many difficulties that they predictably won't see?

I do not expect that any human brain would be safe if scaled up by that amount, because of lack of robustness to relative scale [LW · GW]. My intuition is that alignment is very hard, but I don't have an explicit reason right now.

I think the number of safe people depends sensitively on the details of the 1,000,000,000,000,000xing. For example: Were they given a five minute lecture on the dangers of value lock-in? On the universe's control panel, is the "find out what would I think if I reflected more, and what the actual causes are of everyone else's opinions" button more prominently in view than the "turn everything into my favorite thing" button? And so on.

I think everyone (including me) would go crazy from solitude in this scenario, so that puts the number at 0. If you guarantee psychological stability somehow, I think most adults (~90% perhaps) would be good at achieving their goals (which may be things like "authoritarian regime forever"). This is pretty dependent on the humans becoming more intelligent -- if they just thought faster I wouldn't be nearly as optimistic, though I'd still put the number above 0.

Well, wouldn't it be great if we had sound metaphilosophical principles that help us distinguish epistemic pits from correct conclusions! :P

I actually think humanity is in a bunch of epistemic pits that we mostly aren't even aware of. For example, if you share my view that Buddhist enlightenment carries significant (albeit hard-to-articulate) epistemic content, then basically all of humanity over basically all of time has been in the epistemic pit of non-enlightenment.

If we figure out the metaphilosophy of how to robustly avoid epistemic pits, and build that into an aligned AGI, then in some sense none of our current epistemic pits are that bad, since that AGI would help us climb out in relatively short order. But if we don't figure it out, we'll plausibly stay in our epistemic pits for unacceptably long periods of time.

We can recognize ideas from the past that look like epistemic pits, but if you were to go back in time and try to argue against those ideas, you would be dismissed as incorrect. If you brought proof that the future society held your ideas instead of your interlocutors, that would taken as evidence of the increasing degeneracy of man rather than as evidence that your ideas were more correct than theirs. So what value does the concept of an epistemic pit bring?

I can name one epistemic pit that humanity fell into: slavery. At one point treating other human beings as property to be traded was considered normal, proper and right. After a very long time, and more than a few armed conflicts, western, liberal, capitalist societies updated away from this norm. However, if I were to go back to 1600 and try to argue that slavery was immoral, I would be seen holding an incorrect viewpoint. So, given that, how can one recognize that one is in an epistemic pit in the moment?

I think values and decision-making processes can't be disentangled, because I think people's values often stem from their decision-making processes. For example, someone might be selfish because they perceive the whole world to be selfish and uncaring, and in return acts selfish and uncaring by default. This default behavior might cause the world to act selfishly and uncaringly toward him, further reinforcing his perception. If he fully understood this was happening (rather than the world just being fundamentally selfish), he might experiment with acting more generously with the rest of the world, and observe the rest of the world to act more generously in return, and in turn stop being selfish entirely.

In general I expect amplification to improve decision-making processes substantially, but in most cases to not improve them enough. For example, it's not clear that amplifying someone will cause them to observe that their own policy of selfishness is locking them into a fixed point that they could "Löb out of" into a more preferable fixed point. I expect this to be particularly unlikely if e.g. they believe their object-level values to be fixed and immutable, which migh result in a fairly pernicious epistemic pit.

My intuition is that most decision-making processes have room for subtle but significant improvements, that most people won't realize these improvements upon amplification, and that failing to make these improvements will result in catastrophic amounts of waste. As another example, it seems quite plausible to me that:

• the vast majority of human-value-satisfaction (e.g. human flourishing, or general reduction of suffering) comes from acausally trading with distant superintelligences.
• most people will never care about (or even realize) acausal trade, even upon amplification.