The Argument from Philosophical Difficulty

I may have misunderstood, sorry. I thought you gave it near 100% certainty that there could be only 3 ways, not the more reasonable "my knowledge of this problem is so marginal, I can't give it a good estimate of probability, since it would be drowned in error bars".

would you agree that people trying to solve a problem over some time only to find that all the plausible approaches they could come up with seem quite difficult is useful evidence for that problem being intrinsically difficult?

Certainly it's an indicator, especially if a group of smart people who have been able to successfully solve a number of related problems get stumped by something that appears to be in the same reference class. In my area it was the attempts to quantize gravity some time in the 50s and 60s, after successes with electromagnetism and weak interactions. After all, gravity is the weakest of them all. No one expected that there would be very little progress, half a century later, despite the signs being there.

I am not sure what "intrinsically difficult" means. My best guess is that it requires a Kuhnian paradigm change. Though sometimes it's not enough, and there are also the issues of just having to grind through a lot of calculations, like with the Fermat's last theorem, and the Poincaré conjecture. Special relativity, on the other hand, only required a "paradigm shift", the underlying math is trivial.

It might be interesting to consider this argument from an outside view perspective. Can you give a sample of arguments that you think is comparable to this one so we can check how valid they tend to be in retrospect?

One off-hand example that springs to mind is the Landau pole, inevitable and unavoidable in gauge theories. That resulted in the whole approach having been rejected in the Soviet Union for years, yet the the development of the renormalization formalism made QED the most precise physical theory, while still being mathematically inconsistent to this day. I strongly suspect that similarly adequate progress in AI alignment, for example, can be made without resolving all the mathematical, philosophical or meta-philosophical difficulties. The scenario 5 hints at something like that.

Thanks, this position makes more sense in light of Beyond Astronomical Waste (I guess I have some concept of "a pretty good future" that is fine with something like a bunch of human-descended beings living a happy lives that misses out on the sort of things mentioned in Beyond Astronomical Waste, and "optimal future" which includes those considerations). I buy this as an argument that "we should put more effort into making philosophy work to make the outcome of AI better, because we risk losing large amounts of value" rather than "our efforts to get a pretty good future are doomed unless we make tons of progress on this" or something like that.

"Thousands of millions" was a typo.

I'm super unsure about the intentional case, and agree that I want to see more work on that front, but it feels like a particular problem that can be solved with something like strategy/policy work. Put another way, intentional value corruption seems like a non-central example of problems that arise from philosophical difficulty. I agree that corrigibility + good overseers does not clearly solve it.

For the unintentional case, I think that overseers who care about getting philosophy right are going to think about value drift, because many of us are currently thinking about it. It seems like as long as the overseers make this apparent to the AI system and are sufficiently risk-averse, a corrigible AI system would take care not to corrupt their values. (The AI system might fail at this, but this doesn't seem that likely to me, and it feels very hard to make progress on that particular point without more details on how the AI system works.)

I do think that we want to think about how to ensure that there are overseers who care about getting the questions right, who know about value drift, who will be sufficiently risk-averse, etc.

What would you say is the primary source of the problem?

The fact that humans don't generalize well out of distribution, especially on moral questions; and the fact that progress can cause distribution shifts that cause us to fail to achieve our "true values".

What do you think the implications of this are?

Um, nothing in particular.

I'm not sure why you ask this though.

It's very hard to understand what people actually mean when they say things, and a good way to check is to formulate an implication of (your model of) their model that they haven't said explicitly, and then see whether you were correct about that implication.