I would add that people overestimate their ability to guess others preferences. "He just wants money" or "She just wants to marry him". Such oversimplified models could be not just useful simplifications, buts could be blatantly wrong.

I agree, and I think much of the difficulty people have in accepting the result comes from not seeing implicitly assumed norms we are always applying to understand things. I think this runs much deeper than saying humans have something like an empathy module, though, and is a general problem of humans not seeing reality clearly, and instead thinking they see it when in fact what they are seeing (and especially what they are interpreting and inferring) is tainted by prior evidence in ways that makes everything humans do conditional on the priors such that no seeing is truly free and independent of the conditions in which it arises.

That's fairly abstract, so another way to put it is that we're constantly seeing the world on the assumption that we already know what the world looks like. We can learn to assume less, but the natural, adaptive state is to assume a lot because it has lead to greater reproductive fitness, probably specifically because it allowed hard-to-make inferences possible only by making strong assumptions about the world that were often true (or true enough for our ancestors' purposes).

(I think this ties into the story I've been telling for a long time about developmental psychology, and the newer story I've been telling about human brains doing minimization of prediction error with additional homeostatic set points, but I also think it stands on its own without that, so I write here without reference to them other than this comment.)

The problem with the maths is that it does not correlate 'values' with any real world observable. You give all objects a property, you say that that property is distributed by simplicity priors. You have not yet specified how these 'values' things relate to any real world phenomenon in any way. Under this model, you could never see any evidence that humans don't 'value' maximizing paperclips.

To solve this, we need to understand what values are. The values of a human are much like the filenames on a hard disk. If you run a quantum field theory simulation, you don't have to think about either, you can make your predictions directly. If you want to make approximate predictions about how a human will behave, you can think in terms of values and get somewhat useful predictions. If you want to predict approximately how a computer system will behave, instead of simulating every transistor, you can think in terms folders and files.

I can substitute words in the 'proof' that humans don't have values, and get a proof that computers don't have files. It works the same way, you turn your uncertainty in the relation between the exact and the approximate into a confidence that the two are uncorrelated. Making a somewhat naive and not formally specified assumption along the lines of, "the real action taken optimizes human values better than most possible actions" will get you a meaningful but not perfect definition of 'values'. You still need to say exactly what a "possible action" is.

Making a somewhat naive and not formally specified assumption along the lines of, "the files are what you see when you click on the file viewer" will get you a meaningful but not perfect definition of 'files'. You still need to say exactly what a "click" is. And how you translate a pattern of photons into a 'file'.

We see that if you were running a quantum simulation of the universe, then getting values out of a virtual human is the same type of problem as getting files off a virtual computer.

I wonder if part of messiness might stem from confusing various domains and ranges. For example, for human, we have a complex of wants -- some are driven very much by physiological factors, some by cultural factor and some by individual factors (including things like what I did yesterday or 5 hours ago). We might call these our preference domain.

Then we need some function mapping the preferences into the range of behaviors that are observable. Assuming that there is something approximating a function here (caveat - not a math guy here so maybe that is misused/loaded here). From that we have some hope for deducing the behavior back to the preference.

However, we should not consider the above three sources as coming from the same domain, or mapping to the same range. Confusion may come in from both the fuzziness (I'm implicitly agreeing with the general cannot infer preferences from behavior that well as a general proposition) of the "correct" function as well as a confusion of associating a behavior to one of the three ranges, and then attempting to deduce the preference.

If I see A doing x and ascribe x to the physiological range and then attempt to deduce the preference (in the physiological domain) when x is actually in the individual range for A I will probably see a lot of errors. But maybe not 100% error.

I do think there is something to the we're all human so can recognize a lot of meaning in action from others -- but things like culture (as mentioned) does influence performance here. So, what is an acceptable accuracy rate? Is the goal mathematical certainty or something else?

Your title seems clickbaity, since its question is answered no in the post, and this article would have been more surprising had you answered yes. (And my expectation was that if you ask that question in the title, you don't know the answer anymore.)

having implicit access to categorisation modules that themselves are valid only in typical situations... is not a way to generalise well

How do you know this? Should we turn this into one of those concrete ML experiments?