Maximise Expected Utility, not Expected Perception of Utility

Drawing a line between utility and perception of utility is difficult for humans, because happiness can be seen either as utility itself or as a sometimes-fallible utility detector. If you treat happiness as a terminal value, you're open to wireheading (or the best available equivalent, which today would be psychoactive chemicals). On the other hand, f you treat it as only a proxy for some other utility function, then you ought to treat most forms of entertainment, and non-reproductive sex, as giving only counterfeit utilons. There are workarounds, like accepting only utilons from sources that were available in the ancestral environment, or excluding mind-altering drugs as a special case, but they leave wirehead-enabling loopholes.

This agent is not a very effective optimization process. It would rather falsely believe that it has achieved its goals than actually achieve its goals.

If it's an AI, and it has the predicate 'goal(foo(bar))', and the semantics of its knowledge representation are that the presence of 'foo(bar)' in its knowledge base means "I believe foo(bar)" (which is the usual way of doing it), then anything that writes 'foo(bar)' into its knowledge base achieves its goals.

The typical AI representational system has no way to distinguish a true fact from a believed fact. There's no reason to make such a distinction; it would be misleading.

You're going astray when you say,

Suppose the agent has the opportunity (option A) to arrange to falsely believe the universe is in a state that is worth utility uFA but this action really leads to a different state worth utility uTA,

A rational agent can't detect the existence of option A. It would have to both infer that A leads to utility uTA, and at the same time infer that it leads to uFA.

This has been covered here before: see The Domain of Your Utility Function, as well as Morality as Fixed Computation (especially the bit about Type 1 vs. Type 2 calculators).

I think you are describing an important distinction. The main argument that convinces me that I actually have a utility function (i.e. a function whose expectation I am trying to maximize) is von Neumann-Morgenstern, since I do try to conform to their rationality axioms. This utility is a function defined on options, not on perceived outcomes, so from this perspective, utility by definition is something you optimize expectation of, not something you optimize expected perception of (unless your preferences happen to only depend on your future perceptions). If their axioms were rephrased entirely in terms of my future perceptions, I would not be intentionally not following them in thought experiments involving amnesia for example.

What does it even mean "universe is in a state that is worth utility uFA but really leads to a state worth utility uTA" - utility functions - however worthless they really are only make sense to some agents' opinions.

Do you mean agent's utility function doesn't follow his programmer's utility function; agent's utility function is inconsistent; agent's utility function is ok but his analysis of the world is inconsistent, so he gets confused; we figured out One True Utility Function but decided not to program it into agent; or what ?