More accurate models can be worse

One solution is to have an RL policy that chooses where to update the model, but to use self-supervised (predictive) learning to decide what direction to update the model. For example, the RL policy can choose what to look at / attend to / think about, or more basically, "what to try predicting", then the model makes that prediction, then we update the model on the prediction error.

Then the RL policy can learn to take actions that account for value of information.

(This comment is loosely based on one aspect of what I think the brain does.)

Downvoted for burying the lede. I assumed from the buildup this was something other than what it was, e.g. how a model that contains more useful information can still be bad, e.g. if you run out of resources for efficiently interacting with it or something. But I had to read to the end of the second section to find out I was wrong.

"Rationality is not just about knowing facts. It's about knowing which facts are the most relevant." - Epictetus (c. AD 50)

I would be interested in a more precise definition of what you mean by information here.

In particular, it seems like you're using an unintuitive (to me) definition of information -- though one that's lines up colloquially with how we talk about computers.

For example, let's say I have a thumb drive ("Drive A") with two things on it:

1. A very short program that computes the digits of pi
2. A petabyte of computed digits of pi

And I have a second drive with one thing on it:

1. The millions of lines of source code for the Linux kernel

I might ask someone: which of these has more information on it?

The colloquial computer-storage based answer might be: the first one!  It takes up a petabyte, where the second one takes up less than a gigabyte.

But it feels like something important about the meaning of information (in an AI-understanding-the-world-sense) is being lost here.

(ETA: Also, if determinism factors in here, feel free to replace the petabyte of pi digits with something like a petabyte of recordings from a TRNG or something like that.)

For a more realistic example, this phenomenon of a more accurate model being worse is a common issue in database query optimization.

When a user runs a SQL query, the optimizer uses statistics about the data to estimate the cost of many different ways to execute the query, then picks the plan with the cheapest estimate.

When the optimizer misestimates the cost and chooses a bad plan, the typical solution is to add more detailed statistics about the data. But occasionally adding more statistics can cause the optimizer to choose a plan that's actually worse.

The brain overcomes this issue through the use of saliency-weighted learning. I don't have any references at the moment, but essentially, information is more salient when it is more surprising, either to the agent's world model or to its self model.

For the former, the agent is constantly making predictions about what it will experience along with the precision of these expectations such that when it encounters something outside of these bounds, it takes notice and updates its world model more strongly in the direction of minimizing these prediction errors.

The latter, however, is where the "usefulness" of salient information is most directly apparent. The agent is not just predicting what will happen in the external world like some disembodied observer. It is modeling what it expects to experience conditioned on its model of itself being healthy and functional. When something surprisingly good occurs, it takes special note of all information that was coincident with the pleasure signal to try to make such experiences more likely in the future. And when something surprisingly bad occurs, it also takes notice of all information coincident with the pain signal so that it can make such experiences less likely in the future.

When everything is going as expected, though, the agent will tend not to keep that information around. Saliency-weighted learning is all about steering an agent's models toward better predictive power and steering its behavior toward states of easier survivability (or easier learnability for a curiosity drive), allowing it to discard most information that it encounters in favor of only that which challenges its expectations.

I'm not sure it makes sense to talk about "the model's accuracy" in the abstract. You need to have some way of actually measuring the accuracy. One way to do that might be: of a list of questions sampled with a given probability distribution from the space of possible questions, what fraction does the model get right?

Given the setup, one obvious probability distribution is: questions asked by the actor in the process of deciding what to do. This seems, in fact, to be the only such probability distribution already present in the setup, rather than an additional benchmark brought in for as-yet unstated reasons.

Of course, there's a problem: the system as a whole doesn't know what questions the actor will ask. Or, if it knows the actor will only ask "what action will maximize u", we've just shifted things into the model; it will need to ask sub-questions to give a good answer.

Still, we need to get the questions that the model will be optimized to answer from somewhere. There actually is no such thing as a "best" (or even a "most accurate") model in the abstract. So we'll need to have a model of what questions the actor will ask, and what sub-questions we'll need to answer those. (To avoid Infinite regress, the purpose of that model is to predict what questions the world-model will need to answer.)

I think in your examples, the problem is straightforwardly that the (implicit) question-model being used isn't very accurate. This is not the only way that having a more accurate world-model can harm you though. An obvious example would be if Omicron (a bit less powerful than Omega) wants the password to a vault which you'd prefer not to let it open, but will only threaten to cut off your arms and legs if it (correctly) predicts you actually know the password. It would be a lot more interesting if someone could come up with examples that don't directly punish you simply for knowing more, but I'm out of time for the moment.