Problem of Optimal False Information

Least optimal truths are probably really scary and to be avoided at all costs. At the risk of helping everyone here generalize from fictional evidence, I will point out the similarity to the Cthaeh in The Wise Man's Fear.

On the other hand, a reasonably okay falsehood to end up believing is something like "35682114754753135567 is prime", which I don't expect to affect my life at all if I suddenly start believing it. The optimal falsehood can't possibly be worse than that. Furthermore, if you value not being deceived about important things then the optimality of the optimal falsehood should take that into account, making it more likely that the falsehood won't be about anything important.

Edit: Would the following be a valid falsehood? "The following program is a really cool video game: "

You should choose the false belief, because Omega has optimized it for instrumental utility whereas the true belief has been optimized for disutility, and you may be vulnerable to such effects if only because you're not a perfectly rational agent.

If you were sure that no hazardous true information (or advantageous false information) could possibly exist, you should still be indifferent between the two choices: either of these would yield a neutral belief, leaving you with very nearly the same utility as before.

Ok, after actually thinking about this for 5 minutes, it's ludicrously obviously that the falsehood is the correct choice, and it's downright scary how long it took me to realize this and how many in the comments seems to still not realize it.

Some tools falsehoods have for not being so bad:

• sheer chaos theoretical outcome pumping. aka "you bash your face into the keyboard randomly believing a pie will appear and it programs a friendly AI" or the lottery example mentioned in other comments.

• any damage is bonded by being able to be obviously insane enough that it wont spread, or even cause you to commit suicide you don't believe it very long if you really think believing any falsehood is THAT bad.

• even if you ONLY value the truth, it could give you "all the statements in this list are true:" followed by a list of 99 true statements you are currently wrong about, and one inconsequential false one.

Some tools truths have for being bad:

• sheer chaos theoretical outcome pumping. aka "you run to the computer to type in the code for the FAI you just learnt, but fall down some stairs and die, and the wind currents cause a tornado that kills Eliezer and then runs through a scrapyard assembling a maximally malevolent UFAI"

• perceptual basilisks, aka "what Cthulhu face looks like"

• Roko-type basilisks

• borderline lies-by-commission/not mentioning side effects. "The free energy device does work, but also happens to make the sun blow up"

Note that both these lists are long enough, and the items unconnected enough, that there almost certainly many more points each at least as good as these that I haven't though of, and that both the lie and the true that likely to be using ALL these tools at the same time, in much more powerful ways than we can think of.

Thanks, this one made me think.

a machine inside the box will reprogram your mind such that you will believe it completely

It seems that Omega is making a judgement about making some "minimal" change that leaves you the "same person" afterwards, otherwise he can always just replace anyone with a pleasure (or torture) machine that believes one fact.

If you really believe that directly editing thoughts and memories is equivalent to murder, and Omega respects that, then Omega doesn't have much scope for the black box. If Omega doesn't respect those beliefs about personal identity, then he's either a torturer or a murderer, and the dilemma is less interesting.

But ... least convenient possible world ...

Omega could be more subtle than this. Instead of facts and/or mind reprogrammers in boxes, Omega sets up your future so that you run into (apparent) evidence for the fact that you correctly (or mistakenly) interpret, and you have no way of distinguishing this inserted evidence from the normal flow of your life ... then you're back to the dilemma.

The answer to this problem is only obvious because it's framed in terms of utility. Utility is, by definition, the thing you want. Strictly speaking, this should include any utility you get from your satisfaction at knowing the truth rather than a lie.

So for someone who valued knowing the truth highly enough, this problem actually should be impossible for Omega to construct.

You should make the choice that brings highest utility. While truths in general are more helpful than falsehoods, this is not necessarily true, even in the case of a truly rational agent. The best falsehood will, in all probability, be better than the worst truth. Even if you exclusively value truth, there will most likely be a lie that results in you having a more accurate model, and the worst possible truth that's not misleading will have negligible effect. As such, you should chose the black box.

I don't see why this would be puzzling.

I would pick the black box, but it's a hard choice. Given all the usual suppositions about Omega as a sufficiently trustworthy superintelligence, I would assume that the utilities really were as it said and take the false information. But it would be a painful, both because I want to be the kind of person who pursues and acts upon the truth, and also because I would be desperately curious to know what sort of true and non-misleading belief could cause that much disutility -- was Lovecraft right after all? I'd probably try to bargain with Omega to let me know the true belief for just a minute before erasing it from my memory -- but still, in the Least Convenient Possible World where my curiosity was never satisfied, I'd hold my nose and pick the black box.

Having answered the hypothetical, I'll go on and say that I'm not sure there's much to take from it. Clearly, I don't value Truth for its own sake over and beyond all other considerations, let the heavens fall -- but I never thought I did, and I doubt many here do. The point is that in the real world, where we don't yet have trustworthy superintelligences, the general rule that your plans will go better when you use an accurate map doesn't seem to admit of exceptions (and little though I understand Friendly AI, I'd be willing to bet that this rule holds post-singularity). Yes, there are times where you might be better off with a false belief, but you can't predictably know in advance when that is, black swan blow-ups, etc.

To be more concrete, I don't think there's any real-world analogue to the hypothetical. If a consortium of the world's top psychiatrists announced that, no really, believing in God makes people happier, more productive, more successful, etc., and that this conclusion holds even for firm atheists who work for years to argue themselves into knots of self-deception, and that this conclusion has the strongest sort of experimental support that you could expect in this field, I'd probably just shrug and say "I defy the data". When it comes to purposeful self-deception, it really would take Omega to get me on board.

First, your invocation of Everett branches adds nothing to the problem, as every instance of "you" may well decide not to choose. So, "choose or die" ought to be good enough, provided that you have a fairly strong dislike of dying.

Second, the traumatic memories example is great, but a few more examples would be useful. For example, the "truth" might be "discover LW, undergo religious deconversion, be ostracized by your family, get run over by a car while wondering around in a distraught state" whereas the "lie" might be "get hypnotized into believing Scientology, join the church, meet and hook up with a celebrity you dreamed about, live happily ever after".

Even more drastic: a bomb vs a wireheading device. The "truth" results in you being maimed, the "lie" results in you being happy, if out of touch with the world around you.

At this point it should be clear that there is no single "right" solution to the problem, and even the same person would choose differently depending on the situation, so the problem is not well defined as stated.

Edge case:

The truth and falsehood themselves are irrelevant to the actual outcomes, since another superintelligence (or maybe even Omega itself) is directly conditioning on your learning of these "facts" in order to directly alter the universe into its worst and best possible configurations, respectively.

These seem to be absolute optimums as far as I can tell.

If we posit that Omega has actual influential power over the universe and is dynamically attempting to create those optimal information boxes, then this seems like the only possible resulting scenario. If Omega is sufficiently superintelligent and the laws of entropy hold, then this also seems like the only possible resulting scenario for most conceivable minds, even if Omega's only means of affecting the universe is the information contained in the boxes.

Which box do you choose, and why?

I take the lie. The class of true beliefs has on average a significantly higher utility-for-believing than the class of false beliefs but there is an overlap. The worst in the "true" is worse than the best in "false".

I'd actually be surprised if Omega couldn't program me with a true belief that caused me to drive my entire species to extinction, and probably worse than that. Because superintelligent optimisers are badass and wedrifids are Turing-complete.

Would the following be a True Fact that is supported by evidence?

You open the white box, and are hit by a poison dart, which causes you to drop into a irreversible, excruciatingly painful, minimally aware, coma, where by all outward appearances you look fine, and you find out the world goes downhill, while you get made to live forever, while still having had enough evidence that Yes, the dart DID in fact contain a poison that drops you into an:

irreversible(Evidence supporting this, you never come out of a coma),

excruciatingly painful(Evidence supporting this, your nerves are still working inside your head, you can feel this excruciating pain)

minimally aware (Evidence supporting this, while you are in the coma you are still vaugely aware that you can confirm all of this and hear about bad news that makes you feel worse on a level in addition to physical pain, such as being given the old poison dart because someone thinks it's a treasured memento instead of a constant reminder that you are an idiot.)

coma(Evidence supporting this, you can't actually act upon the outer world as if you were conscious),

where by all outward appearances you look fine(Evidence supporting this, no one appears to be aware that you are in utter agony to the point where you would gladly accept a mercy kill.)

and you find out the world goes downhill (Evidence supporting this, while in a minimally aware state, you hear about the world going downhill, UFAI, brutal torture, nuclear bombs, whatever bad things you don’t want to hear about.)

while you get made to live forever: (Evidence supporting this, you never, ever die.)

I mean, the disutility would probably be worse than that, but... surely you never purposely pick a CERTAINTY of such an optimized maximum disutility, regardless of what random knowledge it might comes with. It would be one thing if the knowledge was such that it was going to be helpful, but since it comes as part and parcel of a optimized maximum disutility, the knowledge is quite likely to be something useless or worse, like “Yes, this dart really did contain a poison to hit you with optimized maximum disutility, and you are now quite sure that is true." (You would probably have been sure of that well before now even if it wasn't explicitly given to you as a true fact by Omega!)

And Omega didn't mislead you, the dart REALLY was going to be that bad in the class of facts about darts!

Since that (or worse) seems likely to be the White Box, I'll probably as carefully as possible select the Black box while trying to be extremely sure that I didn't accidentally have a brain fart and flip the colors of the boxes by mistake in sheer panic. Anyone who would pick the White box intentionally doesn't seem to be giving enough credence to just how bad Omega can make a certainty of optimized maximum disutility and how useless Omega can select the true fact to be.

As stated, the question comes down to acting on an opinion you have on an unknown, but within the principles of this problem potentially knowable conclusion about your own utility function. And that is: Which is larger: 1) the amount of positive utility you gain from knowing the most disutile truths that exist for you OR 2) the amount of utility you gain from knowing the most utile falsehoods that exist for you

ALMOST by definition of the word utility, you would choose the truth (white box) if and only if 1) is larger and you would choose the falsehood (black box) only if 2) is larger. I say almost by definition because all answers of the form "I would choose the truth even if it was worse for me" are really statements that the utility you place on the truth is higher than Omega has assumed, which violates the assumption that Omega knows your utility function and speaks truthfully about it.

I say ALMOST by definition because we have to consider the other piece of the puzzle: when I open box 2) there is a machine that "will reprogram your mind." Does this change anything? Well it depends on which utility function Omega is using to make her calculations of my long term utility. Is Omega using my utility function BEFORE the machine reprogram's my mind, or after? Is me after the reprogramming really still me after the reprogramming? I think within the spirit of the problem we must assume that 1) The utility happens to be maximized for both me before the reprogram and me after the reprogram, perhaps my utility function does not change at all in the reprogramming, 2) Omega has correctly included the amount of disutility I would have to the particular programming change, and this is factored in to her calculations so that the proposed falsehood and mind reprogramming do in fact, on net, give the maximum utility I can get from knowing the falsehood PLUS being reprogrammed.

Within these constraints, we find that the "ALMOST" above can be removed if we include the (dis)utility I have for the reprogramming in the calculation. So:

Which is larger: 1) the amount of positive utility you gain from knowing the most disutile truths that exist for you AND being reprogrammed to believe it OR 2) the amount of utility you gain from knowing the most utile falsehoods that exist for you

So ultimately, the question which would we choose is the question above. I think to say anything else is to say "my utility is not my utility," i.e. to contradict yourself.

In my case, I would choose the white box. On reflection, considering the long run, I doubt that there is a falsehood PLUS a reprogramming that I would accept as a combination as more utile than the worst true fact (with no reprogramming to consider) that I would ever expect to get. Certainly, this is the Occam's razor answer, the ceteris paribus answer. GENERALLY, we believe that knowing more is better for us than being wrong. Generally we believe that someone else meddling with our minds has a high disutility to us.

For completeness I think these are straightforward conclusions from "playing fair" in this question, from accepting an Omega as postulated.

1) If Omega assures you the utility for 2) (including the disutility of the reprogramming as experienced by your pre-reprogrammed self) is 1% higher than the utility of 1), then you want to choose 2), to choose the falsehood and the reprogramming. To give any other answer is to presume that Omega is wrong about your utility , which violates the assumptions of the question.

2) If Omega assures you the utility for 2) and 1) are equal, it doesn't matter which one you choose. As much as you might think "all other things being equal I'll choose the truth" you must accept that the value you place on the truth has already been factored in, and the blip-up from choosing the truth will be balanced by some other disutility in a non-truth area. Since you can be pretty sure that the utility you place on the truth is very much unrelated to pain and pleasure and joy and love and so on, you are virtually guaranteeing you will FEEL worse choosing the truth, but that this worse feeling will just barely be almost worth it.

Finally, I tried to play nice within the question. But it is entirely possible, and I would say likely, that there can never be an Omega who could know ahead of time with the kind of detail required, what your future utility would be, at least not in our Universe. Consider just the quantum uncertainties (or future Everett universe splits). It seems most likely that your future net utility covers a broad range of outcomes in different Everett branches. In that case, it seems very likely that there is no one truth that minimizes your utility in all your possible futures, and no one falsehood that maximizes it in all your possible futures. In this case we would have a distribution of utility outcomes from 1) and 2) and it is not clear that we know how to choose between two different distributions. Possibly utility is defined in such a way that it would be the expectation value that "truly" mattered to us, but that puts I think a very serious constraint on utility functions and how we interact with them that I am not sure could be supported.

The Bohr model of atomic structure is a falsehood which would have been of tremendous utility to a natural philosopher living a few hundred years ago.

That said, I feel like I'm fighting the hypothetical with that answer - the real question is, should we be willing to self-modify to make our map less accurate in exchange for utility? I don't think there's actually a clean decision-theoretic answer for this, that's what makes it compelling.

For me, the falsehood is the obvious choice. I don't particularly value truth as an end (or rather, I do, but there are ends I value several orders of magnitude more). The main reason to seek to have true beliefs if truth is not the end goal is to ensure that you have accurate information regarding how well you're achieving your goal. By ensuring that the falsehood is high-utility, that problem is fairly well utility.

My beliefs are nowhere near true anyway. One more falsehood is unlikely to make a big difference, while there is a large class of psychologically harmful truths that can make a large (negative) difference.

Not really relevant, but

Omega appears before you, and after presenting an arbitrary proof that it is, in fact, a completely trustworthy superintelligence of the caliber needed to play these kinds of games

I idly wonder what such a proof would look like. E.g. is it actually possible to prove this to someone without presenting them an algorithm for superintelligence, sufficiently commented that the presentee can recognise it as such? (Perhaps I test it repeatedly until I am satisfied?) Can Omega ever prove its own trustworthiness to me if I don't already trust it? (This feels like a solid Gödelian "no".)

This doesn't sound that hypothetical to me: it sounds like the problem of which organizations to join. Rational-leaning organizations will give you true information you don't currently know, while anti-rational organizations will warp your mind to rationalize false things. The former, while not certain to be on net bad, will lead you to unpleasant truths, while people in anti-rational groups are often duped into a kind of happiness.

I'm not sure this scenario even makes sense as a hypothetical. At least for me personally, I find it doubtful that my utility could be improved according to my current function by being made to accept a false belief that I would normally reject outright.

If such a thing is possible, then I'd pick the false belief, since utility is necessarily better than disutility and I'm in no position to second guess Omega's assurance about which option will bring more, and there's no meta-utility on the basis of which I can be persuaded to choose things that go against my current utility function. But even granting the existence of Omega as a hypothetical I'd bet against this scenario being able to happen to me.

Edit: this comment has made me realize that I was working under the implicit assumption that the false belief could not be something that would deliver its utility while being proven wrong. If I include such possibilities, there are definitely many ways that my utility could be improved by being convinced of a falsehood, but I would only be temporarily convinced, whereas I parsed the dilemma as one where my utility is increased as long as I continue to believe the falsehood.

Hm. If there is a strong causal relationship between knowing truths and utility, then it is conceivable that this is a trick: the truth, while optimized for disutility, might still present me with a net gain over the falsehood and the utility. But honestly, I am not sure I buy that: you can get utility from a false belief, if that belief happens to steer you in such a way that it adds utility. You can't normally count on that, but this is Omega we are talking about.

The 'other related falsehoods and rationalizing' part has me worried. The falsehood might net me utility in and of itself, but I poke and prod ideas. If my brain becomes set up so that I will rationalize this thought, altering my other thought-patterns to match as I investigate... that could be very bad for me in the long run. Really bad.

And I've picked inconvenient truth over beneficial falsehood before on those grounds, so I probably would pick the inconvenient truth. Maybe, if Omega guaranteed that the process of rationalizing the falsehood would still have me with more utility- over the entire span of my life- than I would earn normally after you factor in the disutility of the inconvenient truth then I would take the false-box. But on the problem as it stands, I'd take the truth-box.

How one responds to this dilemma depends on how one values truth. I get the impression that while you value belief in truth, you can imagine that the maximum amount of long-term utility for belief in a falsehood is greater than the minimum amount of long-term utility for belief in a true fact. I would not be surprised to see that many others here feel the same way. After all, there's nothing inherently wrong with thinking this is so.

However, my value system is such that the value of knowing the truth greatly outweighs any possible gains you might have from honestly believing a falsehood. I would reject being hooked up to Nozick's experience machine on utilitarian grounds: I honestly value the disutility of believing in a falsehood to be that bad*.

(I am wary of putting the word "any" in the above paragraph, as maybe I'm not correctly valuing very large numbers of utilons. I'm not really sure how to evaluate differences in utility when it comes to things I really value, like belief in true facts. The value is so high in these cases that it's hard to see how anything could possibly exceed it, but maybe this is just because I have no understanding of how to properly value high value things.)

SMBC comics has a relevant strip: would you take a pill to ease your suffering when such a suffering no longer serves any purpose? (The strip goes for the all-or-nothing approach, but anything milder than that can be gamed by a Murder-Gandhi slippery slope).

This example is obviously hypothetical, but for a simple and practical case, consider the use of amnesia-inducing drugs to selectively eliminate traumatic memories; it would be more accurate to still have those memories, taking the time and effort to come to terms with the trauma... but present much greater utility to be without them, and thus without the trauma altogether.

Deleting all proper memories of an event from the mind doesn't mean that you delete all of it's traces.

An example from a physiology lecture I took at university: If you nearly get hit by a car on the street that's an event you will consciously forget very soon. Your amygdala however doesn't and will still send stress signals when you come to the same physical location in the future.

Children at the age of one can't form proper memories that they can remember as adults. That doesn't mean that trauma they experience during that time don't have an effect on their lifes as adults.

In the hypnosis community people who have the tools to delete memories generally consider it to be a bad idea to remove specific traumatic memories. It can mess people up because they still have the emotional effects of the trauma but don't know the reason.

I'll take the black box.

You are required to choose one of the boxes; if you refuse to do so, Omega will kill you outright and try again on another Everett branch.

Everett branches don't (necessarily) work like that. If 'you' are a person who systematically refuses to play such games then you just don't, no matter the branch. Sure, the Omega in a different branch may find a human-looking creature also called "Endovior" that plays such games but if it is a creature that has a fundamentally different decision algorithm then for the purpose of analyzing your decision algorithm it isn't "you". (There are also branches in which an actual "you" plays the game but only as a freak anomaly of an event like the way an actual 'me' in a freakishly small Everett branch walked through a brick wall that one time. Still not exactly something to identify with as 'you' doing it.)

It's a trivial example, but "the surprise ending to that movie you're about to see" is a truth that's generally considered to have disutility. ;)

Optimised for utility X sounds like something that would win in pretty much any circumstance. Optimised for disutility Y sounds like something that would lose in pretty much any circumstance. In combination, the answer is especially clear.

By what sort of mechanism does a truth which will not be misleading in any way or cause me to lower the accuracy of any probability estimates nevertheless lead to a reduction in my utility? Is the external world unchanged, but my utility is lowered merely by knowing this brain-melting truth? Is the external world changed for the worse by differing actions of mine, and if so then why did I cause my actions to differ, given that my probability estimate for the false-and-I-already-disbelieved-it statement "these new actions will be more utility-optimal" did not become less accurate?

Can I inject myself with a poison that will kill me within a few minutes and THEN chose the falsehood?

I am worried about "a belief/fact in its class" the class chosen could have an extreme effect on the outcome.

I choose the truth.

Omega's assurances imply that I will not be in the valley of bad rationality mentioned later.

Out of curiosity, I also ask Omega to also show me the falsehood, without the brain alteration, so I can see what I might have ended up believing.

Does the utility calculation from the false belief include utility from the other beliefs I will have to overwrite? For example, suppose the false belief is "I can fly". At some point, clearly, I will have to rationalise away the pain of my broken legs from jumping off a cliff. Short of reprogramming my mind to really not feel the pain anymore - and then we're basically talking about wireheading - it seems hard to come up with any fact, true or false, that will provide enough utility to overcome that sort of thing.

I additionally note that the maximum disutility has no lower bound, in the problem as given; for all I know it's the equivalent of three cents. Likewise the maximum utility has no lower bound, in the problem as written. Perhaps Omega ought to provide some comparisons; for example he might say that the disutility of knowing the true fact is at least equal to breaking an arm, or some such calibration.

As the problem is written, I'd take the true fact.

The parallels with Newcomb's Paradox are obvious, and the moral is the same. If you aren't prepared to sacrifice a convenient axiom for greater utility, you're not really rational. In the case of Newcomb's Paradox, that axiom is Dominance. In this case, that axiom is True Knowledge Is Better Than False Knowledge.

In this instance, go for falsehood.