The utility of information should almost never be negative

Non-contrived instances where ignorance can have utility:

• For entrainment: A movie watcher wants to remain ignorant of a movie's ending until she actually sees it.
• To reduce bias: A scientist conducts a double-blind study.
• To respect privacy: I delete the private emails that have been accidentally forwarded to me.
• To avoid temptation: If I have difficulty keeping secrets, I should avoid learning them.
• Ignorant people can avoid scams by being too dumb to fool. In the ants and prisoner's dilemma tournaments, smart people lost by getting too clever.

The flaw I'd point out is that Clippy's utility function is utility = number of paperclips in the world not utility = number of paperclips Clippy thinks are in the world.

Learning about the creation or destruction of paperclips does not actually increase or decrease the number of paperclips in the world.

You are confusing utility with happiness. But you're confusing them even afterwards.

To see whether the utility of information can be negative, you shouldn't be thinking of scenarios like "I learned my girlfriend is cheating on me" -- (which is useful info to have in terms of determining her future trustworthiness and reliability), you should be thinking of scenarios like "My mother just told me she once discovered my father doing it doggy-style with his secretary on the kitchen table I'm currently eating on." which gross you out but don't really provide much more useful information than "My mother just told me she caught my father cheating on her; she let me know so I wouldn't be confused about why she's divorcing him".

There's a reason the sentence "TMI"="Too Much Information" has been invented. There's worth in information you can act on. Information that just makes you unhappy but you can't significantly act on is a negative, unless you value information for its own sake more than you value your own happiness.

People's happiness isn't the same as people's utility, but people are allowed to have a term for their own happiness in their utility function; most people do.

This posts needs a reference to Information Hazards: A Typology of Potential Harms from Knowledge.

If I were dying in my bed, I probably wouldn't even think to increase my probability that a friend had died just because no-one would have told me if they had.

There is something intuitively wrong with that statement. If you do not gain any new information, your probability calculation should not change. If the probability of not being told of a friend's death is approximately the same either way, there should be little update, and any miniscule update would intuitively be in the direction of 'If someone were dead, I'd be told'. So I did a rough (very very rough) calculation using the general idea that someone is unlikely to tell Dying You if a friend is dead if they are, and even more unlikely to tell Dying You that a friend is dead when they aren't. (That's just not funny, dude.)

0.00843437006 = (0.01x0.8)/((0.01x0.8)+(0.95x.99))

p(friend is dead given someone tells Dying You that they're dead) = p(friend is dead) p(no one tells you your friend is dead)/ p(same as above) + p(no one tells you a friend is dead) p(friend is not dead)

Same is true even for higher values of 'likelihood that a friend dies'.

0.173913043 = (0.2 x 0.8) / ((0.2 x 0.8) + (0.95 x .8))

0.457142857 = (0.5 x 0.8) / ((0.5 x 0.8) + (0.95 x .5))

I'm tempted to adjust the other probabilities for 50%, though. If death is so common among my peers anyways then my friends would probably be more likely to tell me.

EDIT: Haha, I forgot the multiplication sign is used for formatting. Woops.

This seems to be true only if your mental states of knowledge can have no effect on things that you care about... that is to say, it's pretty clearly false!

Sometimes humans care about their own mental states. It might count positively in my utility function that I not be psychologically distressed. Plausibly, learning certain bits of knowledge that I can't use practically ("The Great Old Ones are about to awake and devour the earth!") might have this effect with no compensating benefits, and so learning them would have negative utility.

Alternatively, my state of knowledge may affect something in the world. If I'm very bad at concealing my knowledge, then knowing about the secret conspiracy to take over the world may just get me killed: that's pretty clearly negative utility. Or, say, on being told that the fate of the world rests upon my performance in some task, I'm likely to do worse than if I didn't know (due to nerves).

In short: if you're a disembodied observer who merely picks effects to occur in the world, and whose utility function ranges strictly over things in the world other than yourself, then this might be true. Otherwise, not so much.

I prove this as a mathematical theorem here:

Theorem. Every act of observation has, before you make it, a non-negative expected utility.

Note that the theorem ignores everything that you might expect it to ignore from the formulation: non-rational or non-utility-maximising agents, the cost of acquiring and using the information, the effects of other people knowing you know it, the effects on yourself of knowing you know it (as distinct from merely knowing it and so updating your utility-maximising decisions), etc. etc.

Should I write this up for arXiv? The theorem is not deep, but neither is Aumann's agreement theorem, and that's in a journal.

Another special case: If learning the information requires great resources in computation or memory, and the expected utility of retaining (not using) these is greater than the expected utility of of the info, then you might not want to learn the info. (See Omohundro 2008

This special case can apply to AIs, where the metrics of computation and memory are well-defined.

But it can be true even for humans. The time or money spent learning something might be better spent in another way.

The clippy ought to instead be compelled to listen to paperclip news, just in case it hears of some way to optimize the paperclip number (e.g. it could hear that 10 000 paperclips were destroyed in some way which is likely to repeat but which clippy can prevent). As paperclip optimizer it then will never decide not to listen to the news. (Unless clippy doesn't really care about paperclips and it's bad feeling was merely put there as a heuristic to deal with paperclip destruction in the immediate vicinity, without distant news in mind)

The way the utility of listening to news can be negative though, is if the news are being constructed by an agent which hates paperclips, and knows enough about clippy as to impede the clippy's effectiveness with false news. (Note that any false news are true statements of form "news service said X")

I had this thought too a bit ago, and flailed about a bit trying to make it rigorous. I've thought about it off and on since then. Here's a sketch of what I've got:

Part 1: Suppose we have an entity who acts to maximize her own current credence in various positive outcomes. She will behave identically to a VNM-rational entity trying to maximize the probability of those outcomes iff she is VNM-rational and her utility function is U(P(outcome)=p) = k*p for all p and some constant k.

Part 2: Any credence-maximizer wants its future selves to be probability-maximizers, since this maximizes its current credence in future positive outcomes. Therefore, an entity that pre-commits to act as a probability-maximizer, or equivalently as though it had a linear utility-of-credence function, increases its current utility, at the possible expense of its future selves' utilities. Therefore any credence-maximizer that is capable of pre-commitment or self-modification will act identically to a VNM-rational probability maximizer.

As humans, we don't work that way. We don't constantly feel the pressure of knowledge like “People might have died since I last watched the news,” just because humans don't deal with probability in a rational manner.

Isn't this a thing some people do? Call your mother.

I would dislike finding out that my girlfriend were cheating on me [...] However, this is a consequence of the dodgily designed human brain.

You can argue that this, as well as many other real life examples, is a consequence of finite computational resources in general. The utility lost in trying to figure out what exactly is the proper course of action - now that you have the knowledge - perhaps outweighs the utility of knowing.

Did anyone else try to square 156 within 60 seconds after reading that line?