When is further research needed?

This seems incorrect. It doesn't really matter for blackmailer if you're aware of the blackmail or not, what matters is his estimate of the chance than you know.

Blackmailing is profitable if gain from successful blackmail chance you'll know about it chance you'll give in > cost of blackmail.

Unless you can guarantee 100% solid precommitment to not giving in to blackmail (and let's face it - friendly AI is easier than that), the more you increase the chance of knowing about it, the more blackmailing you'll face.

Acquiring information is never bad, in and of itself.

That idea is usually regarded as being incorrect around here - e.g. see here.

For instance, the document states that one example is "to measure the placebo effect". In that case, if you find out what treatment you actually got, that messes up the trial, and you have to start all over again.

There is a more defensible idea that accquiring accurate information is not ever bad - if you are a super-rational uber-agent, who is able to lie flawlessly, erase information perfectly, etc.

However, that is counter-factual. If you are a human, in practice, acquiring accurate information can harm you - and of course acquiring deceptive or inaccurate information can really cause problems.

Out of curiosity, what are the minimum dimensions of a paperclip? Is a collection of molecules still a paperclip if the only paper it can clip is on the order of a molecule thick?

Avoiding all such knowledge is a perfect precommitment strategy. It's hard to come up with better strategies than that, and even if your alternative strategy is sound blackmailer might very well not believe it and give it a try (if he can get you to know it, then are you really perfectly consistent?). If you can guarantee you won't even know, there's no point in even trying to blackmail you and this is obvious to even a very dumb blackmailer.

By the way, are there lower and upper bounds on number of paperclips in the universe? Is it possible for universe to have negative number of paperclips somehow. Or more paperclips than its numbers of atoms? Is this risk-neutral? (1% chance of 100 paperclips exactly as valuable as 1 paperclip?). I've been trying to get humans to describe their utility function to me, but they can never come with anything consistent, so I though I'd ask you this time.

No, that isn't what taw is saying. The point is that having more information and being known to have it can be extremely bad for you. This is not a counterexample to the theorem, which considers two scenarios whose only difference is in how much you know, but in real-life applications that's very frequently not the case.

I don't think taw's blackmail example is quite right as it stands, but here's a slight variant that is. A Simple Blackmailer will publish the pictures if you don't give him the money. Obviously if there is such a person, and if there are no further future consequences, and if you prefer losing the money to losing your reputation, it is better for you to know about the blackmailer so you can give him the money. But now consider a Clever Blackmailer, who will publish the pictures if you don't give him the money and if he thinks you might give him the money if he doesn't. If there's a Clever Blackmailer and you don't know it (and he knows you don't know it) then he won't bother publishing because the threat has no force for you -- since you don't even know there is one. But if you learn of his existence and he knows this then he will publish the pictures unless you give him the money, so you have to give him the money. So, in this situation, you lose by discovering his existence. But only because he knows that you've discovered it.

Suppose I consider whether to blackmail you. I do not have the ability to prove that I have the means to do so. You thereby would elect not to give me what I want - you're willing to take the risk. So I don't blackmail you.

If I gained the ability to prove that I have the means to do so, you would gain nothing if I didn't have the means, but lose if I did have them, because you would now be blackmailed and forced to give me stuff.

It is explicit in the hypotheses that you know how reliable your observations are, i.e. you know P_c(u|o).

It is explicitly stated in the hypotheses that you know how reliable your observations are.

I didn't downvote, or read the comment until just now for that matter, but perhaps someone had harmful options in mind.

Does the "blind carbon copy" feature in email count as a minimal example of "deniability engineering"? :)

Receiving information cannot hurt you. But if other agents acquire information that you have acquired information - well, that can hurt you.

Allow me to rewrite your post. 'Receiving information cannot hurt you. But receiving information can hurt you.'

It's true for anyone who understands random variables and expectations. There's a one line proof, after all.

The mathematical result is trivial, but its interpretation as the practical advice "obtaining further information is always good" is problematic, for the reason taw points out.

A particular agent can have wrong information, and make a poor decision as a result of combining the wrong information with the new information. Since we're assuming that the additional information is correct, I think it's reasonable to also stipulate that all previous information is correct.

Actually, I thought of that objection myself, but decided against writing it down. First of all, it's not quite right to refer to past information as 'right' or 'wrong' because information doesn't arrive in the form of propositions-whose-truth-is-assumed, but in the form of sense data.* It's better to talk about 'misleading information' rather than 'wrong information'. When adversary A tells you P, which is a lie, your information is not P but "A told me P". (Actually, it's not even that, but you get the idea.) If you don't know A is an adversary then "A told me P" is misleading, but not wrong.

Now, suppose the agent's prior has got to where it is due to the arrival of misleading information. Then relative to that prior, the agent still increases its expected utility whenever it acquires new data (ignoring taw's objection).

(On the other hand, if we're measuring expectations wrt the knowledge of some better informed agent then yes, acquiring information can decrease expected utility. This is for the same reason that, in a Gettier case, learning a new true and relevant fact (e.g. most nearby barn facades are fake) can cause you to abandon a true belief in favour of a false one.)

* Yes yes, I know statements like this are philosophically contentious, but within LW they're assumptions to work from rather than be debated.

This can be tested!

Kidnap people, place them in a room with a slit in its door. In the room is a magic marker and a slip of paper. They have however long they want to write a name of a co-holder of the NHL record for most points by brothers and slip the paper with that name written on it through the slit, and if they get it right on their first and only try, they get to leave the room.

I predict a valley of incorrect answers between the higher performances of the clued in and the clueless.

Ok - then I don't understand it well enough.

Thanks. Wasn't paying attention.