LessWrong 2.0 Reader

Strong-upvoted, this is precisely the kind of feedback needed to make the document better.

In effect, Omega makes you kill people by sending message.

Imagine two populations of agents, Not-Pull and Pull. 100% members of Not-Pull receive the message, don't pull and kill one person. In Pull population 99% members do not get the message, pull and get zero people killed, 1% receive message, pull and in effect kill 5 people. Being member of Pull population has 0.05 expected casualties and being member of Not-Pull population has 1 expected casualty. Therefore, you should pull.

The problem statement says it's true (Omega did indeed send the message, and the problem statement says that only happens when the message is true).

I think, in effect, this boils down to Omega telling you "This stranger is a murderous psychopath. You'd better not give them the opportunity."

My point is that the behavior is not well modeled as "hunting humans". They don't attack humans with the intent to kill and eat as prey. 

I would like to be able to set my defaults so that I never see any of the proposed AI content. Will this be possible?

Thanks for the feedback Garrett.

This was intended to be more of a technical report than a blog post, meaning I wanted to keep the discussion reasonably rigorous/thorough. Which always comes with the downside of it being a slog to read, so apologies for that!

I'll write a shortened version if I find the time!

Thanks James!

One failure mode is that the modification makes the model very dumb in all instances.

Yea, good point. Perhaps an extra condition we'd need to include is that the "difficulty of meta-level questions" should be the same before and after the modification - e.g. - the distribution over stuff it's good at and stuff its bad at should be just as complex (not just good at everything or bad at everything) before and after

Thanks Felix!

This is indeed a cool and surprising result. I think it strengthens the introspection interpretation, but without a requirement to make a judgement of the reliability of some internal signal (right?), it doesn't directly address the question of whether there is a discriminator in there.

The problem is that their understanding of their territory is not the same as our legal understanding, so they can attack on the roads outside their homes. 

The dogs are not hunting humans but want to defend territory or something similar.