Protected From Myself

"But what if you were "optimistic" and only presented one side of the story, the better to fulfill that all-important goal of persuading people to your cause? Then you'll have a much harder time persuading them away from that idea you sold them originally - you've nailed their feet to the floor, which makes it difficult for them to follow if you yourself take another step forward."

Hmmm... if you don't need people following you, could it help you (from a rationality standpoint) to lie? Suppose that you read about AI technique X. Technique X looks really impressive, but you're still skeptical of it. If you talk about how great technique X looks, people will start to associate you with technique X, and if you try to change your mind about it, they'll demand an explanation. But if you lie (either by omission, or directly if someone asks you about X), you can change your mind about X later on and nobody will call you on it.

NOTE: This does require telling the same lie to everyone; telling different lies to different groups of people is, as noted, too messy.

"The universe isn't set up to reward virtue", but I think most people are. If someone is deceiving you then doing what they ask is likely not in your interest, otherwise they could persuade you without deception.

If something is difficult to explain due to technical understanding, you can 'lie' about it, while noting that it is an oversimplification intended to give an idea, and not wholly correct. I believe this is the norm for science publications targeted at the general population.

To lie effectively, I find the only way is to convince myself of something I know to be false. Then I can subsequently tell what I believe to be the truth without things like keeping track of what I told who or body language clues. This is, of course, still perilous and immoral in other ways, and often non-permanent since certain things can trigger the original memory.

Is it only honesty that has this protection-rail tendency, or have other ethics also had it?

This is a worrisome line of thought, as I consider one of the main underlying points of this blog to question the necessity and rationality of conventional ethics.

What if the belief in God grants you some form of protection against threats of which you are not currently aware? For example, the threat of insanity, which we know to be sort of an occupational hazard among AI researchers?

Just for the sake of devil's advocacy:

4) You want to attribute good things to your ethics, and thus find a way to interpret events that enables you to do so.

If we see that adhering to ethics in the past has wound up providing us with utility, the correct course of action is not to throw out the idea of maximizing our utility, but rather to use adherence to ethics as an integral part of our utility maximization strategy.

I wonder if liars or honest folk are happier and or more successful in life.

simon: "Just for the sake of devil's advocacy: 4) You want to attribute good things to your ethics, and thus find a way to interpret events that enables you to do so."

Eliezer: "The universe isn't set up to reward virtue - so why did my ethics help so much? Am I only imagining the phenomenon? That's one possibility."

I think considerations like these are probably not too meaningful. You're likely to be mentally unstable or misguided in some small way that has an overriding influence (at least at this level of effect) that you're unaware of.

The universe isn't set up to reward virtue.

I believe that ethics are an effort to improve the odds of good outcomes. So it's not that the universe is set up to reward ethics, it's that ethics are set up to follow the universe.

The challenge is that what we're taught is good is a mixture of generally useful rules, rules which are more useful to the people in charge than to the people who aren't, and mere mistakes.

When I saw The Dark Knight, I was left thinking how long it's going take before some truth-seeking cop realizes that Batman didn't kill those people and Gordon is part of the conspiracy. Acceptable risk, Batman?

You can't duplicate this protective effect by trying to be clever and calculate the course of "highest utility". The expected utility just takes into account the things you know to expect. It really is amazing, looking over my history, the extent to which my ethics put me in a recoverable position from my unanticipated, fundamental mistakes, the things completely outside my plans and beliefs.

You acted as though you anticipated the unanticipated?

Probably either: you were lucky; your utility function isn't what you consciously thought it was; - or you have supernatural moral powers.

Probably either: you were lucky; your utility function isn't what you consciously thought it was; - or you have supernatural moral powers.

Or it is a tiny note of accord, to be attended to as diligently as the tiny notes of discord. Which is what the post went on to do.

Success is as much to be learned from as failure.

Excellent post. Please write more on ethics as safety rails on unseen cliffs.

Good consequences may come from good virtues, I gather.

pdf23ds: I think considerations like these are probably not too meaningful. You're likely to be mentally unstable or misguided in some small way that has an overriding influence (at least at this level of effect) that you're unaware of.

Is it only honesty that has this protection-rail tendency, or have other ethics also had it?

Interesting question. As far as I can tell, the two main effects that leap out at me are (1) the benefit of having not done various life-complicating bad things in the pursuit of early goals that I later had to change, and (2) the beneficial effect of holding myself to a higher standard when pursuing ethical obligations.

Has my life been better because of my sense of ethical inhibition against taking and wielding power? I honestly don't know - I can't compare my possible selves side-by-side. Maybe that other Eliezer learned to wield power well through practice, and built a large solid organization. Or maybe he turned to the dark side and ended up surrounded by a coterie a la Rand. In the absence of anything that even looks like a really blatant effect, it's hard to extract so much as an anecdote.

Excellent post!

As for explanation, the way I would put it is that ethics consists of hard-won wisdom from many lifetimes, which is how it is able to provide me with a safety rail against the pitfalls I have yet to encounter in my single lifetime.

I'm confused, you aren't really arguing that people hiding Jews from the Nazis should answer to the SS honestly? Sometimes honesty is unethical.

If statements I make shift a listener's priors then we can evaluate the statements I choose to make based on how much they shift the listener's priors towards which truths. This is an interesting way, to compare the decision to make different types of possible statements with lies as a special case. "Successful" lies move at least one of the listener's priors away from truth, their belief about what you believe.

Even if I'm willing to restrict myself to true statements, which in extreme cases I won't, I face the dilemma of choosing which true statements to make.

This relates to your post about the clever arguer and filtered evidence.

I'm confused, you aren't really arguing that people hiding Jews from the Nazis should answer to the SS honestly? Sometimes honesty is unethical.

Yes, I was planning to mention that today - as an illustration of when you would willfully take on the unsimplicity and unforeseen pathways of lies.

If statements I make shift a listener's priors then we can evaluate the statements I choose to make based on how much they shift the listener's priors towards which truths.

That's a dangerous sort of path to go down - the idea that anything that persuades someone of what you believe to be true must be a good argument to make, without further restriction. It doesn't just take us toward the clever arguer; it takes us into the realm of manipulating people "for their own good", using lies for the sake of what is argued to be a greater epistemic good. This is the rationalization brought to me by many of the foolish advisors.

How can ethics be judged other than by referring to their consequences? You certainly can't use ethics to judge themselves.

The idea that "the universe does not reward virtue" gets it wrong. 'Virtue' is a meaningless concept by itself; it only has meaning in terms of what the universe does. Virtue is what the universe rewards, so to speak, to the degree that we can say the universe offers rewards.

It would be more accurate to say that virtue is what works in regards to the universe.

Sometimes honesty is unethical.

"Always lie" is an ethic. Not a very evolutionarily fit ethic, nor a practical one. But it's an ethic.

Russell: "ethics consists of hard-won wisdom from many lifetimes, which is how it is able to provide me with a safety rail against the pitfalls I have yet to encounter in my single lifetime."

Yes, generations of selection for "what works" encoded in terms of principles tends to outweigh assessment within the context of an individual agent in terms of expected utility -- to the extent that the present environment is representative of the environment of adaptation. To the extent it isn't, then the best one can do is rely on the increasing weight of principles perceived hierarchically as increasingly effective over increasing scope of consequences, e.g. action on the basis of the principle known as the "law of gravity" is a pretty certain bet.

increasing weight of principles perceived hierarchically as increasingly effective over increasing scope of consequences

Ack. Could you please invent some terminology so you don't have to keep repeating this unwieldy phrase?

odf23ds: "Ack. Could you please invent some terminology so you don't have to keep repeating this unwieldy phrase?"

I'm eager for an apt idiom for the concept, and one also for "increasing coherence over increasing context."

It seems significant, and indicative of our cultural unfamiliarity -- even discomfort -- with concepts of systems, information, and evolutionary theory, that we don't have such shorthand.

But then I look at the gross misunderestimation of almost every issue of any complexity at every level of supposed sophistication of social decision-making, and then geek speak seems not so bad.

Suggestions?

the threat of insanity, which we know to be sort of an occupational hazard among AI researchers What? That sounds like sci-fi/horror writing, I've never heard of it happening in real life.

odf23ds: "Ack. Could you please invent some terminology so you don't have to keep repeating this unwieldy phrase?"

Well, there are worse things than an unwieldy phrase! Consider how many philosophers have spent entire books trying to communicate their thoughts, and still failed. Looked at that way, Jef's phrase has a very good ratio of length to precision.

For the record, I never intended to argue that any statement which shifts the audience's priors towards what I perceive to be the truth is justified.

What I was starting to get at, and I hope Eliezer will address, is how we should select which true statements to make.

What about true statements which shift at least one of the listener's priors away from the true prior? What about avoiding true statements which would improve the listener's priors?

I believe that intelligent people sometimes avoid telling lies by selectively choosing truths which manipulate someones priors.