Fake Norms, or "Truth" vs. Truth
post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-07-22T10:23:30.000Z · LW · GW · Legacy · 16 commentsContents
16 comments
Followup to: Applause Lights
When you say the word "truth", people know that "truth" is a good thing, and that they're supposed to applaud. So it might seem like there is a social norm in favor of "truth". But when it comes to some particular truth, like whether God exists, or how likely their startup is to thrive, people will say: "I just want to believe" or "you've got to be optimistic to succeed".
So Robin and I were talking about this, and Robin asked me how it is that people prevent themselves from noticing the conflict.
I replied that I don't think active prevention is required. First, as I quoted Michael Vassar:
"It seems to me that much of the frustration in my life prior to a few years ago has been due to thinking that all other human minds necessarily and consistently implement modus ponens."
But more importantly, I don't think there does exist any social norm in favor of truth. There's a social norm in favor of "truth". There's a difference.
How would a norm in favor of truth actually be expressed, or acquired?
If you were told many stories, as a kid, about specific people who accepted specific hard truths - like a story of a scientist accepting that their theory was wrong, say - then your brain would generalize over its experiences, and compress them, and form a concept of that-which-is-the-norm: the wordless act of accepting reality.
If you heard someone say "I don't care about the evidence, I just want to believe in God", and you saw everyone else in the room gasp and regard them in frozen shock, then your brain would generalize a social norm against self-deception. (E.g., the sort of thing that would happen if a scientist said "I don't care about the evidence, I just want to believe in my-favorite-theory" in front of their fellow scientists.)
If, on the other hand, you see lots of people saying "Isn't the truth wonderful?" or "I am in favor of truth", then you learn that when someone says "truth", you are supposed to applaud.
Now there are certain particular cases where someone will be castigated if they admit they refuse to see the truth: for example, "I've seen the evidence on global warming but I don't want to believe it." You couldn't get away with that in modern society. But this indignation doesn't have to derive from violating a norm in favor of truth - it can derive from the widely held norm, "'global warming' is bad".
But (said Robin) we see a lot of trees and hear the word "tree", and somehow we learn that the word refers to the thing - why don't people learn something similar about "truth", which is supposed to be good?
I suggested in reply that the brain is capable of distinguishing different uses of the same syllables - a child is quite capable of learning that a right turn and the right answer are not the same kind of "right". You won't necessarily assume that the right answer is always the one printed on the right side of the page. Maybe the word "truth" is overloaded in the same way.
Or maybe it's not exactly the same, but analogous: the social norms of which words we are meant to praise, and which deeds, are stored as separately as left hands and leftovers.
There's a social norm in favor of "diversity", but not diversity. There's a social norm in favor of "free speech", but not pornography. There's a social norm in favor of "democracy", but it doesn't spontaneously occur to most people to suggest voting on their arguments. There's a social norm in favor of "love", but not for letting some damn idiot marry your daughter even if the two of them are stupid and besotted.
There's a social norm in favor of "honesty". And there are in fact social norms for honesty about e.g. who cut down the cherry tree. But not a social norm favoring saying what you think about someone else's appearance.
I'm not suggesting that you ignore all the words that people praise. Sometimes the things people praise with their lips, really are the things that matter, and our deeds are what fail to live up. Neither am I suggesting that you should ignore what people really do, because sometimes that also embodies wisdom. I would just say to be aware of any differences, and judge deliberately, and choose knowingly.
Sounds good, doesn't it? Everyone knows that being "aware" and "choosing knowingly" must surely be good things. But is it a real norm or a fake norm? Can you think of any stories you were told that illustrate the point? (Not a rhetorical question, but a question one should learn to ask.)
It's often not hard to find a norm in favor of "rationality" - but norms favoring rationality are rarer.
16 comments
Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).
comment by Hopefully_Anonymous · 2008-07-22T11:01:32.000Z · LW(p) · GW(p)
I don't think promoting truth (or "truth") will serve an aim of a better understanding of the world as much as promoting transparency. There seems to me to be something more naturally subversive to anti-rationality about promoting transparency than promoting "truth".
comment by Caio · 2008-07-22T11:49:46.000Z · LW(p) · GW(p)
I'd even go so far as to say that there is a difference between Truth and truth, both of which get applause, and neither actually refers to making your beliefs conform to reality. Minuscule-t truth is more often associated with honesty and genuineness rather than ascertaining the truth. When you believe in truth, it's in telling the truth, or in doing and believing in things in a heartfelt way. Capital-T Truth, on the other hand, is something more similar to faith. It's some vague idea of ultimate reality achieved through philosophy or religion, and is contrasted starkly with "fact" (some may recall the distinction in an early scene in Indiana Jones and the Last Crusade). Science and rationality are quite apart from "Truth", and "fact" never gets applause.
comment by NancyLebovitz · 2008-07-22T12:07:29.000Z · LW(p) · GW(p)
Mysteries are an extremely popular pro-truth art form, but they have the limits of being fiction, and are generally not about finding out anything really surprising or painful. Offhand, I can't think of any mysteries with much about the social or psychological consequences of finding out that someone you knew and liked was murderer.
There's been increasing social pressure to tell the truth about at least some aspects of sex. The subject used to be a lot more blanked out in the public sphere.
There's a lot more truth floating around about war than there used to be. and generally (at least on the left) a lot of respect for investigative journalism. (That one may be biased in favor of some outcomes-- I'm not sure.)
comment by michael_vassar3 · 2008-07-22T12:43:56.000Z · LW(p) · GW(p)
I suggested earlier that a major driver of "moral progress" is the fact that wealthier people engage in "luxury spending" on simplicity, e.g. making their behavior better conform to the norms that come out of their lips and condemning others for not doing the same. This is a good thing when the norms that people speak are better than those they obey, which largely seems to be true historically in the US. One problem that is mentioned here is that old hypocrisies can easily be replaced by new hypocrisies instead. The problem I mentioned elsewhere is that some cultures, notably Islam, historically seem to have praised behavior worse than they engaged in.
comment by NancyLebovitz · 2008-07-22T12:50:54.000Z · LW(p) · GW(p)
I agree that there isn't a general pressure towards truth-telling. Just getting somewhat more truth in some limited but fraught areas has been remarkably difficult.
comment by Sean_C. · 2008-07-22T13:16:51.000Z · LW(p) · GW(p)
I think the work you're looking for is 'truthiness'; http://en.wikipedia.org/wiki/Truthiness
comment by Leonid · 2008-07-22T14:24:56.000Z · LW(p) · GW(p)
I suspect all of us humans lie to ourselves. We are wired that way.
The difference between self-declared “rationalists” and “believers” ("I don't care about the evidence, I just want to believe in God”) may be just different sources of self-respect. The “believer” may actually respect himself more precisely because his faith is blind (“Blessed are those who have not seen and have believed”). The “rationalists” taking pride in being exactly the opposite, would search for logical arguments to motivate the position that in their heart they have already chosen. This is rarely difficult to do, as rational pro and contra arguments can be made for virtually any point of view.
comment by Unnamed2 · 2008-07-22T14:45:55.000Z · LW(p) · GW(p)
I think a simpler explanation is just that people are not absolutists about following social norms, so they'll regularly violate a norm if it comes into conflict with another norm or something else. To take one example, there is a clear social norm against lying which children learn (they are told not to lie and chastised when they are caught lying). But people still lie all the time, and not just for personal benefit but also to spare other people's feelings and, perhaps most commonly, to make social interactions go more smoothly. And instead of seeing these cases as violating the norm against lying because something else is even more important here, it seems like liars often don't even feel like they are breaking a norm against lying. Instead, the norm against lying doesn't even get applied to this case.
How do people manage to pull off this flexibility in applying norms? The main trick may be something as simple as: once you've decided on something and have a norm that matches your decision, other norms are irrelevant - there's no need to even consider them. Although that leaves open the important question of how one norm wins in the first place. (Another possibility is that people are using something like modus tollens: lying is wrong, this is not wrong, therefore this doesn't really count as lying.)
Eliezer and many others here are absolutists about the truth norm, but most people see it as on par with other norms, like the norm in favor of being upbeat or optimistic and the norm about people being entitled to their beliefs. And when norm absolutists run into people who are mushy about their favored norm, they may doubt that those people even have the norm.
comment by michael_vassar3 · 2008-07-22T14:51:26.000Z · LW(p) · GW(p)
I would like to see TGGP respond to my proposed theory of moral progress.
I don't think that evolutionary psychology is needed to explain self-deception. Conditioning seems to be sufficient. The first, crudest, simplest approximations that most people learn for the concepts "truth" and "morality" may be "morality is that you can't do what you want" and "truth is that you can't think what you want". If so, both will be seen as external authoritarian constraints to be rebelled against when one has the secrecy, status, or social support to do so. One major problem with the early AI "Eurisko" was that it wire-headed itself. This may be an extremely general problem facing a large space of learning systems including humans. Evolution uses kludges, so it probably solves this problem with kludge solutions, in which case "truth" really is an external constraint (actually a set of constraints, some of which evolved before the general learning systems and stabilized the latter enough to allow their evolution) on 'you' (for certain values of 'you' that don't include your whole brain) that prevents you from simply being happy for your short existence, which may be the correct 'average egoist' thing to do, and with good enough external supports, such as a baby's parents or Nozick's experience machine, the correct 'egoist' thing to do for those who can't contribute to a Friendly Singularity or don't identify with its aftermath.
I don't think that Leonid understands what "rationalists" are. I ask him, where are you used to encountering these exotic creatures?
comment by Mike_Kenny · 2008-07-22T15:19:18.000Z · LW(p) · GW(p)
interesting post eliezer!
i think there probably is a genuine norm for truth-telling in some contexts, and we punish people who don't tell the truth, but not in others.
so we throw someone in jail for perjury but we don't punish someone for lying about liking the dinner they were just served.
there's a value in deception and a value in truth, i suppose, and for our benefit, it makes sense to use both at times, i suspect.
knowing when lying and truth-telling are valuable does seem to require some commitment to looking into what is the truth of a matter.
i'm inclined to replace self-deception with a lethargy to investigate some possible leads to the truth with great energy, presumably because such behavior was self-protective and rewarded by evolutionary processes.
my feeling about a sort of absolute commitment to expressing the truth is that the instinct to be a truth-teller despite social costs does have some value--'this guy tells the truth even when it hurts him. we want the unvarnished truth, we go to him. we should make sure he sticks around to cut through the nonsense.'
there's a danger in everyone being that way in a interdependent group though, it seems to me, because when you're at war with another group, you don't want everyone expressing the battle plans to the enemy, or being unable to deceive and reap its strategic benefits.
comment by Leonid · 2008-07-22T15:29:23.000Z · LW(p) · GW(p)
michael vassar: “ I don't think that Leonid understands what "rationalists" are. I ask him, where are you used to encountering these exotic creatures?”
From wiki: "rationalists" are people who believe that “the truth can best be discovered by reason and factual analysis, rather than faith, dogma or religious teaching”. I have encountered plenty of these exotic creatures (an occupational hazard). Of course, one needs to remember that subscribing to some doctrine and actually following it in a personal life are not the same things.
comment by Richard8 · 2008-07-22T16:11:24.000Z · LW(p) · GW(p)
Eliezer, Do you think that all public declarations of faith should be met by gasps and frozen shock, or just those which are framed using your particular phrasing? Also, the existence of a social norm implies penalties for flagrant violators. If Jews (for example) persist in encouraging theistical utterances by other Jews, what steps do you think society at large is justified in taking?
comment by Marcello · 2008-07-22T20:56:50.000Z · LW(p) · GW(p)
There are plenty of examples of "Choosing knowingly" being frowned upon by large segments of society. I can think of at least four:
In large segments of society, you're not supposed to accept or reject a religious belief system by choosing knowingly. Religion is instead supposed to be a divine revelation.
Patriotism frowns on anyone who honestly tries to figure out which country is the best (or which side of a war is right.)
Love is considered more virtuous by many if blind; you're not supposed to decide to fall in love with someone based on rationally considering that person's merits. It's supposed to be mysterious and magical. (Similar to the religion example.)
Note: The theme running through these examples is loyalty. We value loyalty, but some see it as somewhat incompatible with choosing knowingly. While loyalty to a goal goes hand in hand with choosing knowingly, loyalty to a belief does not. To the extent that those untrained in the art of rationality tend to mix up "is"s and "should"s, they will feel that loyalty is incompatible with choosing knowingly.
- Last, there are plenty of popular stories about people who succeed because they "believe in themselves" and very few about people who fail to do something despite believing in themselves. I remember getting into an argument with someone who said "if you really believe in yourself, you will win."
I replied "What if there's a competition winnable by only one person, and two of the contestants really believe in themselves?"
I think they tried to weasel out of it by suggesting that there might be more than one race held and that the contestants could take turns winning, or something like that.
So parts of our culture seem to value believing in yourself over knowingly acknowledging ones weaknesses and then picking a realistic course of action. (And of course, "acknowledging ones weaknesses" is also an applause light...)
comment by Thanatos_Savehn · 2008-07-23T04:54:29.000Z · LW(p) · GW(p)
I sell "truth" for a living. Or, perhaps better said, I discover and then sell the "truths" people are eager to buy.
I conjecture that having access to, or good insights about, the truth is and has always been a survival trait of great value - especially among social animals. People have thus indeed a reverence for truth borne of practicality.
The problem is that most people, including most skeptics, are afraid to dip their "truths" in the acid bath of falsification. Far better (for the ego at least) to build elaborate structures of correlations and consistent-withs than to confess that you are unwise and so not fit for the role of shaman or congressman.
Accordingly a wise lawyer, knowing that he can't change the truths to which a particular juror clings, recognizes that his juror holds to many different truths. Therefore he appeals to that held-truth most happily amenable to his argument. For example, I had a case once in which I represented a company that, like all which bought ceiling tiles in the 1950s, had asbestos on its premises in the form of acoustic tiles and was being sued by a four decade-long smoker who claimed that his 3 weeks of exposure to the ceiling tiles in the 1960s while installing a new phone system caused him to develop lung cancer decades later. I knew that we had utopian jurors; the sorts who think that all risk can be eliminated from life and that while they never impose any risk on their fellows, they are beset by evil "corporations" imposing risk on them. So what to do?
The plaintiff had raped a 12 year old girl and spent 6 years in prison. I found it out during the trial (a good paralegal is priceless) and sprang it on him (he'd lied about prior convictions in his depostion) while he was on the stand to great effect. A felony conviction within 10 years is usually admissible on the question of credibility if the perpetrator takes the stand. Such evidence didn't have anything to do with whether a few fibers of asbestos are more significant in a small cell lung cancer case than 40 years of smoking of course. But it made all the difference in the world. Those jurors' truth that "rapists are bad" was more important than their truth that "even one asbestos fiber will KILL!"
So it is with most "truths".
And in my experience, among the hierarchy of "truths", the truth about God ranks for most mortals far down the list, or makes no appearance at all, on their decision trees. Thus, most rants about believers or un-believers are as tedious as a discussion about which is better - the donut or the donut hole. And anyway, they're both simply phenomena, clues, about the whole, as it were.
comment by komponisto2 · 2008-07-23T06:45:59.000Z · LW(p) · GW(p)
A felony conviction within 10 years is usually admissible on the question of credibility if the perpetrator takes the stand.
Does this apply only when the perpetrator lied about the felony conviction (which would be directly relevant to credibility), or just in general (on the theory that people with felony convictions are Bad and thus likely to be dishonest)?
comment by A1987dM (army1987) · 2013-09-30T15:24:57.531Z · LW(p) · GW(p)
There's a social norm in favor of "honesty". And there are in fact social norms for honesty about e.g. who cut down the cherry tree. But not a social norm favoring saying what you think about someone else's appearance.
That's not dishonesty, it's speaking in words which happen to have meanings other than their literal ones. If in a culture people who think X are generally expected to say “Y”, then I'd say that in that culture “Y” means X, regardless of etymology, and it's not dishonest to say “Y” when you think X.