Dialogue on Appeals to Consequences

post by jessicata (jessica.liu.taylor) · 2019-07-18T02:34:52.497Z · score: 27 (19 votes) · LW · GW · 34 comments

This is a link post for https://unstableontology.com/2019/07/18/dialogue-on-appeals-to-consequences/

[note: the following is essentially an expanded version of this LessWrong comment on whether appeals to consequences are normative in discourse. I am exasperated that this is even up for debate, but I figure that making the argumentation here explicit is helpful]

Carter and Quinn are discussing charitable matters in the town square, with a few onlookers.

Carter: "So, this local charity, People Against Drowning Puppies (PADP), is nominally opposed to drowning puppies."

Quinn: "Of course."

Carter: "And they said they'd saved 2170 puppies last year, whereas their total spending was $1.2 million, so they estimate they save one puppy per $553."

Quinn: "Sounds about right."

Carter: "So, I actually checked with some of their former employees, and if what they say and my corresponding calculations are right, they actually only saved 138 puppies."

Quinn: "Hold it right there. Regardless of whether that's true, it's bad to say that."

Carter: "That's an appeal to consequences, well-known to be a logical fallacy."

Quinn: "Is that really a fallacy, though? If saying something has bad consequences, isn't it normative not to say it?"

Carter: "Well, for my own personal decisionmaking, I'm broadly a consequentialist, so, yes."

Quinn: "Well, it follows that appeals to consequences are valid."

Carter: "It isn't logically valid. If saying something has bad consequences, that doesn't make it false."

Quinn: "But it is decision-theoretically compelling, right?"

Carter: "In theory, if it could be proven, yes. But, you haven't offered any proof, just a statement that it's bad."

Quinn: "Okay, let's discuss that. My argument is: PADP is a good charity. Therefore, they should be getting more donations. Saying that they didn't save as many puppies as they claimed they did, in public (as you just did), is going to result in them getting fewer donations. Therefore, your saying that they didn't save as many puppies as they claimed to is bad, and is causing more puppies to drown."

Carter: "While I could spend more effort to refute that argument, I'll initially note that you only took into account a single effect (people donating less to PADP) and neglected other effects (such as people having more accurate beliefs about how charities work)."

Quinn: "Still, you have to admit that my case is plausible, and that some onlookers are convinced."

Carter: "Yes, it's plausible, in that I don't have a full refutation, and my models have a lot of uncertainty. This gets into some complicated decision theory and sociological modeling. I'm afraid we've gotten sidetracked from the relatively clear conversation, about how many puppies PADP saved, to a relatively unclear one, about the decision theory of making actual charity effectiveness clear to the public."

Quinn: "Well, sure, we're into the weeds now, but this is important! If it's actually bad to say what you said, it's important that this is widely recognized, so that we can have fewer... mistakes like that."

Carter: "That's correct, but I feel like I might be getting trolled. Anyway, I think you're shooting the messenger: when I started criticizing PADP, you turned around and made the criticism about me saying that, directing attention against PADP's possible fraudulent activity."

Quinn: "You still haven't refuted my argument. If you don't do so, I win by default."

Carter: "I'd really rather that we just outlaw appeals to consequences, but, fine, as long as we're here, I'm going to do this, and it'll be a learning experience for everyone involved. First, you said that PADP is a good charity. Why do you think this?"

Quinn: "Well, I know the people there and they seem nice and hardworking."

Carter: "But, they said they saved over 2000 puppies last year, when they actually only saved 138, indicating some important dishonesty and ineffectiveness going on."

Quinn: "Allegedly, according to your calculations. Anyway, saying that is bad, as I've already argued."

Carter: "Hold up! We're in the middle of evaluating your argument that saying that is bad! You can't use the conclusion of this argument in the course of proving it! That's circular reasoning!"

Quinn: "Fine. Let's try something else. You said they're being dishonest. But, I know them, and they wouldn't tell a lie, consciously, although it's possible that they might have some motivated reasoning, which is totally different. It's really uncivil to call them dishonest like that. If everyone did that with the willingness you had to do so, that would lead to an all-out rhetorical war..."

Carter: "God damn it. You're making another appeal to consequences."

Quinn: "Yes, because I think appeals to consequences are normative."

Carter: "Look, at the start of this conversation, your argument was that saying PADP only saved 138 puppies is bad."

Quinn: "Yes."

Carter: "And now you're in the course of arguing that it's bad."

Quinn: "Yes."

Carter: "Whether it's bad is a matter of fact."

Quinn: "Yes."

Carter: "So we have to be trying to get the right answer, when we're determining whether it's bad."

Quinn: "Yes."

Carter: "And, while appeals to consequences may be decision theoretically compelling, they don't directly bear on the facts."

Quinn: "Yes."

Carter: "So we shouldn't have appeals to consequences in conversations about whether the consequences of saying something is bad."

Quinn: "Why not?"

Carter: "Because we're trying to get to the truth."

Quinn: "But aren't we also trying to avoid all-out rhetorical wars, and puppies drowning?"

Carter: "If we want to do those things, we have to do them by getting to the truth."

Quinn: "The truth, according to your opinion-"

Carter: "God damn it, you just keep trolling me, so we never get to discuss the actual facts. God damn it. Fuck you."

Quinn: "Now you're just spouting insults. That's really irresponsible, given that I just accused you of doing something bad, and causing more puppies to drown."

Carter: "You just keep controlling the conversation by OODA looping faster than me, though. I can't refute your argument, because you appeal to consequences again in the middle of the refutation. And then we go another step down the ladder, and never get to the truth."

Quinn: "So what do you expect me to do? Let you insult well-reputed animal welfare workers by calling them dishonest?"

Carter: "Yes! I'm modeling the PADP situation using decision-theoretic models, which require me to represent the knowledge states and optimization pressures exerted by different agents (both conscious and unconscious), including when these optimization pressures are towards deception, and even when this deception is unconscious!"

Quinn: "Sounds like a bunch of nerd talk. Can you speak more plainly?"

Carter: "I'm modeling the actual facts of how PADP operates and how effective they are, not just how well-liked the people are."

Quinn: "Wow, that's a strawman."

Carter: "Look, how do you think arguments are supposed to work, exactly? Whoever is best at claiming that their opponent's argumentation is evil wins?"

Quinn: "Sure, isn't that the same thing as who's making better arguments?"

Carter: "If we argue by proving our statements are true, we reach the truth, and thereby reach the good. If we argue by proving each other are being evil, we don't reach the truth, nor the good."

Quinn: "In this case, though, we're talking about drowning puppies. Surely, the good in this case is causing fewer puppies to drown, and directing more resources to the people saving them."

Carter: "That's under contention, though! If PADP is lying about how many puppies they're saving, they're making the epistemology of the puppy-saving field worse, leading to fewer puppies being saved. And, they're taking money away from the next-best-looking charity, which is probably more effective if, unlike PADP, they're not lying."

Quinn: "How do you know that, though? How do you know the money wouldn't go to things other than saving drowning puppies if it weren't for PADP?"

Carter: "I don't know that. My guess is that the money might go to other animal welfare charities that claim high cost-effectiveness."

Quinn: "PADP is quite effective, though. Even if your calculations are right, they save about one puppy per $10,000. That's pretty good."

Carter: "That's not even that impressive, but even if their direct work is relatively effective, they're destroying the epistemology of the puppy-saving field by lying. So effectiveness basically caps out there instead of getting better due to better epistemology."

Quinn: "What an exaggeration. There are lots of other charities that have misleading marketing (which is totally not the same thing as lying). PADP isn't singlehandedly destroying anything, except instances of puppies drowning."

Carter: "I'm beginning to think that the difference between us is that I'm anti-lying, whereas you're pro-lying."

Quinn: "Look, I'm only in favor of lying when it has good consequences. That makes me different from pro-lying scoundrels."

Carter: "But you have really sloppy reasoning about whether lying, in fact, has good consequences. Your arguments for doing so, when you lie, are made of Swiss cheese."

Quinn: "Well, I can't deductively prove anything about the real world, so I'm using the most relevant considerations I can."

Carter: "But you're using reasoning processes that systematically protect certain cached facts from updates, and use these cached facts to justify not updating. This was very clear when you used outright circular reasoning, to use the cached fact that denigrating PADP is bad, to justify terminating my argument that it wasn't bad to denigrate them. Also, you said the PADP people were nice and hardworking as a reason I shouldn't accuse them of dishonesty... but, the fact that PADP saved far fewer puppies than they claimed actually casts doubt on those facts, and the relevance of them to PADP's effectiveness. You didn't update when I first told you that fact, you instead started committing rhetorical violence against me."

Quinn: "Hmm. Let me see if I'm getting this right. So, you think I have false cached facts in my mind, such as PADP being a good charity."

Carter: "Correct."

Quinn: "And you think those cached facts tend to protect themselves from being updated."

Carter: "Correct."

Quinn: "And you think they protect themselves from updates by generating bad consequences of making the update, such as fewer people donating to PADP."

Carter: "Correct."

Quinn: "So you want to outlaw appeals to consequences, so facts have to get acknowledged, and these self-reinforcing loops go away."

Carter: "Correct."

Quinn: "That makes sense from your perspective. But, why should I think my beliefs are wrong, and that I have lots of bad self-protecting cached facts?"

Carter: "If everyone were as willing as you to lie, the history books would be full of convenient stories, the newspapers would be parts of the matrix, the schools would be teaching propaganda, and so on. You'd have no reason to trust your own arguments that speaking the truth is bad."

Quinn: "Well, I guess that makes sense. Even though I lie in the name of good values, not everyone agrees on values or beliefs, so they'll lie to promote their own values according to their own beliefs."

Carter: "Exactly. So you should expect that, as a reflection to your lying to the world, the world lies back to you. So your head is full of lies, like the 'PADP is effective and run by good people' one."

Quinn: "Even if that's true, what could I possibly do about it?"

Carter: "You could start by not making appeals to consequences. When someone is arguing that a belief of yours is wrong, listen to the argument at the object level, instead of jumping to the question of whether saying the relevant arguments out loud is a good idea, which is a much harder question."

Quinn: "But how do I prevent actually bad consequences from happening?"

Carter: "If your head is full of lies, you can't really trust ad-hoc object-level arguments against speech, like 'saying PADP didn't save very many puppies is bad because PADP is a good charity'. You can instead think about what discourse norms lead to the truth being revealed, and which lead to it being obscured. We've seen, during this conversation, that appeals to consequences tend to obscure the truth. And so, if we share the goal of reaching the truth together, we can agree not to do those."

Quinn: "That still doesn't answer my question. What about things that are actually bad, like privacy violations?"

Carter: "It does seem plausible that there should be some discourse norms that protect privacy, so that some facts aren't revealed, if such norms have good consequences overall. Perhaps some topics, such as individual people's sex lives, are considered to be banned topics (in at least some spaces), unless the person consents."

Quinn: "Isn't that an appeal to consequences, though?"

Carter: "Not really. Deciding what privacy norms are best requires thinking about consequences. But, once those norms have been decided on, it is no longer necessary to prove that privacy violations are bad during discussions. There's a simple norm to appeal to, which says some things are out of bounds for discussion. And, these exceptions can be made without allowing appeals to consequences in full generality."

Quinn: "Okay, so we still have something like appeals to consequences at the level of norms, but not at the level of individual arguments."

Carter: "Exactly."

Quinn: "Does this mean I have to say a relevant true fact, even if I think it's bad to say it?"

Carter: "No. Those situations happen frequently, and while some radical honesty practitioners try not to suppress any impulse to say something true, this practice is probably a bad idea for a lot of people. So, of course you can evaluate consequences in your head before deciding to say something."

Quinn: "So, in summary: if we're going to have suppression of some facts being said out loud, we should have that through either clear norms designed with consequences (including consequences for epistemology) in mind, or individuals deciding not to say things, but otherwise our norms should be protecting true speech, and outlawing appeals to consequences."

Carter: "Yes, that's exactly right! I'm glad we came to agreement on this."

34 comments

Comments sorted by top scores.

comment by jkaufman · 2019-07-19T11:18:36.964Z · score: 40 (11 votes) · LW · GW

The motivating example for this post is whether you should say "So, I actually checked with some of their former employees, and if what they say and my corresponding calculations are right, they actually only saved 138 puppies", with Quinn arguing that you shouldn't say it because saying it has bad consequences. The problem is, saying this has very clearly good consequences, which means trying to use it as a tool for figuring out what you think of appeals to consequences sets up your intuitions to confuse you.

(It has clearly good consequences because "how much money goes to PADP right now" is far less import than "building a culture of caring about the actual effectiveness of organizations and truly trying to find/make the best ones". Plus if, say, Animal Charity Evaluators trusted this higher number of puppies saved and it had lead them to recommend PADP as I've if their top charities, that that would mean displacing funds that could have gone to more effective animal charities. The whole Effective Altruism project is about trying to figure out how to get the biggest positive impact, and you can't do this if you declare discussing negative information about organizations off limits.)

The post would be a lot clearer if it had a motivating example that really did have bad consequences, all things considered. As a person who's strongly pro transparency is hard for me to come up with cases, but there are still contexts where I think it's probably the case. What if Carter were a researcher who had run a small study on a new infant vaccine and seen elevated autism rates on the experimental group. There's an existing "vaccines cause autism" meme that is both very probably wrong and very probably harmful, which means Carter should be careful about messaging for their results. Good potential outcomes include:

  • Carter's experiment is replicated, confirmed, and the vaccine is not rolled out.

  • Carter's experiment fails to replicate, researchers look into it more, and discover that there was a problem in the initial experiment / in the replication / they need more data / etc

Bad potential outcomes include:

  • Headlines that say "scientists finally admit vaccines do cause autism"

Because of the potential harmful consequences of handling this poorly, Carter should be careful about how they talk about their results and to who. Trying to get funding to scale up the experiment, making sure the FDA is aware, letting other researchers know, etc, all are beneficial and have good consequences. Going to the mainstream media with a controversial sell-lots-of-papers story, by contrast, would have predictably bad consequences.

When talking with friends or within your field it's hard to think of cases where you shouldn't just say the interesting thing you've found, while with larger audiences and in less truth-oriented cultures you need to start being more careful.

EDIT: expanded this into https://www.jefftk.com/p/appeals-to-consequences

comment by Kaj_Sotala · 2019-07-19T11:54:09.447Z · score: 18 (6 votes) · LW · GW

The post would be a lot clearer if it had a motivating example that really did have bad consequences, ask things considered.

The extreme case would be a scientific discovery which enabled anyone to destroy the world, such as the supernova thing in Three Worlds Collide [LW · GW] or the thought experiment that Bostrom discusses in The Vulnerable World Hypothesis:

So let us consider a counterfactual history in which Szilard invents nuclear fission and realizes that a nuclear bomb could be made with a piece of glass, a metal object, and a battery arranged in a particular configuration. What happens next? Szilard becomes gravely concerned. He sees that his discovery must be kept secret at all costs. But how. His insight is bound to occur to others. He could talk to a few of his physicist friends, the ones most likely to stumble upon the idea, and try to persuade them not to publish anything on nuclear chain reactions or on any of the reasoning steps leading up to the dangerous discovery. (That is what Szilard did in actual history.)

[...] Soon, figuring out how to initiate a nuclear chain reaction with pieces of metal, glass, and electricity will no longer take genius but will be within reach of any STEM student with an inventive mindset.

comment by jessicata (jessica.liu.taylor) · 2019-07-19T16:17:28.963Z · score: 5 (3 votes) · LW · GW

Note, I'm not arguing for a positive obligation to always inform everyone (see last few lines of dialogue), it's important for people to use their discernment sometimes.

But, in the case you mentioned, if your study really did find that a vaccine caused autism, by the logic of the dialogue, that casts doubt on the "vaccines don't cause autism and antivaxxers are wrong and harmful" belief. (Maybe you're not the only one who has found that vaccines cause autism, and other researchers are hiding it too). So, you should at least update that belief on the new evidence before evaluating consequences. (It could be that, even after considering this, the new study is likely to be a fluke, and discerning researchers will share the new study in an academic community without going to the press)

comment by jkaufman · 2019-07-19T16:55:50.955Z · score: 22 (7 votes) · LW · GW

My main objection is that the post is built around a case where Quinn is very wrong in their initial "bad consequences" claim, and that this leads people to have misleading intuitions. I was trying to propose an alternative situation where the "bad consequences" claim was true or closer to true, but where Quinn would still be wrong to suggest Carter shouldn't describe what they'd found.

(Also, for what it's worth, I find the Quinn character's argumentative approach very frustrating to read. This makes it hard to take anything that character describes seriously.)

comment by romeostevensit · 2019-07-18T07:10:19.140Z · score: 32 (8 votes) · LW · GW

The bar that is set for appeals to consequences imply the sort of equilibrium world you'll end up in. Erring on the side of higher is better, because it is hard to go the other way because epistemic standards tend to slide in the face of local incentives.

I also want to note an argumentative tactic that occurs on the tacit level whereby people will push you into a state where you need to expend more energy on average per truth bit than they do, so they eventually win by attrition. Related to evaporative cooling. The subjective experience of this feels like talking to the cops. You sense that no big wins are available (because they have their bottom line) but big losses are, so you stop talking. If you've encountered this dynamic, you recognize things like this

> "You still haven't refuted my argument. If you don't do so, I win by default."

as part of the supporting framework for the dynamic and it will make you very angry...which others will then use as part of the dynamic which makes you angry which......

comment by Said Achmiz (SaidAchmiz) · 2019-07-18T05:24:15.993Z · score: 28 (11 votes) · LW · GW

Well, I certainly agree with the position you’re defending. Yet I can’t help but feel that the arguments in the OP lack… a certain concentrated force, which I feel this topic greatly deserves.

Without disagreeing, necessarily, with anything you say, here is my own attempt, in two (more or less independent) parts.

The citadel of truth

If the truth is precious, its pursuit must be unburdened by such considerations as “what will happen if we say this”. This is impractical, in the general case. You may not be interested in consequences, after all, but the consequences are quite interested in you…

There is, however, one way out of the quagmire of consequential anxiety. Let there be a place around which a firewall of epistemology is erected. Let appeals to consequences outside that citadel, be banned within its walls. Let no one say: “if we say such a thing, why, think what might happen, out there, in the wider world!”. Yes, if you say this thing out there, perhaps unfortunate consequences may follow out there. But we are not speaking out there; so long as we speak in here, to each other, let us consider it irrelevant what effects our words may produce upon the world outside. In here, we concern ourselves only with truth. All that we do and say, within the citadel, serves only truth.

Any among us who have something to protect, in the world beyond the citadel, may wish to take the truths we find, and apply them to that outside world, and discuss these things with others who feel as they do. In these discussions, of plans and strategies for acting upon the wider world, the consequences of their words, for that world, may be of the utmost importance. But if so, to have such discussions, these planners will have to step outside the citadel’s walls. To talk in such a way in here—to speak, and to hold yourself and others to considering the consequences of their words upon the world—is to violate the citadel’s rule: that all that we do and say within, serves truth, and only truth.

Without evaluation, consequences have no meaning

The one comes to you and says: if you say such a thing, why, this-and-such will happen!

Well, and what of it? For this to be compelling, it is not enough that the consequences of speech be a certain way, but that you evaluate them a certain way. To find that “this-and-such will happen if you say that” is a reason for not saying it, you must evaluate the given consequence as negative, and furthermore you must weigh it against the other consequences of your speech act—all the other consequences—and judge that this one downside tilts the balance, in favor of silence.

And now suppose one comes to me, and says: Said, this thing you said—consider the consequences of saying it! They are bad. Well, perhaps they are. But consider, I respond, the consequences of allowing you, concerned citizen, to convince me, by this argument of yours, to keep silent. They include corruption of the truth, and the undermining of the search for it, and distortion of the accurate beliefs of everyone I speak to, and my own as well. These, too, are consequences. And I evaluate these effects to have so negative a value, that, short of either the inescapable annihilation of the human race (or outcomes even more dire), or serious personal harm to me or those close to me, no unpleasant consequence you might threaten me with could possibly compare. So I thank you—I continue—for your most sincere concern, but your admonition cannot move me.

In short: it is the case, I find, in most such arguments-from-consequences, that anyone who may be moved by them, does not value truth—does not, at any rate, value it enough that they may rightly be admitted to any circle of truth-seekers which pretends really to be serious in the pursuit of its goal.

comment by Scott Alexander (Yvain) · 2019-07-19T20:28:53.423Z · score: 25 (8 votes) · LW · GW

Instead of Quinn admitting lying is sometimes good, I wish he had said something like:

“PADP is widely considered a good charity by smart people who we trust. So we have a prior on it being good. You’ve discovered some apparent evidence that it’s bad. So now we have to combine the prior and the evidence, and we end up with some percent confidence that they’re bad.
If this is 90% confidence they’re bad, go ahead. What if it’s more like 55%? What’s the right action to take if you’re 55% sure a charity is incompetent and dishonest (but 45% chance you misinterpreted the evidence)? Should you call them out on it? That’s good in the world where you’re right, but might disproportionately tarnish their reputation in the world where they're wrong. It seems like if you’re 55% sure, you have a tough call. You might want to try something like bringing up your concerns privately with close friends and only going public if they share your opinion, or asking the charity first and only going public if they can’t explain themselves. Or you might want to try bringing up your concerns in a nonconfrontational way, more like ‘Can anyone figure out what’s going on with PADP’s math?’ rather than ‘PADP is dishonest’. After this doesn’t work and lots of other people confirm your intuitions of distrust, then your confidence reaches 90% and you start doing things more like shouting ‘PADP is dishonest’ from the rooftops.
Or maybe you’ll never reach 90% confidence. Many people think that climate science is dishonest. I don’t doubt many of them are reporting their beliefs honestly - that they’ve done a deep investigation and that’s what they’ve concluded. It’s just that they’re not smart, informed, or rational enough to understand what’s going on, or to process it in an unbiased way. What advice would you give these people about calling scientists out on dishonesty - again given that rumors are powerful things and can ruin important work? My advice to them would be to consider that they may be overconfident, and that there needs to be some intermediate ‘consider my own limitations and the consequences of my irreversible actions’ step in between ‘this looks dishonest to me’ and ‘I will publicly declare it dishonest’. And that step is going to look like an appeal to consequences, especially if the climate deniers are so caught up in their own biases that they can't imagine they might be wrong.
I don’t want to deny that calling out apparent dishonesty when you’re pretty sure of it, or when you’ve gone through every effort you can to check it and it still seems bad, will sometimes (maybe usually) be the best course, but I don’t think it’s as simple as you think.”

...and seen what Carter answered.

comment by Wei_Dai · 2019-07-19T22:53:41.037Z · score: 7 (4 votes) · LW · GW

Part of this is pretty close to what I wrote [LW · GW] in the actual debate. The part about climate science is new though and I'd like to see a response to it.

comment by dxu · 2019-07-25T17:44:06.908Z · score: 11 (3 votes) · LW · GW

The part about climate science seems like a pretty bog-standard outside view argument, which in turn means I find it largely uncompelling. Yes, there are people who are so stupid, they can only be saved from their own stupidity by executing an epistemic maneuver that works regardless of the intelligence of the person executing it. This does not thereby imply that everyone should execute the same maneuver, including people who are not that stupid, and therefore not in need of saving. If someone out there is so incompetent that they mistakenly perceive themselves as competent, then they are already lost, and the fact that an illegal (from the perspective of normative probability theory) epistemic maneuver exists which would save them if they executed it, does not thereby make that maneuver a normatively good move. (And even if it were, it's not as though the people who would actually benefit from said maneuver are going to execute it--the whole reason that such people are loudly, confidently mistaken is that they don't take the outside view seriously.)

In short: there is simply no principled justification for modesty-based arguments [LW · GW], and--though it may be somewhat impolite to say--I agree with Eliezer that people who find such arguments compelling are actually being influenced by social modesty norms (whether consciously or unconsciously), rather than any kind of normative judgment. Based on various posts that Scott has written in the past, I would venture to say that he may be one of those people.

comment by Dagon · 2019-07-18T16:27:27.223Z · score: 16 (6 votes) · LW · GW

I think this is strawmanning the appeal to consequences argument, by mixing up private beliefs and public statements, and by ending with a pretty superficial agreement on rule-consequentialism without exploring how to pick which rules (among one for improving private beliefs, one for sharing relevant true information and one for suppressing harmful information) applies.

The participants never actually attempt to resolve the truth about puppies saved per dollar, calling the whole thing into question - both whether their agreement is real and whether it's the right thing. Many of these discussions should include a recitation of [ https://wiki.lesswrong.com/wiki/Litany_of_Tarski ], and a direct exploration whether it's beliefs (private) or publication (impacting presumed-less-rational agents) that is at issue.

In any case, appeals to consequences at the meta/rule level still HAS to be grounded in appeals to consequences at the actual object consequence level. A rule that has so many exceptions that it's mostly wrong is actively harmful. My objection to the objection to "appeal to consequences" is that the REAL objection is to bad epistemology of consequence prediction, not to the desire to predict consequences.

In a completely separate direction, consequences of speech acts in public/group settings are WAY more complicated than epistemic consequences of a truth-seeking discussion among a small group of fairly close rationalist-inclined friends. Both different rules/defaults/norms apply, and different calculations of consequences of specific speech actions are made.

All that said, I prefer norms that lean toward truth-telling and truth-seeking, and it makes me suspicious when that is at odds with consequences of speech acts. I have a higher standard of evidence for my consequence predictions for lying than I have for withholding relevant facts than I have for truth-telling.


comment by juliawise · 2019-07-19T14:24:13.385Z · score: 14 (5 votes) · LW · GW

[speaking for myself, not for any organization]

If this is an allegory against appeals to consequences generally, well and good.

If there's some actual question about whether wrong cost effectiveness numbers are being promoted, could people please talk about those numbers specifically so we can all have a try at working out if that's really going on? E.g. this post made a similar claim to what's implied in this allegory, but it was helpful that it used concrete examples so people could work out whether they agreed (and, in that case, identify factual errors).

comment by jessicata (jessica.liu.taylor) · 2019-07-19T16:13:21.486Z · score: 5 (3 votes) · LW · GW

This is an allegory. While I didn't have any particular real-world example in mind, my dialogue-generation was influenced by a time I had seen appeals to consequences in EA; see EA Has A Lying Problem and this comment thread. So this was one of the more salient cases of a plausible moral case for shutting down true speech.

comment by PeterMcCluskey · 2019-07-19T19:01:55.040Z · score: 13 (3 votes) · LW · GW

Quinn: “Hold it right there. Regardless of whether that’s true, it’s bad to say that.”

Carter: “That’s an appeal to consequences, well-known to be a logical fallacy.”

The link in Carter's statement leads to a page that clearly contradicts Carter's claim:

In logic, appeal to consequences refers only to arguments that assert a conclusion's truth value (true or false) without regard to the formal preservation of the truth from the premises; appeal to consequences does not refer to arguments that address a premise's consequential desirability (good or bad, or right or wrong) instead of its truth value.

comment by jkaufman · 2019-07-19T20:37:25.267Z · score: 10 (2 votes) · LW · GW

It sounds to me like Jessica is using "appeal to consequences" expansively to include not just "X has bad consequences so you should not believe X" to "saying X has bad consequences so you should not say X"?

comment by jessicata (jessica.liu.taylor) · 2019-07-19T21:22:52.292Z · score: 10 (2 votes) · LW · GW

Yes. In practice, if people are discouraged from saying X on the basis that it might be bad to say it, then the discourse goes on believing not-X. So, the discourse itself makes an invalid step that's analogous to an appeal to consequences "if it's bad for us to think X is true then it's false".

comment by Dagon · 2019-07-19T21:38:46.184Z · score: 5 (2 votes) · LW · GW

Be careful with unstated assumptions about belief aggregation. "the discourse" doesn't have beliefs. People have beliefs, and discourse is one of the mechanisms for sharing and aligning those beliefs. It helps a lot to give names to people you're worried about, to make it super-clear whether you're talking about your beliefs, your current conversational partner's beliefs, or beliefs of other people who hear a summary from one of you.

If Alice discourages Bob from saying X, then Charlie might go on believing not-X. This is a very different concern from Bob being worried about believing a false not-X if not allowed to discuss the possibility. Both concerns are valid, IMO, but they have different thresholds of importance and different trade-offs to make in resolution..

comment by jessicata (jessica.liu.taylor) · 2019-07-19T21:42:18.051Z · score: 6 (3 votes) · LW · GW

In a math conversation, people are going to say and possibly write down a bunch of beliefs, and make arguments that some beliefs follow from each other. The conversation itself could be represented as a transcript of beliefs and arguments. The beliefs in this transcript are what I mean by "the discourse's beliefs".

comment by Evan_Gaensbauer · 2019-07-25T09:22:44.074Z · score: 6 (5 votes) · LW · GW

Summary: I'm aware of a lot of examples of real debates that inspired this dialogue. It seems in those real cases, a lot of disagreement or criticism of public claims or accusations of lying of different professional organizations in effective altruism, or AI risk, have repeatedly been generically interpreted as a blanket refusal to honestly engage with the clams being made. Instead of a good-faith effort to resolve different kinds of disputes with public accusations of lying being made, repeat accusations, and justifications for them, are made into long, complicated theories. These theories don't appear to respond at all to the content of the disagreements with the public accusations of lying and dishonesty, and that's why these repeat accusations and justifications for them are poorly received.

These complicated theories don't have anything to do with what people actually want when public accusations of dishonesty or lying are being made, what is typically called 'hard' (e.g., robust, empirical, etc.) evidence. If you were to make narrow claims of dishonesty with more modest language, based on just the best evidence you have, and being willing to defend the claim based on that; instead of making broad claims of dishonesty with ambiguous language, based on complicated theories, they would be received better. That doesn't mean the theories of how dishonesty functions in communities, as an exploration of social epistemology, shouldn't be written. It's just that they do not come across as the most compelling evidence to substantiate public accusations of dishonesty.

For me it's never been so complicated as to require involving decision theory. It's as simple as some of the basic claims being made into much larger, more exaggerated or hyperbolic claims being a problem. They also come along with readers, presumably a general audience among the effective altruism or rationality communities, apparently needing to have prior knowledge of a bunch of things they may not be familiar with. They will only be able to parse the claims being made by reading a series of long, dense blog posts that don't really emphasize the thing these communities should be most concerned about.

Sometimes the claims being made are that Givewell is being dishonest, and sometimes they are something like because of this the entire effective altruism movement has been totally compromised, and is also incorrigibly dishonest. There is disagreement, sometimes disputing how the numbers were used in the counterpoint to Givewell; and some about the hyperbolic claims made that appear as though they're intended to smear more people than whoever at Givewell, or who else in the EA community, is responsible. It appears as though people like you or Ben don't sort through, try parsing, and working through these different disagreements or criticisms. It appears as though you just take all that at face value as confirmation the rest of the EA community doesn't want to hear the truth, and that people worship Givewell at the expense of any honesty, or something.

It's in my experience too, that with these discussions of complicated subjects that appear very truncated for those unfamiliar, that the instructions are just to go read some much larger body of writing or theory to understand why and how people deceiving themselves, each other, and the public in the ways you're claiming. This is often said as if it's completely reasonable to claim it's the responsibility of a bunch of people with other criticisms or disagreements with what you're saying to go read tons of other content, when you are calling people liars, instead of you being able to say what you're trying to say in a different way.

I'm not even saying that you shouldn't publicly accuse people of being liars if you really think they're lying. In cases of a belief that Givewell or other actors in effective altruism have failed to change their public messaging in the face of, by their own convictions, being correctly pointed out as them being wrong, then just say that. It's not necessary to claim that thus the entire effective altruism community are also dishonest. That is especially the case for members of the EA community who disagree with you, not because they dishonestly refused the facts they were confronted with, but because they were disputing the claims being made, and their interlocutor refused to engage, or deflected all kinds of disagreements.

I'm sure there are lots of responses to criticisms of EA which have been needlessly hostile. Yet reacting, and writing strings of posts as though, the whole body of responses were consistent in just being garbage, is just not accurate of the responses you and Ben have received. Again, if you want to write long essays about what rational implications how people react to public accusations of dishonesty has for social epistemology, that's fine. It would just suit most people better if that was done entirely separately from the accusations of dishonesty. If you're publicly accusing some people of being dishonest, just accuse those and only those people of being dishonest very specifically. Stop tarring so many other people with such a broad brush.

I haven't read your recent article accusing some actors in AI alignment of being liars. This dialogue seems like it is both about that, and a response to other examples. I'm mostly going off those other examples. If you want to say someone is being dishonest, just say that. Substantiate it with what the closest thing you have to hard or empirical evidence that some kind of dishonesty is going on. It's not going to work with an idiosyncratic theory of how what someone is saying meets some kind of technical definition of dishonesty that defies common sense. I'm very critical of a lot of things that happen in effective altruism myself. It's just that the way that you and Ben have gone about it is so poorly executed, and backfires so much, I don't think there is any chance of you resolving the problems you're trying to resolve with your typical approaches.

So, I've given up on keeping up with the articles you're writing criticizing things in effective altruism happening, at least on a regular basis. Sometimes others nudge me to look at them. I might get around to them eventually. It's honestly at the point, though, where the pattern I've learned to follow is to not being open-minded that the criticisms being made of effective altruism are worth taking seriously.

The problem I have isn't the problems being pointed out, or that different organizations are being criticized for their alleged mistakes. It's how the presentation of the problem, and the criticism being made, are often so convoluted I can't understand them, and that's before I can figure out if I agree or not. I find that I am generally more open-minded than most people in effective altruism to take seriously criticisms made of the community, or related organizations. Yet I've learned to suspend that for the criticisms you and Ben make, for the reasons I gave, because it's just not worth the time and effort to do so.

comment by jessicata (jessica.liu.taylor) · 2019-07-25T09:38:29.623Z · score: 3 (3 votes) · LW · GW

This is a fictional dialogue demonstrating a meta-level point about how discourse works, and your comment is pretty off-topic. If you want to comment on my AI timelines post, do that (although you haven't read it so I don't even know which of my content you're trying to comment on).

comment by dxu · 2019-07-25T17:24:50.585Z · score: 4 (2 votes) · LW · GW

This is a fictional dialogue demonstrating a meta-level point about how discourse works, and your comment is pretty off-topic.

I think that if a given "meta-level point" has obvious ties to existing object-level discussions, then attempting to suppress the object-level points when they're raised in response is pretty disingenuous. (What I would actually prefer is for the person making the meta-level point to be the same person pointing out the object-level connection, complete with "and here is why I feel this meta-level point is relevant to the object level". If the original poster doesn't do that, then it does indeed make comments on the object-level issues seem "off-topic", a fact which ought to be laid at the feet of the original poster for not making the connection explicit, rather than at the feet of the commenter, who correctly perceived the implications.)

Now, perhaps it's the case that your post actually had nothing to do with the conversations surrounding EA or whatever. (I find this improbable, but that's neither here nor there.) If so, then you as a writer ought to have picked a different example, one with fewer resemblances to the ongoing discussion. (The example Jeff gave in his top-level comment, for example, is not only clearer and more effective at conveying your "meta-level point", but also bears significantly less resemblance to the controversy around EA.) The fact that the example you chose so obviously references existing discussions that multiple commenters pointed it out is evidence that either (a) you intended for that to happen, or (b) you really didn't put a lot of thought into picking a good example.

comment by jessicata (jessica.liu.taylor) · 2019-07-25T17:39:38.202Z · score: 4 (2 votes) · LW · GW

I shouldn't have to argue about the object-level political consequences of 1+4=5 in a post arguing exactly that. This is the analytic synthetic distinction / logical uncertainty / etc.

Yes, I could have picked a better less political example, as recommended in Politics is the Mind Killer. In retrospect, that would have caused less confusion.

Anyway, Evan has the option of commenting on my AI timelines post, open thread, top level post, shortform, etc.

comment by romeostevensit · 2019-07-30T14:30:34.761Z · score: 2 (3 votes) · LW · GW

In metaphysical conflicts people don't win by coming up with the best evidence, they win by controlling what gets counted as evidence. By default, memeplexes gain stability by creating an environment in which evidence against them can't be taken seriously. Arguments that EA has failed to actually measure the things it claims are worth measuring should be taken very seriously on their face, since that is core to the claims of moral obligation (which is itself a bad frame, but less serious.)

comment by Wei_Dai · 2019-07-21T00:13:44.183Z · score: 6 (2 votes) · LW · GW

So, in summary: if we’re going to have suppression of some facts being said out loud, we should have that through either clear norms designed with consequences (including consequences for epistemology) in mind, or individuals deciding not to say things, but otherwise our norms should be protecting true speech, and outlawing appeals to consequences.

  1. Are you happy with a LW with multiple norm sets, where this is one of the norm sets you can choose?

  2. What's your plan if communities or sub-communities with these norms don't draw enough participants to become or stay viable? (One could argue that's what happened to LW1, at least in part. What do you think?)

comment by jessicata (jessica.liu.taylor) · 2019-07-21T09:56:58.684Z · score: 2 (1 votes) · LW · GW
  1. Yes.

  2. Think about why that is and adjust strategy and norms correspondingly. (Sorry that's underspecified, but it actually depends on the reasons). I don't know what happened to LW1, but it did have pretty high intellectual generativity for a while.

comment by Ruby · 2019-07-21T20:32:41.813Z · score: 2 (1 votes) · LW · GW
I don't know what happened to LW1, but it did have pretty high intellectual generativity for a while.

I think Wei Dai said that too elsewhere. When each of you says intellectual generativity, do you the site a whole (post + discussions), or specifically that the discussions in comments were more generative?

Other question is if you think you can quantitatively state some factor by which LW1 was more generative than LW2? If it was only 2x, that would suggest less generativity per person/comment than current LW, since old LW had much more than double the number of users and comments. If it was 10x, then LW1 was qualitatively better in some way.

(I'd expect the output to be a right-tailed distribution over individuals. LW2 could be less generative than LW1 because the top N users which produced 80% of the value left, so it's not really about the raw number of users/comments.

The most interesting scenario would be if it were all the same people, but they were being less generative.)

comment by jessicata (jessica.liu.taylor) · 2019-07-21T21:43:53.577Z · score: 2 (1 votes) · LW · GW

The site as a whole.

I wasn't around in early LW, so this is hard for me to estimate. My very, very rough guess is 5x. (Note, IMO the recent good content is disproportionately written by people willing to talk about adversarial optimization patterns in a somewhat-forceful way despite pressures to be diplomatic)

comment by Benquo · 2019-07-22T01:02:13.897Z · score: 4 (2 votes) · LW · GW

This is maybe half or more of what Robin Hanson wrote about back when it was still all on overcomingbias.com

comment by Dagon · 2019-07-25T18:32:00.668Z · score: 3 (2 votes) · LW · GW

I have noted this as well, and I find it worrisome. Many recent interesting conversations are more about social and interpersonal communication / alignment than about personal or theoretical rationality and decision-making. I like it because they are actually interesting topics. I worry that they're crowding out or hiding a painful decline of more core rationality discussions. I don't worry that they're too close to politics (I think they are close to politics, but are narrow enough that they seem to fall prey to the standard problems more because they're trying to skate around the issue rather than being direct).

I had not framed them as "adversarial optimization patterns", mostly because they seriously bury that lede. A direct acknowledgement would be useful that almost all groups of more than one (and in some models, including an individual human) contain multiple simultaneous games, with very different payout matrices and equilibria which impact other games. Values start out divergent, and this can't be assumed away for any part of reality.

comment by Ruby · 2019-07-21T23:09:01.753Z · score: 2 (1 votes) · LW · GW

Yeah, granted that it's going to be rough.

5x seems consistent with the raw activity numbers [LW · GW] though. Eyeballing it, seems like 4x more active in terms of comments and commenters. Number of posts is pretty close.

comment by Raemon · 2019-07-22T00:57:34.776Z · score: 3 (1 votes) · LW · GW

One of my current beliefs, based on skimming older posts periodically (esp. since recommendations), is that a lot of the old comments just weren't that good. Not sure about posts.

comment by cousin_it · 2019-07-18T12:42:11.728Z · score: 3 (1 votes) · LW · GW

So if evidence against X is being suppressed, then people's belief in X is unreliable, so it can't justify suppressing evidence against X. That's a great argument for free speech, thanks! Do you know if it's been stated before?

comment by Said Achmiz (SaidAchmiz) · 2019-07-18T13:19:13.158Z · score: 11 (5 votes) · LW · GW

This doesn’t seem quite right to me.

Consider this example:

“Evidence against the Holocaust is being suppressed[1]. Therefore people’s belief in the Holocaust is unreliable. And so we cannot justify suppressing Holocaust denial by appealing to the (alleged) fact of the Holocaust having occurred.”

Something is wrong here, it seems to me. Not with the conclusion, mind you, the policy proposal, as it were; that part is all right. But the logic feels odd, don’t you think?

I don’t have a full account, yet, of cases like this, but it seems to me that some of the relevant considerations are as follows. Firstly, we previously undertook a comprehensive project (or multiple such) to determine the truth of the matter, which operated under no such restrictions as we now defend, and came to conclusions which cannot be denied. Secondly, we have people whose belief in the facts of the matter come from personal experience, and are not at all contingent on (nor even alterable by) any evidence we may or may not now present. Thirdly, as the question is one of historical fact, no new evidence may be generated; previously unknown but existing evidence may be uncovered, or currently known evidence may be shown to be misleading or fraudulent, but there is (it seems) no question of experimentation, or similar de novo generation of data.[2]

Now, I do not say that these considerations absolutely refute the given logic. But it seems to me that they seriously undermine its force. Here is a situation where (or so it seems!) we may be quite certain of our conclusions, have no reason to expect anything more than the most infinitesimal (for practical purposes, nil) chance for our understanding of the facts to ever change, and thus may suppress the presentation of any alleged evidence against our current view, while maintaining the rational belief that we are not thereby risking some distortion of our grasp of the facts.

I can think of certain counterarguments (and in any case I do not endorse the conclusion suggested by this line of argument, for somewhat-unrelated reasons), but I am curious to see what you make of this.

Full disclosure: I have quite a few family members who survived the Holocaust (and many more, of course, who did not). I am also strongly opposed to laws against Holocaust denial.


  1. If you live in parts of Europe, for example, where there are criminal penalties for Holocaust denial. ↩︎

  2. Barring, perhaps, the invention of time travel, inter-temporal observation, or similar fantastic technologies. ↩︎

comment by cousin_it · 2019-07-18T14:03:05.712Z · score: 9 (5 votes) · LW · GW

Yes, having strong unfiltered evidence for X can justify suppressing evidence against X. But if suppression is already in effect, and someone doesn't already have unfiltered evidence, I'm not sure where they'd get any. So the share of voters who can justify suppression will decrease over time.

comment by Slider · 2019-07-20T04:03:49.534Z · score: 1 (1 votes) · LW · GW

"If we want to do those things, we have to do them by getting to the truth"

This seems fair if it focuses on the rationalist strategy on trying to interface with the world and how truth is essential. However it's probably not literally true in that there are probably Dark Arts and such which provide those spesific sought goods with outrageous prices. "Have" in this context means "within our options we have created for ourselfs" and not "it is not possible to produce the effect via other means"

Carter states that the norm for discussion norms is whether they obscure or reveal the truth. But then on radical honesty it is not counted in radical honestys favour that truth is more likely to come out and some unspesified "is bad for people" is found to be sufficient reason to abandon it. It's is not clear whethe it means "epistemologically bad" or in the ordinary sense "bad consequences". This ends up being a total cop-out in my mind how the known downsides for radical honesty overcome his stated principle obscuring is the standard. I think this reveals that he is a hypocrite about finding good only via truth as here it is not applied but a common sense knowledge about social harshness overrides it. The door would be open that some other metric could have norm-level good impact that would outwieght the epistemological impact.

It would also seem very abusable if all truths should always be evaluated on their object level. In law there is a principle of the poisonous tree that evidence obtained via illegal means can't be used to establish quilt. If a court would be forced to take into account all true facts cops would be tempted to commit small crimes to get evidence for big crimes. A court can have divided loyalties as it can be reasoned that fairness is not the same as truth and procedure that ensures fairness but is truth diminishing can be acceptable.

The opening fact statement did contain information that is conductive to credibility evaluation (who said) in additon to the puppy number. But I could very well imagine that this "foundation" would be insufficently firm to let the discussion fly. If for example I said "I heard the wind whisper to me that so and so organization saved X puppies last year" a natural curiocity would be "the wind whispered you?" and this would be processed before processing of X would start and it would be possible to have a conclusion of "I am not hearing about your hallucinations one bit more". This foundation building is a natural place to place other checks but even if concerned with truth there must be some reason why the information is relevant. Before you take too seriously what is written on a paper you must to a degree believe that the paper existed. But there is no door to pure hypothethicals. You don't get to submit X for consideration if you don't have a slight degree of justified belief in it. And maybe some base level of indication is given by "because I am saying so" Afterall what are observations but stubbornly correlated hallucinations? But by your word alone it's a mere claim.