Dialogue on Appeals to Consequences

The motivating example for this post is whether you should say "So, I actually checked with some of their former employees, and if what they say and my corresponding calculations are right, they actually only saved 138 puppies", with Quinn arguing that you shouldn't say it because saying it has bad consequences. The problem is, saying this has very clearly good consequences, which means trying to use it as a tool for figuring out what you think of appeals to consequences sets up your intuitions to confuse you.

(It has clearly good consequences because "how much money goes to PADP right now" is far less import than "building a culture of caring about the actual effectiveness of organizations and truly trying to find/make the best ones". Plus if, say, Animal Charity Evaluators trusted this higher number of puppies saved and it had lead them to recommend PADP as I've if their top charities, that that would mean displacing funds that could have gone to more effective animal charities. The whole Effective Altruism project is about trying to figure out how to get the biggest positive impact, and you can't do this if you declare discussing negative information about organizations off limits.)

The post would be a lot clearer if it had a motivating example that really did have bad consequences, all things considered. As a person who's strongly pro transparency is hard for me to come up with cases, but there are still contexts where I think it's probably the case. What if Carter were a researcher who had run a small study on a new infant vaccine and seen elevated autism rates on the experimental group. There's an existing "vaccines cause autism" meme that is both very probably wrong and very probably harmful, which means Carter should be careful about messaging for their results. Good potential outcomes include:

• Carter's experiment is replicated, confirmed, and the vaccine is not rolled out.

• Carter's experiment fails to replicate, researchers look into it more, and discover that there was a problem in the initial experiment / in the replication / they need more data / etc

Bad potential outcomes include:

• Headlines that say "scientists finally admit vaccines do cause autism"

Because of the potential harmful consequences of handling this poorly, Carter should be careful about how they talk about their results and to who. Trying to get funding to scale up the experiment, making sure the FDA is aware, letting other researchers know, etc, all are beneficial and have good consequences. Going to the mainstream media with a controversial sell-lots-of-papers story, by contrast, would have predictably bad consequences.

When talking with friends or within your field it's hard to think of cases where you shouldn't just say the interesting thing you've found, while with larger audiences and in less truth-oriented cultures you need to start being more careful.

EDIT: expanded this into https://www.jefftk.com/p/appeals-to-consequences

Instead of Quinn admitting lying is sometimes good, I wish he had said something like:

“PADP is widely considered a good charity by smart people who we trust. So we have a prior on it being good. You’ve discovered some apparent evidence that it’s bad. So now we have to combine the prior and the evidence, and we end up with some percent confidence that they’re bad.

If this is 90% confidence they’re bad, go ahead. What if it’s more like 55%? What’s the right action to take if you’re 55% sure a charity is incompetent and dishonest (but 45% chance you misinterpreted the evidence)? Should you call them out on it? That’s good in the world where you’re right, but might disproportionately tarnish their reputation in the world where they're wrong. It seems like if you’re 55% sure, you have a tough call. You might want to try something like bringing up your concerns privately with close friends and only going public if they share your opinion, or asking the charity first and only going public if they can’t explain themselves. Or you might want to try bringing up your concerns in a nonconfrontational way, more like ‘Can anyone figure out what’s going on with PADP’s math?’ rather than ‘PADP is dishonest’. After this doesn’t work and lots of other people confirm your intuitions of distrust, then your confidence reaches 90% and you start doing things more like shouting ‘PADP is dishonest’ from the rooftops.

Or maybe you’ll never reach 90% confidence. Many people think that climate science is dishonest. I don’t doubt many of them are reporting their beliefs honestly - that they’ve done a deep investigation and that’s what they’ve concluded. It’s just that they’re not smart, informed, or rational enough to understand what’s going on, or to process it in an unbiased way. What advice would you give these people about calling scientists out on dishonesty - again given that rumors are powerful things and can ruin important work? My advice to them would be to consider that they may be overconfident, and that there needs to be some intermediate ‘consider my own limitations and the consequences of my irreversible actions’ step in between ‘this looks dishonest to me’ and ‘I will publicly declare it dishonest’. And that step is going to look like an appeal to consequences, especially if the climate deniers are so caught up in their own biases that they can't imagine they might be wrong.

I don’t want to deny that calling out apparent dishonesty when you’re pretty sure of it, or when you’ve gone through every effort you can to check it and it still seems bad, will sometimes (maybe usually) be the best course, but I don’t think it’s as simple as you think.”

...and seen what Carter answered.

The bar that is set for appeals to consequences imply the sort of equilibrium world you'll end up in. Erring on the side of higher is better, because it is hard to go the other way because epistemic standards tend to slide in the face of local incentives.

I also want to note an argumentative tactic that occurs on the tacit level whereby people will push you into a state where you need to expend more energy on average per truth bit than they do, so they eventually win by attrition. Related to evaporative cooling. The subjective experience of this feels like talking to the cops. You sense that no big wins are available (because they have their bottom line) but big losses are, so you stop talking. If you've encountered this dynamic, you recognize things like this

> "You still haven't refuted my argument. If you don't do so, I win by default."

as part of the supporting framework for the dynamic and it will make you very angry...which others will then use as part of the dynamic which makes you angry which......

When someone is arguing that a belief of yours is wrong, listen to the argument at the object level, instead of jumping to the question of whether saying the relevant arguments out loud is a good idea, which is a much harder question.”

It seems to me that they key issue here is the need for both public and private conversational spaces.

In public spaces, arguments are soldiers. They have to be, because others treat them that way, and because there are actual policies that we're all fighting / negotiating over. In those contexts it is reasonable (I don't know if it is correct, or not), to constrain what things you say, even if they're true, because of their consequences. It is often the case that one piece of information, though true, taken out of context, does more harm than good, and often conveying the whole informational context to a large group of people is all but impossible.

But we need to be able to figure out which policies to support, somehow, separately from supporting them on this political battlefield. We also need private spaces, where we can think and our initial thoughts can be isolated from their possible consequences, or we won't be able to think freely.

It seems like Carter thinks they are having a private conversation, in a private space, and Quinn thinks they're having a public conversation in a public space.

Well, I certainly agree with the position you’re defending. Yet I can’t help but feel that the arguments in the OP lack… a certain concentrated force, which I feel this topic greatly deserves.

Without disagreeing, necessarily, with anything you say, here is my own attempt, in two (more or less independent) parts.

The citadel of truth

If the truth is precious, its pursuit must be unburdened by such considerations as “what will happen if we say this”. This is impractical, in the general case. You may not be interested in consequences, after all, but the consequences are quite interested in you…

There is, however, one way out of the quagmire of consequential anxiety. Let there be a place around which a firewall of epistemology is erected. Let appeals to consequences outside that citadel, be banned within its walls. Let no one say: “if we say such a thing, why, think what might happen, out there, in the wider world!”. Yes, if you say this thing out there, perhaps unfortunate consequences may follow out there. But we are not speaking out there; so long as we speak in here, to each other, let us consider it irrelevant what effects our words may produce upon the world outside. In here, we concern ourselves only with truth. All that we do and say, within the citadel, serves only truth.

Any among us who have something to protect, in the world beyond the citadel, may wish to take the truths we find, and apply them to that outside world, and discuss these things with others who feel as they do. In these discussions, of plans and strategies for acting upon the wider world, the consequences of their words, for that world, may be of the utmost importance. But if so, to have such discussions, these planners will have to step outside the citadel’s walls. To talk in such a way in here—to speak, and to hold yourself and others to considering the consequences of their words upon the world—is to violate the citadel’s rule: that all that we do and say within, serves truth, and only truth.

Without evaluation, consequences have no meaning

The one comes to you and says: if you say such a thing, why, this-and-such will happen!

Well, and what of it? For this to be compelling, it is not enough that the consequences of speech be a certain way, but that you evaluate them a certain way. To find that “this-and-such will happen if you say that” is a reason for not saying it, you must evaluate the given consequence as negative, and furthermore you must weigh it against the other consequences of your speech act—all the other consequences—and judge that this one downside tilts the balance, in favor of silence.

And now suppose one comes to me, and says: Said, this thing you said—consider the consequences of saying it! They are bad. Well, perhaps they are. But consider, I respond, the consequences of allowing you, concerned citizen, to convince me, by this argument of yours, to keep silent. They include corruption of the truth, and the undermining of the search for it, and distortion of the accurate beliefs of everyone I speak to, and my own as well. These, too, are consequences. And I evaluate these effects to have so negative a value, that, short of either the inescapable annihilation of the human race (or outcomes even more dire), or serious personal harm to me or those close to me, no unpleasant consequence you might threaten me with could possibly compare. So I thank you—I continue—for your most sincere concern, but your admonition cannot move me.

In short: it is the case, I find, in most such arguments-from-consequences, that anyone who may be moved by them, does not value truth—does not, at any rate, value it enough that they may rightly be admitted to any circle of truth-seekers which pretends really to be serious in the pursuit of its goal.

[speaking for myself, not for any organization]

If this is an allegory against appeals to consequences generally, well and good.

If there's some actual question about whether wrong cost effectiveness numbers are being promoted, could people please talk about those numbers specifically so we can all have a try at working out if that's really going on? E.g. this post made a similar claim to what's implied in this allegory, but it was helpful that it used concrete examples so people could work out whether they agreed (and, in that case, identify factual errors).

I think this is strawmanning the appeal to consequences argument, by mixing up private beliefs and public statements, and by ending with a pretty superficial agreement on rule-consequentialism without exploring how to pick which rules (among one for improving private beliefs, one for sharing relevant true information and one for suppressing harmful information) applies.

The participants never actually attempt to resolve the truth about puppies saved per dollar, calling the whole thing into question - both whether their agreement is real and whether it's the right thing. Many of these discussions should include a recitation of [ https://wiki.lesswrong.com/wiki/Litany_of_Tarski ], and a direct exploration whether it's beliefs (private) or publication (impacting presumed-less-rational agents) that is at issue.

In any case, appeals to consequences at the meta/rule level still HAS to be grounded in appeals to consequences at the actual object consequence level. A rule that has so many exceptions that it's mostly wrong is actively harmful. My objection to the objection to "appeal to consequences" is that the REAL objection is to bad epistemology of consequence prediction, not to the desire to predict consequences.

In a completely separate direction, consequences of speech acts in public/group settings are WAY more complicated than epistemic consequences of a truth-seeking discussion among a small group of fairly close rationalist-inclined friends. Both different rules/defaults/norms apply, and different calculations of consequences of specific speech actions are made.

All that said, I prefer norms that lean toward truth-telling and truth-seeking, and it makes me suspicious when that is at odds with consequences of speech acts. I have a higher standard of evidence for my consequence predictions for lying than I have for withholding relevant facts than I have for truth-telling.

Carter is a mistake theorist, Quinn is a conflict theorist. At no point does Quinn ever talk about truth, or about anything, really. His words are weapons to achieve an end by whatever means possible. There is no more meaning in them than in a fist. Carter's meta-mistake is to believe that he is arguing with someone. Quinn is not arguing; he is in a fist fight.

Quinn: “Hold it right there. Regardless of whether that’s true, it’s bad to say that.”

Carter: “That’s an appeal to consequences, well-known to be a logical fallacy.”

The link in Carter's statement leads to a page that clearly contradicts Carter's claim:

In logic, appeal to consequences refers only to arguments that assert a conclusion's truth value (true or false) without regard to the formal preservation of the truth from the premises; appeal to consequences does not refer to arguments that address a premise's consequential desirability (good or bad, or right or wrong) instead of its truth value.

Summary: I'm aware of a lot of examples of real debates that inspired this dialogue. It seems in those real cases, a lot of disagreement or criticism of public claims or accusations of lying of different professional organizations in effective altruism, or AI risk, have repeatedly been generically interpreted as a blanket refusal to honestly engage with the clams being made. Instead of a good-faith effort to resolve different kinds of disputes with public accusations of lying being made, repeat accusations, and justifications for them, are made into long, complicated theories. These theories don't appear to respond at all to the content of the disagreements with the public accusations of lying and dishonesty, and that's why these repeat accusations and justifications for them are poorly received.

These complicated theories don't have anything to do with what people actually want when public accusations of dishonesty or lying are being made, what is typically called 'hard' (e.g., robust, empirical, etc.) evidence. If you were to make narrow claims of dishonesty with more modest language, based on just the best evidence you have, and being willing to defend the claim based on that; instead of making broad claims of dishonesty with ambiguous language, based on complicated theories, they would be received better. That doesn't mean the theories of how dishonesty functions in communities, as an exploration of social epistemology, shouldn't be written. It's just that they do not come across as the most compelling evidence to substantiate public accusations of dishonesty.

For me it's never been so complicated as to require involving decision theory. It's as simple as some of the basic claims being made into much larger, more exaggerated or hyperbolic claims being a problem. They also come along with readers, presumably a general audience among the effective altruism or rationality communities, apparently needing to have prior knowledge of a bunch of things they may not be familiar with. They will only be able to parse the claims being made by reading a series of long, dense blog posts that don't really emphasize the thing these communities should be most concerned about.

Sometimes the claims being made are that Givewell is being dishonest, and sometimes they are something like because of this the entire effective altruism movement has been totally compromised, and is also incorrigibly dishonest. There is disagreement, sometimes disputing how the numbers were used in the counterpoint to Givewell; and some about the hyperbolic claims made that appear as though they're intended to smear more people than whoever at Givewell, or who else in the EA community, is responsible. It appears as though people like you or Ben don't sort through, try parsing, and working through these different disagreements or criticisms. It appears as though you just take all that at face value as confirmation the rest of the EA community doesn't want to hear the truth, and that people worship Givewell at the expense of any honesty, or something.

It's in my experience too, that with these discussions of complicated subjects that appear very truncated for those unfamiliar, that the instructions are just to go read some much larger body of writing or theory to understand why and how people deceiving themselves, each other, and the public in the ways you're claiming. This is often said as if it's completely reasonable to claim it's the responsibility of a bunch of people with other criticisms or disagreements with what you're saying to go read tons of other content, when you are calling people liars, instead of you being able to say what you're trying to say in a different way.

I'm not even saying that you shouldn't publicly accuse people of being liars if you really think they're lying. In cases of a belief that Givewell or other actors in effective altruism have failed to change their public messaging in the face of, by their own convictions, being correctly pointed out as them being wrong, then just say that. It's not necessary to claim that thus the entire effective altruism community are also dishonest. That is especially the case for members of the EA community who disagree with you, not because they dishonestly refused the facts they were confronted with, but because they were disputing the claims being made, and their interlocutor refused to engage, or deflected all kinds of disagreements.

I'm sure there are lots of responses to criticisms of EA which have been needlessly hostile. Yet reacting, and writing strings of posts as though, the whole body of responses were consistent in just being garbage, is just not accurate of the responses you and Ben have received. Again, if you want to write long essays about what rational implications how people react to public accusations of dishonesty has for social epistemology, that's fine. It would just suit most people better if that was done entirely separately from the accusations of dishonesty. If you're publicly accusing some people of being dishonest, just accuse those and only those people of being dishonest very specifically. Stop tarring so many other people with such a broad brush.

I haven't read your recent article accusing some actors in AI alignment of being liars. This dialogue seems like it is both about that, and a response to other examples. I'm mostly going off those other examples. If you want to say someone is being dishonest, just say that. Substantiate it with what the closest thing you have to hard or empirical evidence that some kind of dishonesty is going on. It's not going to work with an idiosyncratic theory of how what someone is saying meets some kind of technical definition of dishonesty that defies common sense. I'm very critical of a lot of things that happen in effective altruism myself. It's just that the way that you and Ben have gone about it is so poorly executed, and backfires so much, I don't think there is any chance of you resolving the problems you're trying to resolve with your typical approaches.

So, I've given up on keeping up with the articles you're writing criticizing things in effective altruism happening, at least on a regular basis. Sometimes others nudge me to look at them. I might get around to them eventually. It's honestly at the point, though, where the pattern I've learned to follow is to not being open-minded that the criticisms being made of effective altruism are worth taking seriously.

The problem I have isn't the problems being pointed out, or that different organizations are being criticized for their alleged mistakes. It's how the presentation of the problem, and the criticism being made, are often so convoluted I can't understand them, and that's before I can figure out if I agree or not. I find that I am generally more open-minded than most people in effective altruism to take seriously criticisms made of the community, or related organizations. Yet I've learned to suspend that for the criticisms you and Ben make, for the reasons I gave, because it's just not worth the time and effort to do so.

So, in summary: if we’re going to have suppression of some facts being said out loud, we should have that through either clear norms designed with consequences (including consequences for epistemology) in mind, or individuals deciding not to say things, but otherwise our norms should be protecting true speech, and outlawing appeals to consequences.

1. Are you happy with a LW with multiple norm sets, where this is one of the norm sets you can choose?

2. What's your plan if communities or sub-communities with these norms don't draw enough participants to become or stay viable? (One could argue that's what happened to LW1, at least in part. What do you think?)

So if evidence against X is being suppressed, then people's belief in X is unreliable, so it can't justify suppressing evidence against X. That's a great argument for free speech, thanks! Do you know if it's been stated before?

"If we want to do those things, we have to do them by getting to the truth"

This seems fair if it focuses on the rationalist strategy on trying to interface with the world and how truth is essential. However it's probably not literally true in that there are probably Dark Arts and such which provide those spesific sought goods with outrageous prices. "Have" in this context means "within our options we have created for ourselfs" and not "it is not possible to produce the effect via other means"

Carter states that the norm for discussion norms is whether they obscure or reveal the truth. But then on radical honesty it is not counted in radical honestys favour that truth is more likely to come out and some unspesified "is bad for people" is found to be sufficient reason to abandon it. It's is not clear whethe it means "epistemologically bad" or in the ordinary sense "bad consequences". This ends up being a total cop-out in my mind how the known downsides for radical honesty overcome his stated principle obscuring is the standard. I think this reveals that he is a hypocrite about finding good only via truth as here it is not applied but a common sense knowledge about social harshness overrides it. The door would be open that some other metric could have norm-level good impact that would outwieght the epistemological impact.

It would also seem very abusable if all truths should always be evaluated on their object level. In law there is a principle of the poisonous tree that evidence obtained via illegal means can't be used to establish quilt. If a court would be forced to take into account all true facts cops would be tempted to commit small crimes to get evidence for big crimes. A court can have divided loyalties as it can be reasoned that fairness is not the same as truth and procedure that ensures fairness but is truth diminishing can be acceptable.

The opening fact statement did contain information that is conductive to credibility evaluation (who said) in additon to the puppy number. But I could very well imagine that this "foundation" would be insufficently firm to let the discussion fly. If for example I said "I heard the wind whisper to me that so and so organization saved X puppies last year" a natural curiocity would be "the wind whispered you?" and this would be processed before processing of X would start and it would be possible to have a conclusion of "I am not hearing about your hallucinations one bit more". This foundation building is a natural place to place other checks but even if concerned with truth there must be some reason why the information is relevant. Before you take too seriously what is written on a paper you must to a degree believe that the paper existed. But there is no door to pure hypothethicals. You don't get to submit X for consideration if you don't have a slight degree of justified belief in it. And maybe some base level of indication is given by "because I am saying so" Afterall what are observations but stubbornly correlated hallucinations? But by your word alone it's a mere claim.