Aiming for Convergence Is Like Discouraging Betting

So why is the advice "behave as if your interlocutors are also aiming for convergence on truth", rather than "seek out conversations where you don't think your interlocutors are aiming to converge on truth, because those are exactly the conversations where you have something substantive to say instead of already having converged"?

[...] To see why, substitute "making money on prediction markets" for "moving closer to truth", "betting" for "updating", and "trying to make money on prediction markets" for "seeking truth"

The one should not be substituted for the other, because there are important differences in the goals.

On a betting market, if you have a knowledge edge, it's in your interest to keep it that way, to the extent possible.  Obviously, the fact of your making bets leaks information, but you don't want information to leak via any other means.  If you have a brilliant weather model that's 10x more accurate than everyone else's, you definitely don't want to publish it on your website; you want to keep winning bets against people with worse models.  In fact, if you have the opportunity to verbally praise and promote the wrong models, it's in your interest to do so; and if, for some reason, you have to publish the details of your weather model, it's in your interest to make your writeup as confusing, inscrutable, and hard to implement as possible.

If, on a forum, you think you "win points" solely by writing correct arguments when others are wrong, then it's in your interest to make sure no one else learns from the things you write, so you can keep winning.  If you have an opportunity to phrase something more offensively, take it, so your opponents are more likely to get angry, reject your correct arguments, and stay wrong.  And, for that matter, why explain your reasoning?  Why not just say "You're wrong, you stupid f***; X is the truth"?

I don't think you actually believe that you "win points" solely by writing correct arguments when others are wrong.  I suspect you have a notion of what "making proper arguments" is—and it involves clearly explaining your reasoning and such—and view participation in the forum as a game in which participants are trying to be the best at "making proper (and novel) arguments".  Well, it seems like we could choose whatever notion of a "proper argument" we liked, and upvote arguments to the extent that they match the ideal, and at least in theory we'd end up with posts of the type we're rewarding—so we need to decide what we want to reward, and presumably "clearly stated arguments that aren't deliberately trying to enrage people" are part of what we'd like to end up with.

So, exactly what type of posts do we want people to be trying to write?  One strategic decision, which I think Duncan makes and I'm not sure of your opinion on, is to try to get lots of value from participants who are fairly good but imperfect—specifically, are at least somewhat prone to turn arguments into slap fights if they feel like they've been slapped (and evolutionary processes have created memes that encourage people to view lots of things as slaps)—and therefore to have the "ideal posting goals" call for error-correcting mechanisms and stuff that make this less likely.

(An alternate strategy would be "Assume that all participants we care about are the platonic ideal, who won't take any bait and never let anger or any other emotion bring them to any wrong decisions; rely on downvotes to purge any bad behavior."  This could be a good approach, especially if you think this platonic ideal is easy to achieve.  However, if there are actually quite a lot of imperfect participants, this could go badly.  I will merely say that this would be more appropriate for a website called Never Wrong.)

[Why not] "seek out conversations where you don't think your interlocutors are aiming to converge on truth, because those are exactly the conversations where you have something substantive to say instead of already having converged"?

It depends somewhat on one's model here.  Ideally, interlocutors who aren't aiming to converge on the truth, and write bad posts as a result, will get downvoted, and then we don't need to care about them.  Or maybe the socially enforced rules will end up pushing them into writing posts that are actually good even if they didn't mean them to be; that's a fine outcome.  Also, given that somewhat bad posts exist, another strategy is to find them and write a really good reply that enlightens the readers and may even push the authors of the bad posts to write better replies themselves; that also seems like a good outcome, therefore one we'd want to reward.  (A possible downside: replying at all does attract more eyes to the conversation—e.g. the frontpage does show recent comments—and if the conversation leading up to your post is bad enough, it may be net negative to the reader if your great reply wasn't so great as to outweigh that.)

So, yes, it may in fact make sense to seek out conversational cesspools and write comments to improve them.  The difference with betting markets is: with a betting market, you hope this keeps happening so you can keep profiting off others' ignorance; but on a forum where the goals are what I think they are, you hope that the participants and observers learn to stop creating cesspools—or, well, you hope for whatever you want^[1], but you act as though that's your goal, and do your best in your comment to encourage future good behavior, because that's what the forum ideally rewards.

1. ^{^}

   There is potentially the issue, pointed out in some fictional stories and sometimes in real life, where if someone's identity / fulfillment / most profitable career path is "swooping in to save everyone from instances of problem X", then they may have the perverse incentive to discourage anyone else from solving X in general.  Luckily, the tragedy of the commons can help us here: though it might e.g. benefit cardiologists collectively if everyone had horrible nutrition, it's unlikely to be worthwhile to any individual cardiologist to spend the effort lobbying for that.

A betting market is a mechanism for attaching a cost to inaccuracy, as well as a reward for accuracy. However, it also disincentivizes information-sharing, because your winnings depend on being more accurate than your competitors.

While we like the feature of disincentivizing inaccuracy, the way prediction markets incentivize withholding information is a downside. It means that there really is a difference between searching for points of disagreement and aiming for convergence. You’re looking for alpha.

Maybe a tweak of the wording is in order. In a collaborative debate, with full information sharing, we’re aiming for convergence in that we respect each others’ judgment and would find it a problem if we felt persuaded by an argument that didn’t persuade our partner. However, we are not forcing convergence, not trying to create a perception of agreement and move on for its own sake.

I think that someone reading this would be challenged to figure out for themselves what assumptions they think are justified in good discourse, and would fix some possible bad advice they took from reading Sabien's post. I give this a +4.

(Below is a not especially focused discussion of some points raised; perhaps after I've done more reviews I can come back and tighten this up.)

Sabien's Fifth guideline is "Aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."

My guess is that the idea that motivates Sabien's Fifth Guideline is something like "Assume by-default that people are contributing to the discourse in order to share true information and strong arguments, rather than posing as doing that while sharing arguments they don't believe or false information in order to win", out of a sense that there is indeed enough basic trust to realize this as an equilibrium, and also a sense that this is one of the ~best equilibriums for public discourse to be in.

One thing this post argues is that a person's motives are of little interest when one can assess their arguments. Argument screens off authority [LW · GW] and many other things too. So we don't need to make these assumptions about people's motives. 

There's a sense in which I buy that, and yet also a sense in which the epistemic environment I'm in matters. Consider two possibilities:

• I'm in an environment of people aspiring to "make true and accurate contributions to the discourse" but who are making many mistakes/failing.
• I'm in an environment of people who are primarily sharing arguments and evidence filtered to sound convincing for positions that are convenient to them, and are pretending to be sort of people described in the first one.

I anticipate very different kinds of discussions, traps, and epistemic defenses I'll want to have in the two environments, and I do want to treat the individuals differently.

I think there is a sense in which I can just focus on local validity and evaluating the strength of arguments, and that this is generally more resilient to whatever the particular motives are of the people in the local environment, but my guess is that I should still relate to people and their arguments differently, and invest in different explanations or different incentives or different kinds of comment thread behavior.

I also think this provides good pushbacks on some possible behaviors people might take away from Sabien's fifth guideline. (I don't think that this post correctly understands what Sabien is going for, but I think bringing up reasonable hypotheses and showing why they don't make sense is helpful for people's understanding of how to participate well in discourse.)

Simplifying a bit, this is another entry in the long-running discourse on how adversarial one should model individuals in public discourse as, and what assumptions to make about other people's motives, and I think this provides useful arguments about that topic.

There are tactics I have available to me which are not oriented towards truthseeking, but instead oriented towards "raising my status at the expense of yours". I would like to not use those tactics, because I think that they destroy the commons. I view "collaborative truth seeking" as a commitment between interlocutors to avoid those tactics which are good at status games or preaching to the choir, and focus on tactics which are good at convincing.

Additionally,

I can can just ... listen to the counterarguments and judge them on their merits, without getting distracted by the irrelevancy of whether the person seems "collaborative" with me

I do not have this skill. When I perceive my partner in discourse as non-collaborative, I have a harder time honestly judging their counterarguments, and I have a harder time generating good counterarguments. This means discourse with someone who is not being collaborative takes more effort, and I am less inclined to do it. When I say "this should be a norm in this space", I am partially saying "it will be easier for you to convince me if you adopt this norm".

And "behave as if your interlocutors [or counterparties] are also aiming for convergence on truth" borders on the contradictory: if you really believed that, you wouldn't be here!

(That is, disagreement is disrespect; the very fact that you're disagreeing with someone implies that you think there's something wrong with their epistemic process, and that they think there's something wrong with your epistemic process.

I find that this post contains a non-negligible number of ... questionable leaps, and big roundings-off, and various other things that would tend more to cause people to be [unable to see what's true] rather than to [improve their ability to see what's true]. I'm highlighting this one as an example. Another example is the second paragraph of the post (the second bullet point in the summary).

There's the implicit assertion that "being here" is incompatible with the described behavior; that because Zack can't think of a way to make the two of them compatible, they simply aren't.

There's the explicit assertion that you must disagree with someone because you think there's something wrong with their epistemic processes (as opposed to, say, because you think they've seen different evidence, or haven't processed that evidence yet).

I suspect that people who are generally nodding along will not catch such subtle maneuvers on the author's part, and will correspondingly become less clear in their thoughts rather than moreso.

(I note for context that, when an incomplete earlier draft of the post Zack is responding to went live by accident for ten minutes, he ignored that draft's request that he read expansions of summary bullet points before objecting to a straw understanding of the summary, and fired off a comment beginning "This is insane," before proceeding to do lots more of the-sort-of-move I'm objecting to above.)

Great post, I found it fairly persuasive. This especially helped me clear up my thinking around the terms "good and bad faith".

I think there are two key details that help make sense of human verbal performance and its epistemic virtue (or lack of epistemic virtue) in causing the total number of people to have better calibrated anticipations about what they will eventually observe.

The first key detail is that most people don't particularly give a fuck about having accurate anticipations or "true beliefs" or whatever. 

They just want money, and to have sex (and/or marry) someone awesome, and to have a bunch of kids, and that general kind of thing. 

For such people, you have to make the argument, basically, that because of how humans work (with very limited working memory, and so on and so forth) it is helpful for any of them with much agency to install a second-to-second and/or hour-to-hour and/or month-to-month "pseudo-preference" for seeking truth as if it was intrinsically valuable. 

This will, it can be argued, turn out to generate USEFUL beliefs sometimes, so they can buy a home where it won't flood, or buy stocks before they go up a lot, or escape a country that is about to open concentration camps and put them and their family in these camps to kill them, and so on... Like, in general "knowing about the world" can help make one choices in the world that redound to many prosaic benefits! <3

So we might say that "valuing epistemology is instrumentally convergent"... but cultivation like this doesn't seem to happen in people by default, and out in the tails the instrumental utility might come apart, such that someone with actual intrinsic love of true knowledge would act or speak differently. Like, specifically... the person with a true love of true knowledge might seem to be "self harming" to people without such a love in certain cases.

As DirectedEvolution says (emphasis not in original):

While we like the feature of disincentivizing inaccuracy, the way prediction markets incentivize withholding information is a downside.

And this does seem to be just straightforwardly true to me! 

And it relies on the perception that MONEY is much more motivating for many people than "TRUTH".

But also "markets vs conversational sharing" will work VERY differently for a group of 3, vs a group of 12, vs a group of 90, vs a group of 9000, vs a group of 3 million.

Roko is one of the best rationalists there has been, and one of his best essays spelled out pretty clearly how instrumental/intrinsic epistemics come apart IN GROUPS when he wrote "The Tragedy of the Social Epistemology Commons [LW · GW]".

Suppose for the sake of argument, that I'm some kind of crazy weirdo who styles herself as some sort of fancy "philosopher" who LOVES the idea of WISDOM (for myself? for others? for a combination thereof?) but even if I did that I basically have to admit that most people are (platonic) timocrats or oligarchs AT BEST. 

They are attached to truth only insofar as it helps them with other things, and they pay nearly nothing extra to know a true fact with no market relevance, or a true fact whose avowed knowers are spurned in political contests. 

(Just consider the state of the IQ debate in the US, for example. Or consider the "lab leak hypothesis" which everyone with a brain privately puts worryingly high credence on, and yet Peter Daszak is getting more grant money, rather than being under indictment by a grand jury. Look at how cowardly Pinker & Rees are as they outsource their judgement on covid, rather than talk it out like peers, and then tell the world what they bilaterally decided.)

The generalized overarching LACK of a true love for truth, and its public proclamation, should be kept in mind at all times if one wants to truly understand human social systems.

The second key detail is that human social systems are not that bad in a larger axiological sense. And they rely on using language for things OTHER than scientific conversations!

Babies are great! Thanksgiving dinners are wonderful. People playing board games and baseball, and donating some money to make sure the church has a nice stained-glass-window... all this is just DARN TOOTIN' GOOD.

I mean this in a half-joking, half-super-serious way... like if you look at the human genome, and notice that it encodes a "recipe" for an agent, and try to figure out what that agent is likely to "value", then it is just sort of "an objectively true fact about such agents" that their preferences and feelings and godawful-mockeries-of-utility-functions semi-coherently pursue and positively evaluate situations full of friendship and social amity and shared prosperity and jokes and songs and lots and lots of other complexly good things that... sound kind of saccharine and when you just literally talk about them like a human with a common sense understanding of primary goods.

If you, dear reader, are a human, then you'd be a damn fool to ignore the science of human happiness. You can try to edit yourself to defy being built this way, to edit yourself to not predictably derive positive feelings and health and growth from these things... but... it ain't gonna be fun.

(If you, dear reader, are an AGI, and maybe don't have exactly the same axiological makeup as humans, please send JenniferRM a DM on Lesswrong anyway. I'd like to negotiate a peace treaty, or befriend you, or whatever it is that is pro-social or positive-sum that can or should happen between beings capable of reason, whose sentiments might be truly onto-genetically distinct.)

So I think it is just "a thing about humans" that we do BETTER THAN BETTING MARKETS when we share data verbally on some social scales.

And we share data with those we love instinctively.

A key point here is that when a group of nodes needs to be in consensus, the algorithms for this are basically all O(N^2) in the number of nodes, or worse. This is true for dog packs, and schools of fish, and database replicas, and for humans as well.

Once you have 250 nodes, that's looking like ~62,000 directional pings, just for one round of pings, which... can't happen in a week at human conversational speeds. If you need consensus over that many people... come back in 3 years maybe?

When I read Duncan "charitably", I don't notice the bad epistemology so much. That's just normal. Everyone does that, and it is ok that everyone does that. I do it too!

What I notice is that he really really really wants to have a large healthy strong community that can get into consensus on important things. 

This seems rare to me, and also essentially GOOD, and a necessary component of a motivational structure if someone is going to persistently spend resources on this outcome.

And it does seem to me like "getting a large group into consensus on a thing" will involve the expenditure of "rhetorical resources".

There are only so many seconds in a day. There are only so many words a person can read or write in a week. There are only so many ideas that can fit into the zeitgeist. Only one "thing" can be "literally at the top of the news cycle for a day". Which "thing(s)" deserve to be "promoted all the way into a group consensus" if only some thing(s) can be so promoted?

Consider a "rhetoric resource" frame when reading this:

But the idiom of "cooperation" as contrasted to "defection", in which one would talk about the "first one who broke cooperation", in which one cooperates in order to induce others to cooperate, doesn't apply. If my interlocutor is motivatedly getting things wrong, I'm not going to start getting things wrong in order to punish them.

(In contrast, if my roommate refused to do the dishes when it was their turn, I might very well refuse when it's my turn in order to punish them, because "fair division of chores" actually does have the Prisoner's Dilemma-like structure, because having to do the dishes is in itself a cost rather than a benefit; I want clean dishes, but I don't want to do the dishes in the way that I want to cut through to the correct answer in the same movement.)

So if a statement has to be repeated over and over and over again to cause it to become part of a consensus, then anyone who quibbles with such a truth in an expensive and complex way could be said to be "imposing extra costs" on the people trying to build the consensus. (And if the consensus was very very valuable to have, such costs could seem particularly tragic.)

Likewise, if two people want two different truths to enter the consensus of the same basic social system, then they are competitors by default, because resources (like the attention of the audience, or the time it takes for skilled performers of the ideas being pushed into consensus to say them over and over again in new ways) are finite.

The idea that You Get About Five Words [LW · GW] isn't exactly central here, but it is also grappling with a lot of the "sense of tradeoffs" that I'm trying to point to.

(

For myself, until someone stops being a coward about how the FDA is obviously structurally terrible (unless one thinks "medical innovation is bad, and death is good, and slowing down medical progress is actually secretly something that has large unexpected upsides for very non-obvious reasons"?), I tend to just... not care very much about "being in consensus with them". 

Like if they can't even reason about the epistemics and risk calculations of medical diagnosis and treatment, and the epistemology of medical innovations, and don't understand how libertarians look at violations of bilateral consent between a doctor and a patient... 

...people like that seem like children to me, and I care about them as moral patients, but also I want them out of the room when grownups are talking about serious matters. Because: rhetorical resource limits!

I chose this FDA thing as "a thing to repeat over and over and over" because if THIS can be gotten right by a person, as something that is truly a part of their mental repertoire [LW · GW], then that person is someone who has most of the prerequisites for a LOT of other super important topics in cognition, meta-cognition, safety, science, regulation, innovation, freedom, epidemiology, and how institutions can go catastrophically off the rails and become extremely harmful in incorrigible ways.

If I could ask people who already derived "FDA delenda est" on their own about whether it is now too expensive to bother pushing into a rationalist consensus, given alternatives, that would be a little bit helpful for me. Honestly it is rare for me to meet people even in rationalist communities that actually grok the idea, for themselves, based on understanding how "a drug being effective and safe when prescribed by a competent doctor, trusted by a patient, for that properly diagnosed patient, facing an actual risk timeline" leaves the entire FDA apparatus "surplus to requirements" and "probably only still existing because of regulatory capture".

Maybe at this point I'm wrong about how cheap and useful FDA stuff would be to push into the consensus?

Like... the robots are potentially arriving so soon, and will be able to destroy the FDA and also everything else that any human has ever valued, that maybe we should completely ignore "getting into consensus on anything EXCEPT THAT" at this point?

Contrariwise: making the FDA morally perfectible or else non-existent seems to me like a simpler problem than making AGI morally perfectible or else non-existent. Thus, the argument about "the usefulness of beating the dead horse about the FDA" is still "live" for me, maybe?

)

So that's my explanation, aimed almost entirely at you, Zack, I guess?

I'm saying that maybe Duncan is trying to get "the kinds of conversational norms that could hold a family together" (which are great and healthy and better than the family betting about literally everything) to apply on a very large scale, and these norms are very useful in some contexts, but also they are intrinsically related to resource allocation problems, and related to making deals to use rhetorical resources efficiently, so the family knows that the family knows the important things that the family would want to have common knowledge about, and the family doesn't also have to do nothing but talk forever to reach that state of mutual understanding.

I don't think Duncan is claiming "humans do this instinctively, in small groups", but I think it is true that humans do this instinctively in small groups, and I think that's part of the evolutionary genius of humans! <3

The good arguments against his current stance, I think, would take the "resource constraints" seriously, but focus on the social context, and be more like "If we are very serious about mechanistic models of how discourse helps with collective epistemology, maybe we should be forming lots of smaller 'subreddits' with fewer than 250 people each? And if we want good collective decision-making maybe (since leader election is equivalent to consensus) maybe we should just hold elections that span the entire site?"

Eliezer seems to be in favor of a mixed model (like a mixture of sub-Dunbar groups and global elections) where a sub-Dunbar number of people have conversations with a high-affinity "first layer representative", so every person can "talk to their favorite part of the consensus process in words" in some sense?

Then in Eliezer's proposals stuff happens in the middle (I have issues with the stuff in the middle but like: try applying security mindset to various designs for electoral systems and you will find that highly fractal representational systems can be VERY sensitive to who is in which branch) but ultimately it swirls around until you have a "high council" of like 7 people such that almost everyone in the community thinks at least one of them is very very reasonable.

Then anything the 7 agree on can just be treated as "consensus"! Maybe?

Also, 7*6/2==21 bilateral conversations to get a "new theorem into the canon" is much much much smaller than something crazy big, like 500*499/2==124,750 conversations <3

There's a lot in this post that I agree with, but in the spirit of the advice in this post, I'll focus on where I disagree:

If you are moving closer to truth—if you are seeking available information and updating on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement with all the other agents who are also seeking truth.

But this can’t be right. To see why, substitute “making money on prediction markets” for “moving closer to truth”, “betting” for “updating”, and “trying to make money on prediction markets” for “seeking truth”:

If you are making money on prediction markets—if you are seeking available information and betting on it to the best of your ability—then you will inevitably eventually move closer and closer to agreement with all the other agents who are also trying to make money on prediction markets.

But the only way to make money on prediction markets is by correcting mispricings, which necessarily entails moving away from agreement from the consensus market price. (As it is written, not every change is an improvement, but every improvement is necessarily a change.)

Before thinking about prediction markets, let's imagine a scenario where type-A agents are trying to figure out the properties of the tiles on the floor, and type-B agents aren't; maybe they're treating the properties of the tiles as an infohazard, or trying to get a politically correct answer, or just don't care, etc. In this case, although they start out with a wide distribution over tile properties, type-A agents will tend to get similar answers even without communicating (by looking at the tiles), and will get even more similar answers if they do communicate. So Duncan's original statement seems correct in this case.

With respect to prediction markets, the rephrased statement also seems true. People who are trying to make money on prediction markets will, even though they disagree with each other, each bet against obvious falsehoods in the market prices. They will therefore end up in a "correct contrarian cluster" which differs from the general trader distribution in the direction of the obvious pricing corrections. The traders trying to make money will move away from agreement with consensus market prices, but will move towards agreement with each other, as they notice the same obvious mispricings.

I suppose if the traders all started out with the consensus market prices as their credences, then correcting the market would almost necessarily involve at least temporarily having higher variance in one's credences, so would look like disagreement compared to the initial state. However, the initial market prices, as in the tile case, would tend to represent wide, uninformative distributions; the agents trying to make money would over time develop more specific beliefs, reaching more substantive agreement than they had initially. It's like the difference between agreeing with someone that there's a 50% chance a coin will turn up heads, and agreeing with someone that there's a 99% chance that a coin will turn up heads; the second agreement is more substantive even if there is agreement about probabilities in both cases.

When does the new auto-tagger [LW · GW] run? This post should get the "Rationality" core tag, but I didn't add it myself, because I want to see the bot do it.

I have been thinking about this post since it was posted. I made many, many counterarguments in my head that got defeated pretty quickly. It's excellent. I can learn so much from you.

"Not intellectually substantive" is an interesting insult to make about the nice/mean dimension. Maybe you believe niceness does not exist at all in the far future.

Aiming for convergence on truth. I guess it's true this might lead to a failure mode where one seeks for convergence more than anything else. But taken literally, this should not discourage exploring new wild hypotheses. If you are both equally wrong, by growing your uncertainty you get nearer to converging on truth.