Intransitive Trust

post by Screwtape · 2024-05-27T16:55:29.294Z · LW · GW · 15 comments

Contents

  I.
  II.
  III.
  IV.
  V.
None
15 comments

I.

"Transitivity" is a property in mathematics and logic. Put simply, if something is transitive it means that there's a relationship between things where when x relates to y, and y relates to z, there's the same relationship between x and z. For a more concrete example, think of size. If my car is bigger than my couch, and my couch is bigger than my hat, you know that my car is bigger than my hat. 

(Epistemic status: I am not a math major, and if there's a consensus in the comments that I'm using the wrong term or otherwise making math mistakes I can update the post.)

This is a neat property. Lots of things do not have it.

II.

Consider the following circumstance: Bob is traveling home one night, late enough there isn't anyone else around. Bob sees a shooting star growing unusually bright, until it resolves into a disc-shaped machine with lights around the edges. He finds himself levitated up into the machine, gets poked and prodded by the creatures inside for a while, and then set back down on the road.

Assuming Bob is a rational, rationalist, well-adjusted kind of guy, he now has a problem. Almost nobody in his life is going to believe a word of this.

From Bob's perspective, what happened? He might not be certain aliens are real (maybe he's just had a schizophrenic break, or someone slipped him some interesting drugs in his coffee) but he has to be putting a substantially higher percentage on the idea. Sure, maybe he hallucinated the whole thing, but most of us don't have psychotic breaks on an average day. Break out Bayes.

[WARNING: There's a discussion in the comments suggesting I'm doing the setup and math wrong. Seems plausible, this is my first time doing Bayes with an audience.]

What are Bob's new odds aliens abduct people, given that his experiences? Let's say his prior probability on alien abductions being real was 1%, about one in a hundred. (That's P(A).) He decides the sensitivity of the test - that he actually got abducted, given he had this experience - is 5% since he knows he doesn't have any history of drug use, mental illness, or prankish friends with a lot of spare time and weird senses of humour. (That's P(B|A).) If you had asked him before his abduction what the false positive rate was - that is, how often people think they've been abducted by aliens even though they haven't - he'd say .1%, maybe one in a thousand people have seemingly causeless hallucinations or dedicated pranksters. (That's P(B|¬A).) 

$P(A\mid B)=\frac{P(B\mid A)\cdot P\left(A\right)}{P\left(B\right)}$

$P(aliens\mid experiences)=\frac{P(experiences\mid aliens)\cdot P\left(aliens\right)}{P\left(experiences\right)}$

$P\left(Experiences\right)=P(Experiences\mid Aliens)\cdot P\left(Aliens\right)+P(Experiences\mid \neg Aliens)\cdot P\left(\neg Aliens\right)$

$P\left(Experiences\right)=(0.05\cdot 0.01)+(0.001\cdot 0.99)$

$P\left(Experiences\right)=0.00149$

$P(A\mid B)=\frac{.05\cdot .01}{.00149}$

P(A|B) = 0.3356, or about 33%.

The whole abduction thing is a major update for Bob towards aliens. If it's not aliens, it's something really weird at least.

Now consider Bob telling Carla, an equally rational, well-adjusted kind of gal with the same prior, about his experience. Bob and Carla are friends; not super close, but they've been running into each other at parties for a few years now.

Carla has to deal with the same odds of mental breakdown or secret drug dosages that Bob does. Lets take lying completely off the table: for some reason, both Carla and Bob can perfectly trust that the other person isn't deliberately lying (maybe there's a magic Zone of Truth effect) so I think this satisfies Aumman's Agreement Theorem. Everything else is a real possibility though. She also has to consider the odds that Bob has a faulty memory or is hallucinating or she's misunderstanding him somehow. 

(True story: my undergraduate university had an active Live Action Roleplaying group. For a while, my significant other liked to tell people that our second date was going to watch the zombies chase people around the campus. This was true, in that lots of people looked like they had open wounds, were moaning "Braaaaains," and were chasing after other people. My S.O.'s family correctly did not panic about a zombie apocalypse when they heard we'd watched zombies chase people around the campus despite the literal interpretation of those words being highly concerning. They didn't know what the explanation was, but they assumed the explanation was not the plot of Dawn of the Dead.)

Even if Bob says there's evidence, the aliens did some kind of surgery on him and left a scar the surgical scar isn't ironclad evidence. I have several scars that people don't notice unless I point them out, and I feel like it'd be hard to tell the difference between alien surgery scars, appendectomy scars, and a scar from falling on a nail as a teenager. Plus, and I don't know how to say this politely, but I feel like it's normal to have more confidence in your own mental clarity than that of your friends? 

So when Bob tells Carla a story about the aliens, her prior probability is 1% just like his was. (P(A)). The sensitivity of the test - that aliens actually abduct people, given someone is telling her aliens abducted him - is 2.5% since she doesn't really know his drug habits and hasn't ruled out there's a LARP she's missing the context for. (That's P(B|A).) If you had asked her before this conversation what the false positive rate was - that is, how often people tell someone they've been abducted by aliens even if they haven't - she'd say 1%, they don't usually come up to you at nice parties but she can find a dozen in well chosen reddit thread and people at rationalist parties can be kind of weird. (That's P(B|¬A).) 

$P(A\mid B)=\frac{P(B\mid A)\cdot P\left(A\right)}{P\left(B\right)}$

$P(aliens\mid story)=\frac{P(story\mid aliens)\cdot P\left(aliens\right)}{P\left(story\right)}$

$P\left(Story\right)=P(Story\mid Aliens)\cdot P\left(Aliens\right)+P(Story\mid \neg Aliens)\cdot P\left(\neg Aliens\right)$

$P\left(Story\right)=(0.025\cdot 0.01)+(0.001\cdot 0.99)$

$P\left(Story\right)=0.01015$

$P(A\mid B)=\frac{.025\cdot .01}{.01015}$

P(A|B) = 0.0246, or about 2.46%.

She makes a much smaller jump, even with our magical cheat above that they can trust each other not to deliberately lie. Obviously if Carla thinks Bob might be lying, then she's updating even less towards aliens. If Carla tries to tell her friend Dean about what Bob reports, Dean is going to update even less. At some point in the chain this fairly convincing experience that Bob had becomes equivalent to "somebody I don't know said they got abducted by aliens." That's already true, and I don't believe in alien abductions right now, so I'm not sure how Dean or even Carla would go about convincing me. That said, Carla at least should be updating a little bit towards aliens existing: conservation of expected evidence, aliens aren't less likely in worlds where your friends tell you they personally got abducted by aliens. (Dean might not update at all; "a friend of a friend says they got abducted by aliens" might already be part of his prior.)

Even if Bob trusts his observations a lot, Carla doesn't trust Bob's report of his observations as much. Trust is a little bit, but not completely, transitive. 

III.

In Decentralized Exclusion Jeff K. suggests that relatively decentralized communities like effective altruism dinners or rationality meetups or contra dances can sometimes exclude people, but that it takes a detailed public accusation. I've noticed that detailed public accusations are a high bar to clear: people worry about excessive reputation damage^[1], retaliation, and defamation suits among other things. People disagree about whether the accuser is trustworthy, whether the accusation was shared in the right way, and whether the misdeed is actually as bad as it sounds. What I've seen more of is individual branches of the community internally banning someone, and then news of that ban spreads because organizers talk to each other.

(All specifics in the following example have been changed, but this is a composite of some real examples.) 

Say you're one of the main organizers for a rationality meetup in Dallas, and you find out that Toronto's rationality meetup has banned Emma Lastnamehere. Emma then moves to Dallas and shows up at your meetup. There are internal or at least well attested statements from the organizers of Toronto, but nothing big and public, and it sounds like it was a messy and tiring process. You're acquainted with the Toronto organizers, maybe been to a couple of conferences together or hung out on a shared Discord server, but you're not especially close to them. What do you do about Emma attending your meetups?

Maybe your prior on a random new attendee being a problem is about 0.5%. (That's P(A)). What's your sensitivity and false positive rate? 

A high sensitivity (P(B|A)) means that if Emma really is a problem, a lot of organizers for groups she's attended will have banned her from their meetups. A low sensitivity means that even if Emma is a problem, it's unusual for her to have been banned by other groups.  It's actually pretty rare for a rationalist group to ban someone, so sensitivity isn't super high. I'm going to call it 30% for this example, but you should think about where your number would be.

A high false positive rate (P(B|¬A)) means that even if Emma isn't a problem, she might have gotten banned from a group or have statements from people complaining about her. A low false positive rate means that if Emma isn't a frustrating or troublesome attendee, it's unlikely anyone bans or complains about her. I think the false positive rate on bans is pretty low; obviously it's hard to tell, but I think getting an initial ban from a rationalist group doesn't happen unless there's been a long pattern of problems. I'll use 1% for this example; there are (hand-waves vigorously) about two hundred rationalist groups and it seems reasonable two or three would ban for reasons I didn't agree with. 

$P(A\mid B)=\frac{P(B\mid A)\cdot P\left(A\right)}{P\left(B\right)}$

$P(problem\mid ban)=\frac{P(ban\mid problem)\cdot P\left(problem\right)}{P\left(ban\right)}$

$P\left(ban\right)=P(ban\mid problem)\cdot P\left(problem\right)+P(ban\mid \neg problem)\cdot P\left(\neg problem\right)$

$P\left(ban\right)=(0.3\cdot 0.005)+(0.001\cdot 0.995)$

$P\left(ban\right)=0.01145$

$P(A\mid B)=\frac{.3\cdot .005}{.01145}$

P(A|B) = 0.131, or about 13.1%.

Everyone gets to draw their own lines, but I don't think I'd automatically copy a ban from another group. Maybe it was some kind of personal friction, or the organizer was in the wrong. (Though I want to clearly state that I support the local organizer making that call; well kept gardens die by pacifism [LW · GW], and on the margin I suspect rationalist organizers should do more local bans.) I'd keep an eye on Emma, but she can come.

I picked the words "initial ban" intentionally. The first ban is a lot of evidence. What if you find out Emma got banned from Miami as well? The second ban is a decent chunk more evidence, but it depends how independent the bans were. Let's say for the moment they were completely independent. Miami and Toronto never communicated anything about Emma with each other. Now I'm starting from a prior of 13.1%.

$P(A\mid B)=\frac{P(B\mid A)\cdot P\left(A\right)}{P\left(B\right)}$

$P(problem\mid ban)=\frac{P(ban\mid problem)\cdot P\left(problem\right)}{P\left(ban\right)}$

$P\left(ban\right)=P(ban\mid problem)\cdot P\left(problem\right)+P(ban\mid \neg problem)\cdot P\left(\neg problem\right)$

$P\left(ban\right)=(0.3\cdot 0.131)+(0.001\cdot 0.869)$

$P\left(ban\right)=0.04799$

$P(A\mid B)=\frac{.3\cdot .131}{.04799}$

P(A|B) = 0.8186, or about 81%.

I no longer want Emma at my meetup.^[2] If Emma shows up with an explanation of how the Miami and Toronto organizers are liars and I should disregard everything they say, I'm starting from the prior that she's not a reliable source of information and that accusation doesn't reassure me that I'll be glad to have her around. Trust (and mistrust) does spread through the network, albeit incompletely.

IV.

There's a real hazard of cascading. 

If you told me that someone was banned from six meetup groups, at that point I suspect that groups four through six were probably making their decisions based on the reports from groups one, two, and three. It might be worth talking to them all at least briefly to see where the firsthand evidence is coming from; if group one was the only group Emma previously attended and groups two through six were going off of that first report, maybe I'd give her another chance. 

(It's even possible for reports like this to be an example of citogenesis, where the first report wasn't actually a ban but just an incorrect rumour of a ban, the second group banned based off of that rumour, and the first group heard about the second group and actually banned Emma. Check your sources where you can!)

If you already have the evidence that Emma was banned from six meetup groups, how much does hearing a seventh has banned them change whether you'll ban her from your group? Not a lot. Whatever is going on, you already knew she got banned from meetup groups. At some point, the question becomes how much those groups are updating on each other and whether whatever trait those groups have in common is something you share. If the problem is that someone is too loud and therefore has been banned from six different Librarians' Clubs, and you run a Heavy Metal Jam Club, then maybe you don't care about the bans. 

If trust is too transitive — if every group mirrors each other's ban lists, if you trust every third-hand report of aliens as much as you trust your most honest friend — then they're really only providing as much evidence as one example. This is one of the central flaws in gossip as a method of conveying someone's reputation. Gossip reliably fails to distinguish between one person experiencing a problem and a dozen people experiencing the problem. 

If trust is completely intransitive — if you take no one's word, if you try to figure out everything based solely on your own observations — then you're ignoring genuinely useful signal. This is one of the central flaws in refusing to consider other people's character evaluations. You are going to give Emma the benefit of the doubt, she's going to show up to your meetups, and then you're going to go through a messy and tiring process that you could have skipped. 

V.

I think about Bob and Carla's situation a lot. 

The numbers make it feel like a quantitative difference of degrees, when from the inside it feels like a qualitative difference of some different kind of knowledge. Bob remembers the caustic smell of the alien spaceship and the gut-wrenching sensation of being caught in the tractor beam. He's pointing at the star-shaped puckered scar on his hip, pointing out that surgical scars don't look like that dangit. Maybe it's not aliens, fine, but something weird happened and it's forced him to grapple with whether he can trust his own mind.

Meanwhile, Carla is nodding politely and trying to think of how to get out of this conversation. This is a little weird, Bob seemed like a level-headed guy the last time she saw him, but it's not causing her to rethink her place in the universe. Some people are just weird.

I can't think of a way to bridge this. Even assuming Bob and Carla can't lie, and both know the other person can't lie, it's so much more likely that Bob is failing to bring up some reason he would have hallucinated than that alien abductions happen. I've been in Carla's shoes a few times, and even been on Bob's side in much less dramatic or world-reshaping circumstances. Sometimes there's extra proof you can find, but other times you're just left with someone's word. 

(And the nagging suspicion that they're lying. But the more maddening circumstances are where the lie just makes no sense. Once, someone sounded deeply certain I had said something I hadn't said, something that would have been very uncharacteristic of me to say, a minute or so after I'd supposedly said it. One of us was wrong, and it was such a weird lie to tell I was confused what the point was of telling it.)

Thing is, if Bob expects people to disbelieve him and think he's nuts if he talks about the alien thing, then he doesn't tell anyone. That would be a really dumb way for aliens to successfully hide from humanity; maybe they abducted half of the world, one at a time, and almost nobody talks about it because we each think nobody would believe us. 

But Bob's not wrong! If I had the experience of being abducted by aliens, I don't think anything in my life would go better in the short term from me saying that to people. Long term, it helps everyone else (who can adjust their confidence in me appropriately) and eventually me conditional on this being a mental break that can be fixed by anti-psychotics.

I think there are cases where trust is more transitive. Close knit teams who have worked together for years and shared a lot of models of the world with each other, married couples with solid relationships, your best friend when you both grew up together in a small town and stayed in the town still seeing each other, where if they told you they got abducted by aliens you would actually update as much as if it happened to you. There's one person in my life that, if they told me they got picked up by little green men from mars, I would put higher odds on me hallucinating them saying that than I would on them hallucinating the aliens.

The obvious failure case for the meetup bans is the ban being sufficiently private that other groups never learn about it, leading Emma to get banned from groups one by one after an exhausting process every time. The second, less obvious failure case is Emma getting banned once, a half-dozen other groups mirroring the ban without saying they're just mirroring the first ban, and every other group assuming Emma got banned from seven groups one by one after seven independent decisions. Short term, I don't know if anything gets better for a group that makes a ban public. Long term, it helps everyone else when you're right. It also makes you the target of more focused adversarial action.

I claim trust works best if you say what you think you know, and how you think you know it.

1. ^{^}

   If the problem is that someone is a known serial rapist, then excessive reputation damage might be acceptable. (Though you should hopefully be fairly confident about it!) If the problem is that someone is rude and annoying to work with, then a big public statement can feel like overkill.

2. ^{^}

   Though now that I've said that, I'm going to have to worry about adversarial action. Expect a less organized post at some point about this, but any time you explain how you make decisions, if anyone whose interests are not aligned with yours would want those decisions to come out differently, you need to be at least a little aware you might be getting set up.

15 comments

Comments sorted by top scores.