Formal epistemiology for extracting truth from news sources

Welcome! Your situation sucks, there doesn't seem to be a good solution. Most people don't care about technical truth; they are interested about being in harmony with their friends. In addition to these usual problems, saying certain things in Russia (from my perspective: true things) carries a prison penalty. Be careful lest you explicitly say something illegal (such as "there is a war").

I live in Slovakia, which is a country in EU most susceptible to Russian propaganda. I know this type of people. Every source that disagrees with them is automatically American propaganda; their leader and their fellow followers are apparently the only independently thinking people in the entire universe. (No, it is not suspicious at all.) If you try to argue against that openly, the only thing you can achieve is being called an American propagandist yourself.

What you can do, is nitpick. Like, pretend that you agree with 95% of what they say (so they see you as someone on their side, definitely not an American propagandist) and gently point out that some 5% is factually wrong... not that it changes the big picture at all! just saying. I don't know if this, iterated 1000 times, improves things, but I kinda hope so. I mean, it might get some of them used to the novel idea of "being wrong", and maybe at some moment in future they will apply this new skill to a larger part of their worldview. (It would be too late for the 2022 Ukraine war, but there certainly will be other conflicts in the future.)

I'm interested if there were any attempts at formal rules of transforming media feed into world model. Preferably with Bayesian interference and cool math.

I think this wouldn't work at all. If you start from the position "everyone is lying", then no matter how much you read, it all fits into the "they are lying" hypothesis. Also, it's not like humans can do Bayesian calculations dispassionately when discussing politics.

If you assume that all news sources are directed by the same group of people, then the fact that all of them happen to agree on X doesn't mean that you should multiply the probabilities -- it is actually what you would expect in the world where all news sources are actually directed by the same group of people. (But if they disagree, well, maybe they are just trying to confuse you!)

What I am trying to do, is to read many sources, including the ones I disagree with. And I assume that after reading enough, it will somehow "click" and I will see that some things make sense and some don't. But I wouldn't try to put numbers on that. It is difficult, and it would invite clever people to game the system. At the end, the only person you can educate semi-reliably is yourself. Because it requires honest cooperation, and most people are not going to cooperate at being proven wrong.

they are mostly smart people (knowledge jobs, high IQ).

Sometimes, intelligent people are just better at inventing clever arguments in favor of stupid conclusions. But of course, they would reply that this actually applies to you, so no point telling them.

It is really hard, especially as these highly emotive situations tend to result in the frontal cortex shutting down due to blood supply being diverted. Thus you see otherwise smart people saying unbelievably stupid things. 

My heuristics.:

1. Are they actually experts? Look at their track record. Have they been able to anticipate future events? 

Did they say that a Russian invasion was likely? Did they predict the fall of communism in Eastern Europe? Were they sceptical about past hoaxes like the Gulf of Tonkin incident, the Iraq WMD hoax, the "Itaqis ripping babies out of incubators" hoax? How sceptical about the claims Trump colluded with the Russians on the election? What was their track record on covid?

Have policies they advocated worked? Did they support the invasion of Libya and did they anticipate the results? 

2. Other influences on their stated views. Do their views seem to reflect the ideological landscape? Are their views predictable from the ideological landscape? e.g. Some were against the Trump vaccine but suddenly all for the Biden vaccine, even though it was the same vaccine? Are they for the science in some areas but against it in other areas?

In general (2) (being dominated by ideology) trumps (1) any level of actual skill. 

Media can be OK on some issues but totally ideologically or financially determined in other areas.

Wow, I really wish I had a good answer. Instead the current conflict makes for an almost maximally hostile epistemic environment. I think the realistic thing to do is to accept that it is really hard to know what is actually happening with any certainty. This is by design. The whole problem is anti-inductive [LW · GW] by nature: As soon as you find such a method that works, there will be an incentive by others to circumvent it. Know that there is propaganda and bias on this side too though less overt.

This gloomy view is of course not the same as saying that there is no information to be extracted. Try non-news sources. Like the UN page condemning Russian aggression. I don't know if your family and friends would except that as strong evidence. Does the Pope have any credibility in Russia? I would never trust the Papacy myself as I'm atheistic, but some regard him as an authority on moral matters, and he seems to think there is a war going on at least. 
Perhaps consider Aljazeera as an alternative, clearly-not-US news outlet? They are often used by Swedish media as an alternative to CNN and BBC. Or Indian media, like WION, though WION are prone to hyperbole, and not very thorough in checking their sources. They have been very active on Youtube lately. This clip might be effective at showing that there is a lot of suppression going on, if that was ever in doubt. Of course, asymmetric suppression of information severely reduces the expected truth-value of any information that is allowed to stay, which is obvious to most here, but not necessarily easy to explain or convince people of. 

Or, you know, anything that media on both sides agree on. I've been looking at English rt.com from time to time to get a glimpse of the narrative in Russia. Here are some RT articles that might show cracks in Putins picture. I would be very interested to know if you can access the English version in Russia, and if the translation matches the original.

I would also be very interested to read anything and everything you have to say about what the situation looks like from inside of Russia.

Stay safe. People have been warning that Russia will close it's borders soon. If you want to get out before then, there is no time to loose. I am not saying that you should, and I don't know if you could. If you do, you are welcome to stay here, in Sweden, send me a PM. Last I heard the border to Finland is still open, but requires a Visa. 

Do you know about the street epistemology (Russian site)? I think it is designed for similar cases. In Russia there is the big community. You can visit online-meetings for a practice.

But I think it need to remember that all such conversations need a very much time due to big inferential distances [LW · GW].

I have a similar situation. I adhere to several heuristics.

First, I look at what they say about science, agreement with scientific conclusions is only a weak signal, but if someone believes in the harm of GMOs and vaccines, the effectiveness of homeopathy, and so on, then this is clearly a bad sign.

Secondly, for example, they tell you about a certain fact, you extrapolate it and ask about a discrepancy, to which they offer you a new rule to explain it, and for exceptions to it already the third, such more complex hypotheses are not equal alternatives, they require more rules to explain, and therefore more complex and require more bits of proof just to stay even.

As others pointed out it is very difficult to have a formal theory in these circumstances - but I still believe that we can have heuristics about evaluating sources. Something like https://astralcodexten.substack.com/p/bounded-distrust?s=r

An important case for me personally would be evaluating Bellingcat - I can understand how cynical people believe they are just CIA psyops, or that they could easily be misled by media purposefully planted in the places they look for information - etc. It is a new kind of media/investigation institution and it has many surprising strengths, but most probably also many unexpected weaknesses.

There are few formal rules you can use with certainty here, but if you want better ways of convincing them you’re right, you should play metaculus, good judgement open, manifold, or otherwise engage in some other forecasting site. In enough time, you will be able to see which of you has the best world model, and start using the sources they’re using.

I'm interested if there were any attempts at formal rules of transforming media feed into world model. Preferably with Bayesian interference and cool math. So I can try to discuss these with my friends and maybe even update my own model.

So you are interested in changing other people's minds on a complicated issue that has more to do with the limbic system than rational hardware by using reason. This distribution of influence is one reason why their intelligence isn't really important here, and it is also why your strategy won't work.

More generally, you are in a trap. Be skeptical of your own motivations. The least worst course of action available is probably to disengage.

I think Zvi calls this a hostile epistemic environment since there are actors that try really hard to produce convincing propaganda. Maybe a helpful heuristic is this: Are there checks and balanches for the media? As far as I know, this is hardly the case in Russia right now since independent media outlets have been shut down and you can be jailed for expressing your sincere opinion. This is a very bad sign. (If there were some kind of freedom of speech, more people would be scrutinizing important claims, so that not hearing these critics would be evidence for the truthfulness of these claims, I guess.) Unfortunately, the EU also started blocking Russion state media outlets and thereby complicating the situation, but still, you don't have to worry being jailed for expressing a contrarian opinion.

Besides these quick thoughts, I want to propose a framing of the problem. Assume there's a coin in the world and everybody has high stakes in whether it is fair or biased. Now, different news outlets report what they found out when they flipped the coin themselves. So some report that they got "1000x tails" and others state that their experiments suggest the coin is fair. Maybe they are, technically, both correct in their statements but ignored some coin flips that did not fit into their narrative. [Disclaimer: This doesn't capture everything of real-world news but gives a feeling for the more complex topics where you build your opinion from lots of tiny pieces of evidence.]

The bottom line is that in a no-trust environment (which exists when people with disjoint trusted sources try to communicate), it's not possible to settle whether the coin is fair.

A solution that I find, theoretically, especially exciting is adversarial collaboration. You team up with a person of opposite opinion and devise some kind of experiment (or active observation) that helps settle the diagreement. In the above framing, flip the coin several times in the presence of the other person and follow a previously agreed protocol of determining which side is supported by the evidence.

In practice, this is hard. We (most of us) cannot just go to Ukraine (if we're not already there) to observe what really happens. But what if we think bigger? Imagine thousands of people with diverse opinions of the topic to join forces. They would have a lot more resources to do active observations to reconciliate their differing opinions. For example, as a large group they have better chances to interview important people. If they are honest players, they might agree on a small group of people to travel and actively make observations together. It is also easier for a large group to gather and prominent answers to unsettled questions. The precondition is the honest will to engage with the other side and truthfully settle the disputes.

Unfortunately, this is just a theoretical idea I wasn't able to test in practice, yet -- and it seems hard to imagine to found such an organization in a state where one can be punished for critical inquiry.

Hello! I am Russian too, and I have a similar problem. However, I just try to avoid talking to my friends and family about politics. I know I won't make them change their mind. I'll just upset and probably hurt them by this.

About your question: it is weird how much you trust the Western media. Of course, they are an independent source. But they don't have any motivation to actually find out the truth. They have the idea that "Russia has unprovokedly and unlawfully attacked Ukraine. This war is inhumane and unnecessary. Ukrainians are brave people who defend their homeland", and then they align facts to that idea.

The idea is generally right, but that's not how you seek for facts.

Of course, I do not work for the Western media. It's just the impression their articles give me.

I'd advise you to pay attention to those telegram channels. Not only the ones controlled by Z-men, but also the Ukrainian ones. Especially the Ukrainian ones. If they agree on some fact, then you can be sure that it has actually happened. If Western media supports them, too, then it is solid fact.

And if there are disagreements, you can analyze them to understand different propaganda lines.