Uninformed Elevation of Trust

A CFAR workshop. Just kidding, those are super trustworthy.

I don't know, has anyone been to a CΓAR workshop? I've heard they're oddly pro-smoking, but the website says that is 'only in worlds where smoking doesn't have negative effects. (EDIT: of which this is not one)'.

The Sequences are great, except in my area of expertise, where they are terrible

Can I get an example of a section of The Sequences where someone with the relevant area of expertise would say that that it's terrible?

Here's my attempt to give a concise definition both for Gell-Mann Amnesia (GMA) and your hypothesis, which I'll call Shminux Amnesia (SA). I'll present them in soft form ("adequately"). For the hard form, replace "fail to adequately update" with "fail to update"

GMA: People systematically fail to adequately update their priors on a source's general credibility based on their knowledge of its credibility in their field of expertise.

• Example: A COVID-19 epidemiologists considers the reporting on COVID-19 epidemiology in a certain newspaper to be terrible, but that its reportage is generally adequate. They mildly downgrade their assessment of the reliability of its economic reportage from "adequate" to "mildly problematic." However, economists generally consider that newspaper's economic reportage to be terrible, not just mildly problematic.

SA: People systematically fail to adequately update their priors on the credibility of specific statements based on their knowledge of the credibility the source itself assigns them.

• Example: A reader reads a newspaper reporting what it calls "totally off-base speculation" that a tax increase is about to be announced. The newspaper also refers to a starfish die-off as "extremely likely to be caused by a fungal infection." They regard the newspaper as moderately reliable in general. Prior to reading this article, they had no specific information about the likelihood of a tax increase or the cause of the starfish die-off.
• After reading the two articles, they believe the prediction of a tax increase to be "unlikely, but not too out there," and the prediction of a fungal cause of the starfish die-off to be "fairly likely, but nowhere close to a sure thing."

It's not super clear to me that either GMA or SA are examples of poor epistemic strategies, or that they're especially common. Even if they're true, important, and common, it's not clear what their magnitude is. My personal experience is that I form fairly detailed models of how reliable certain sources are on the topics I care about, and don't bother with the topics I don't care about. I also do neglect to take into account reliability claims to the full extent that would be ideal.

To add on to this, I'd throw one more form of amnesia - Assumption Amnesia. This form of amnesia means that people tend to ignore or neglect the assumptions that motivate theorems and inferences, in areas ranging from mathematics to politics. If they are presented with the following statement:

"We tend to equate the reliability of the data with the subjectively perceived trustworthiness of the source of data whenever we have no independent means of checking the veracity of the data."

What they will remember is this:

"This sounds unobjectionable on the surface. We tend to equate the reliability of the data with the subjectively perceived trustworthiness of the source of data."

And neglect this:

"Whenever we have no independent means of checking the veracity of the data."

The conclusions differ radically if we ignore that assumption.

• With the assumption, we take away the idea that, people lean on the general credibility of the source if they have nothing else to go on.
• Without the assumption (due to assumption amnesia), we take away the idea that people will believe whatever they're told as long as it comes from a source they think is credible, even if it contradicts their own senses.

In general, this cluster of "amnesias" point to an overall tendency of the human mind to radically simplify its models of the world. This can be beneficial to prevent overfitting. But if a key assumption or constraint gets lost, it can lead to major misfires of cognition.

Here is the gist: we trust the data as much as we trust the source, regardless of how much the source trusts the data.

Rant incoming, apologies. This is, sadly, not correct from the get-go. In general, besides your example which more closely is attributed to some form of psychological bias, we tend to lend source importance based on the lack of trust from a source. However, that implies there is any source vetting whatsoever.

I am sure there are a number of individuals here who have worked intelligence, and I am lucky enough to have both worked intelligence and ditched intelligence, so I'm not very interested in my NDA.

There is something incredible about being source of raw intelligence to the point where I have trouble trusting anything I do not hear and verify for myself so many years later. This election drove me absolutely insane. Not only for a particular side's tendencies, but everyone's failure at source vetting. Abandon the 'source'. Find out the true source. I ordered 50 year old magazines to double check a transcription. Nice collector's piece, though. The transcription was right btw.

The source itself is not relevant unless you are the collector; then what matters is not how you present it, but who it goes to, and how it goes to them. However, when you are consuming collected intelligence/information, you have to weigh the medium against the data. All information is neccesarily useful, when used correctly, even misinfo. Misinformation and propaganda neccesarily tells the bias of the consumer/creator, and can lead you in trends, since propaganda tends to be single or sparsely-sourced.

For instance: You can get a reasonable understanding of an event minus all the important bits through the news media.

You can get zero understanding of an event, and a great amount of confusion, from social media.

When I read a news article, depending on the importance of the event (ie: Trump signs stimulus bill!), I will go make sure it is consistent across reporters. If it is, it tells me one of two possibilities:

a. The source all the reporters got was similar or the same. Specific details can tell you this quite easily.

b. All the reporters are in league together and conspiring against this particular news item

Since b is highly improbable (aside from the possibility of accidental conspiring, which is completely in realms of possibility), I generally stick with a.

If it's not very important, I don't waste my time. It was probably a waste of time reading the article; news reporters are very droll. Thankfully, they have mastered the art of the thesis statement.

Given my experience with classified info, how can I rate news media's accuracy or otherwise on relevant subjects they have reported on related to my career?

Absolutely awful, and generally mischaracterizing, if not completely libel. They are easily one of the most dangerous groups that can be unleashed on anything that has the word secret anywhere near it. It is really hard to manipulate the media; they would prefer to not report something if it is not potentially breaking. It's why I think the social media conspiracy is a particularly good one; I can completely believe the or a algorithm can be trained to filter out specific posts, because it's not very hard to do, and those posts are very predictable. I absolutely think it's happening. Was there voter fraud? Probably. Who did it? Probably not us. To bet on there being no foreign meddling in the USA's elections is already a lost bet.

Trump referred to this as more secure than Afghanistan's election. Well, we designed that for them. It was so bad, they agreed to just both be president. I wish I was joking. Ghani just bullied the other individual out and he got the title Peace Negotiator. Thankfully, Trump will get no such title.

This sounds unobjectionable on the surface. We tend to equate the reliability of the data with the subjectively perceived trustworthiness of the source of data whenever we have no independent means of checking the veracity of the data. What is lost in this near-automatic logic is one small piece: the credence the source itself assigns to the data. 

You have no idea how the source assigns credence to the data. It is easy to obfuscate or lie. Lying happens even with people you trust. Imagine how it goes for everyone you don't care about. Well, we all know.

This all neglects source protections, which aren't as important unclassified. Suffice to say, there is nothing more important than the source itself, yet nothing as completely worthless. The importance lies in not understanding why the source is how it is, but rather in keeping it consistent.

Bad information is just as useful as good, what matters is whether you are cognizant of its quality without too much due effort (given the expiry date on data). That is most characterized by subject matter expertise.

/rant