Individual vs. Group Epistemic Rationality

Even amongst altruists (at least human ones), excessive skepticism can be a virtue, due to the phenomenon of belief bias, in which "someone's evaluation of the logical strength of an argument is biased by their belief in the truth or falsity of the conclusion".

I wonder if there are ways to either teach scientists to compartmentalize in such a way that their irrational skepticism (or skepticism-like mind state) affects only their motivation to find flaws and no other decisions, or to set up institutions that make it possible for scientists to be irrationally skeptical without having the power to let that skepticism affect anything other than their motivation to find flaws.

More in general, in all these cases where it seems human psychology causes irrationality to do better than rationality, it seems like we should be able to get further improvements by sandboxing the irrationality.

In general the ability to fool someone, and trick them into believing something they wouldn't really want to believe if they knew what you know, can be leveraged into tricking people to work toward some group goal, against their personal preferences. So sure it is possible that the ways that people are already tricked may happen to align in order to promote group goals. But really, how believable is it that human error tends to actually promote group goals? Sure, sometimes it will, and there will be some weak group selection features, but mostly no, individual errors are mostly also group errors.

Assertion for the purpose of establishing the nature of possible substantive disagreement:

Anything an irrational agent can do due to an epistemic flaw a rational agent can do because it is the best thing for it to do.

(Agree?)

ETA: The reason I ask is that I think the below:

But I argue that from a group perspective, it's sometimes better to have a spread of individual levels of confidence about the individually rational level. Perhaps paradoxically, disagreements among individuals can be good for the group.

... applies to humans and not pure rational agents. It seems to me that individuals can act as if they have a spread of confidence about the individually rational level without individually differing from the ideal subjectively-objective beliefs that epistemic rationality prescribes to them.

I am trying to establish whether you agree with my original assertion (I think you do) and then work out whether we agree that it follows that all the benefits from having individuals act as if they differ from the individually rational level is just as good as having them actually have the wrong confidence levels. If not then I am trying to understand why we disagree.

I don't really think that you actually need to focus on ends. I don't believe in homeopathy and I'm still perfectly capable of seeing that a lot of people who label themselves as Rationalists or Skeptics make stupid arguments when they argue against homeopathy because they misstate the claims that people who practice homeopathy make.

You can either focus on creating good arguments or you can focus on good maps of reality. Your event that has probability of 0.999 might break down into ten arguments which while they are strong together can still be questioned independently.

There's for example the claim that according to the doctrine of homeopathy all water on earth should have homeopathic powers because all water contains small amounts of nearly everything. That just not true as homeopathists follow a certain procedure when it comes to diluting their solutions with involves diluting the substance in specific steps and shaking it in between.

Let's say there a bacteria which builds some form type of antibody when you add poison into a solution. Let's when one of the bacteria who doesn't produce antibodies come in contact with lots of antibodies and feels a lot of physical pressure that bacteria copies the antibody design that floats around and targets the poison and produces antibodies as well to defend itself against the poison.

It wouldn't violate any physical law for such a bacteria to exist and do it's work when you dilute enough at each step to get new bacteria who weren't exposed to antibodies and shake to give the bacteria the physical pressure that it needs to copy the antibody design.

If such an bacteria or other agent would exist than it's plausible that the agent could work under the protocol of (dilute by 1:10 / 10*shake)^20 but the bacteria or other agent wouldn't do the work in the absence of that protocol in the free ocean.

Now I know that homeopathy uses distiled water and it's therefore unlikely that there any bacteria involved but that still negates the ocean argument and the suggestion that all water should work as homeopathic solutions if homeopathy is right.

Seeking to make good arguments might be a better goal than always thinking about the ends like whether homeopathy is true in the end.

I'm inclined to agree with your actual point here, but it might help to be clearer on the distinction between "a group of idealized, albeit bounded, rationalists" as opposed to "a group of painfully biased actual humans who are trying to be rational", i.e., us.

Most of the potential conflicts between your four forms of rationality apply only to the latter case--which is not to say we should ignore them, quite the opposite in fact. So, to avoid distractions about how hypothetical true rationalists should always agree and whatnot, it may be helpful to make explicit that what you're proposing is a kludge to work around systematic human irrationality, not a universal principle of rationality.

Somewhat related:

"The reasonable man adapts himself to the world; the unreasonable man persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. (George Bernard Shaw)"

I argue that from a group perspective, it's sometimes better to have a spread of individual levels of confidence about the individually rational level.

In the same way that a depressive person can bring benefits to a group in the form of reducing irrational exuberance; in the same way an uneducated person can bring benefits to the group in the form of novel ideas outside the normally taught methods; in the same way a person with mania can bring benefits to a group by accomplishing a great deal in a short time, despite burning out quickly; etc.

Human diversity allows for many tricks to overcome stagnant agreement, but the key word in the sentence I quoted above is "sometimes." I would also say "some" people should have negative traits, to provide a set of human minds that will reach in areas not sought by more rational, intelligent, balanced, well-educated people, on the off chance there might be something interesting there. But it should not be the norm, or the majority, or the most applied method.

Proportionally, I'd say your "sometimes" is likely less than 10 percent.

I think our shared goal, as indicated by the motto "rationalists win", is ultimately to improve group instrumental rationality.

I wouldn't say that is the dominant shared goal and also find that 'rationalists win' provokes a cringe reaction in many listeners who consider the motto inaccurate. But this is an interesting question in it's own right and I am now somewhat curious about your reasoning and also what intuitions others have about how the various motivations of lesswrong participants is spread among the 4 categories mentioned.

Closely related to your point is the paper, "The Epistemic Benefit of Transient Diversity"

It describes and models the costs and benefits of independent invention and transient disagreement.

Mancur Olson's The Logic of Collective Action might serve as a very useful theoretical tool here, for talking about groups. We might extend Olson's analysis by thinking of how different kinds of groups produce rationality and scientific information.

I'm sorry; how is scientific knowledge a public good? Yes, it is nonrivalrous in consumption, but certainly not nonexcludable. Legitimate, peer-reviewed journals charge for subscriptions, individual issues, or even for individual articles online.

if we're looking for ways to spread out into shakier ground than the top posts generally cover I must reiterate my desire for an "ask lesswrong" tab where people can post much less rigorous ideas for feedback.

Perhaps we should appoint devil's advocates from a pool of volunteers to each argue for one item from a list of positions we largely agree are false but could be disastrously wrong about.

Individual epistemic rationality you're talking about here seems to be some sort of mildly biased version of the version we're usually talking about here. That's misleading

We place a lot of emphases here on calibrating individual levels of confidence (i.e., subjective probabilities), and on the idea that rational individuals will tend to converge toward agreement about the proper level of confidence in any particular idea as they update upon available evidence.

No individual has time to look at all evidence, therefore it makes sense when different individuals look at different evidence. In any question where all evidence is integrated into the running model it might be best to run new experiments that produce new data.

It's not like all data confirms our ideas. The number of open question in science seems to grow rather then decrease. It makes sense to focus attention in areas where we don't understand what's going on.

Perhaps a member of a well organized group of bounded rationalists could, without deviating from probabilities ey would have as an individual rationalist, promote to active consideration certain theories with a lower probability than ey otherwise would actively consider, on the basis that the organization of the group systematically assigns different theories with similar low probabilities to other group members, such that if any member finds that the probability for one of eir assigned theories increases past a certain threshold, it can be promoted for the entire group's consideration, by presentation of the accumulated evidence.

This sort of arrangement should allow use of the parallel processing available by having many group members, without requiring the members to give up individual rationality.

Good point Somehow I've always been attracted to taking the road less taken. Maybe there is a group level reason for that, it would be nice if it turns out my actions are not entirely egoistic. I don't make much of a distinction between processes inside humans and processes between humans. For me, pjeby's akrasia theory about conflicting motor programs can be explained in a similar way as you are doing now for scientific progress that uses differences between scientific theories as a way to generate enough diversity for an optimization process to work with. In that light I am a bit skeptical about the negative attitude about akrasia I often find here (even the term itself seems a bit hostile). In fact, observing different motor programs or scientific theories battling it out amongst themselves can be a way to gain knowledge, it should not be something to be fought. Even if the desired outcome is one of coherent action, it matters a lot which action, and we need to keep that pool of optional actions alive as much as we need to effectively handle a current situation. Maybe it would be possible to find a balance between exploration and solidification (does anyone know a better term?) with a mathematical or theoretical basis. It could be akin to the heuristics used in programming genetic algorithms.