The Triumph of Humanity Chart

This is so, you can test it yourself if you want Under your approach, the goal is achieving consensus. Under my system, the goal is to provide replicability and show that it actually works.

I think we have to separate two ideas here.

1. There's the data you get from an experiment

2. There's the conclusions you can draw from that data.

I would agree that the data does not depend on the calibration of particular people. But the conclusions you get from that data DO need to be calibrated. Furthermore, other scientists may want to do experiments based on those conclusions... their decision to do that will really be based on how likely they think the conclusions are accurate. The process of science is building new conclusions on the basis of those old conclusions - if it's just about gathering the data, you never gain a deeper understanding of reality.

That seems very different from my perception of science

Aren't both these views of science oversimplifications? I mean, in practice most of the people making use of the work scientists have done aren't really testing the scientists' work for themselves (they're kinda doing it implicitly by making use of that work, but the whole point is that they are confident it's not going to fail).

Reality certainly is the ultimate arbiter, but regrettably we don't get to ask Reality directly whether our theories are correct; all we can do is test them somewhat (in some cases it's not even clear how to begin doing that; I'm looking at you, string theory) and that testing is done by fallible people using fallible equipment, and in many cases it's very difficult to do in a way that actually lets you separate the signal from the noise, and most of us aren't well placed to evaluate how fallibly it's been done in any given case, and in practice usually we have to fall back on something like "scientific consensus" after all.

I think you and MattG are at cross purposes about the role he sees for calibration in science. The process by which actual primary scientific work becomes useful to people who aren't specialists in the field goes something like this:

• Alice does some work where she exposes laboratory rats to bad journalism and measures the rate at which they get cancer. (So do Alex, Amanda, Aloysius, et Al.)
  • She forms some opinions about this stuff; we could, in LW style, represent these opinions as some kind of probability distribution over relationships between bad journalism and cancer. Both her point estimates and her estimates of the distribution around them are strongly constrained by the work she's done, but of course there are probably things she's failed to think of. If she's sensible, her opinions will include explicit allowance for having (maybe) made mistakes and missed things. Such considerations will probably not appear explicitly in the articles she publishes.
• Bob talks to Alice (and Alex, Amanda, ...) or reads the articles they publish.
  • As a result, Bob too forms opinions about this stuff, which again we can represent in probabilistic terms. Bob's knowledge of the actual work is less direct than Alice's, and his opinions are going to depend not only on Alice's observed risk ratios and samples sizes and p-values and whatnot but also on how much he trusts Alice (having read her papers) to have done good work. And of course he will be trying to integrate what he learns from Alice with what he learns from Alex, Amanda et Al.
  • Bob may actually also be a primary researcher in the field, but here we're considering him in his role as someone who has looked at the primary researchers' work and drawn some conclusions.
• Bob and Bill and Beth and Bert and all the other journo-oncologists (some of whom are in fact Alice and Alex etc.) all read more or less the same articles, and talk to one another at conferences, and write articles commenting on other people's work. Over the next few years, journo-oncological opinion converges to a rough consensus that reading the Daily Mail probably does causes cancer, that further work might pin that down further, but that the field has higher research priorities.
• Carol, a non-specialist who wants to know whether reading the Daily Mail causes cancer, talks to some experts in the field or reads a popular book on the subject or even gets into the journals and finds a review article or two.
  • As a result, Carol also forms opinions about journo-oncology. If she has the necessary skills she may also look cursorily at some of the primary literature and get some idea of how rigorous that work is, how big the sample sizes are, whether the research was funded by Rupert Murdoch, etc., but on the whole she's dependent on what Bob and the other Bs tell her. So her opinions are going to be mostly shaped by what Bob says and what she thinks of Bob's accuracy on this point.

Calibration (in the sense we're talking about here) isn't of much relevance to Alice when she's doing the primary research. She will report that the Daily Mail is positively associated with brain cancer in rats (RR=1.3, n=50, CI=[1.1,1.5], p=0.01, etc., etc., etc.) and that's more or less it. (I take it that's the point you've been making.)

But Bob's opinion about the carcenogenicity of the Daily Mail (having read Alice's papers) is an altogether slipperier thing; and the opinion to which he and Beth and the others converge is slipperier still. It'll depend on their assessment of how likely it is that Alice made a mistake, how likely it is that Aloysius's results are fraudulent given that he took a large grant from the DMG Media Propaganda Fund, etc.; and on how strongly Bob is influenced when he hears Bill say "... and of course we all know what a shoddy operation Alex's lab is."

It is in these later stages that better calibration could be valuable, and that I think Matt would like to see more explicit reference to it. He would like Bob and Bill and Beth and the rest to be explicit about what they think and why and how confidently, and he would like the consensus-generating process to involve weighing people's opinions more or less heavily when they are known to be better or worse at the sort of subjective judgement required to decide how completely to mistrust Aloysius because of his funding.

I'm not terribly convinced that that would actually help much, for what it's worth. But I don't think what Matt's saying is invalidated by pointing out that Alice's publications don't talk about (this kind of) calibration.