Distribution of knowledge and standardization in science

Rather than speculating in the abstract, the best solution is perhaps to try to come up with concrete standardization ideas, and discuss whether they would work.

Or look at existing efforts. There's two main categories I know of, reporting checklists and quality-evaluation checklists (in addition to the guidelines/recommendations published by professional groups like the APA's manual based apparently on JARS or AERA's standards).

Some reporting checklists:

• STROBE (checklists; justification) for cohort, case-control, and cross-sectional studies
• TREND (checklist; justification) for non-randomized experiments
• CONSORT (checklist; justification) for RCTs
• QUOROM/PRISMA (checklist; justification) for reviews/meta-analyses

Some quality-evaluation scales:

• Jadad scale
• NOS (scale; manual)
• CEBM

I'm generally in favor of these. The recommendations are usually quite reasonable, and a tremendous amount of research neglects the basics or adds in pointless variation - certainly, people will plead that methodologist-nazis will cramp their style, but when I try to meta-analyze creatine and find that in the ~10 studies, they collectively employ something like 31 different endpoints (some simply made up by that particular set of researchers), most for no compelling reason, so that only 2 metrics were used in >3 studies, it's hard for me to take this concern seriously - we clearly are nowhere near an equilibrium where researchers are being herded into using all the same inappropriate metrics.

Obligatory xkcd

IMO what we need is online revision-controlled semantic databases of scientific knowledge which can be edited by all researchers (at least), are freely accessible by everyone and have convenient APIs for data mining.

I feel like I would have gotten more out of this post if there were examples, or at least links to examples of some of the claims.

What is an example of a claim made by historians rooted in folk psychology?

What is an example of a social scientists argument which is flawed and known to be flawed by philosophers?

What are some examples of scientific papers making relatively simple mistakes?

ETA: Even better would be some analysis on how widespread these problems are in their respective fields.

one where researchers follow a more or less standardized procedure. I take it that this is already done to some extent.

I don't see why this is good. For example, crossing the 5% "statistical significance" threshold is a more or less standard requirement to get published. That's not something desirable.

And, of course, there is the overall objection that science is about finding out new stuff and that new stuff does not always fit well (or at all) into the usual/established/standard ways of doing things.

Part of doing science is the development of new methods and terminology.

Rather than speculating in the abstract, the best solution is perhaps to try to come up with concrete standardization ideas, and discuss whether they would work.

No, the best way is too look at standardization effort that actually exist in the real world.

But standardization can also be dangerous as it generates power. The ACS (American Chemical Association) for example introduced CAS numbers for researchers to identify chemical compounds and encouraged researchers in articles to use those compounds to identify what substances they are speaking about.

The problem is that they want you to pay money every time you want to look up a compounds via it's CAS number. They sued Wikipedia for putting CAS numbers in their articles. If you want to mine the literature you have to do hundred thousands of lookups and can't just pay what the ACS wants at per compounds lookup prices.

Given how crapy the new DSM5 has become the NHI made a move to less standardization and encouraged researchers to develop other ways of mental disease classification.

If a biologists finds a method that gives him 99% of the time the results he wants he is really happy. If a mathematician has a methods that gave everyone in history the same result that's an assumption that might or might not be true and is in need of a mathematical proof. Trying to get the biologist and the mathematician to agree on the same terminology is going to be very hard because they have very different standards.

The need for computer understandable text has driven bioinformaticians to publish huge ontologies of vocubulary.

When one reads scientific papers using statistical techniques, one is often struck by the number of relatively simple mistakes. These mistakes are not due to some mysterious lack of tacit "practical knowledge" but are rather quite easy to point out and explicate.

I think that's primarily a problem of peer review where only the colleagues of a researcher are asked but it would be good someone with a statistics background would look over every paper.

At least some of the frontier of science will always be unstardized.

Standards FOLLOW knowledge and are difficult to establish. Standards developed too early are often abandoned or at least deprecated as standards developed after more knowledge prove to be better. In an important sense, English units in science and measurement, gallons, feet, yards, knots, miles, BTU, psi, etc are inferior in many ways to metric a.k.a. SI standards, which came about later. Interestingly SI has not fully driven out English units, but it has certainly deprecated them. Meanwhile, I deal all of the time with things like grams/mile of CO2 emissions from cars, where the astute reader will recognize that this should be lbs/mile or grams/kilometer, but in the US we drive in miles and there is no avoiding that, while our scientists are rather more able to deal with grams of CO2.

SO as long as science is advancing, there will be areas where standards are being threshed out, and probably some areas where they are being threshed out too soon, and so much of the valuable work will be unable to use these nascent standards.

It is not a new observation that standards are useful and so should be developed, but rather an old and ongoing one.

A trivial example: Medical tests are usually reported without units, but there are usually several common units, in particular mass per volume and moles per volume.

Quite often you have the possibility to go for a standard or to avoid doing so. If everyone would just go with the official standard that weeks begin with Mondays, distance is measured in meter and mass get's measured in kg, that would be progress. Getting all those people who derivative for the standard to change their behavior is difficult.

If you really like standardization read a few ISO standards that touch domains in which you publish information.

Of course we can standardise science without much creative loss. We just have to be aware of the potential loss and give space to non-standardised methods. This is similar to how obviously we can standardise education or at least testing without much creative loss. In both cases people, usually teachers, bemoan some kind of loss of freedom or creativity. I posit that in both cases similar incentives stop standardisation being implemented.

More generally, we need a better picture of our social institutions to analyse this kind of situation. What do we want to encourage in science and education? If we can't answer this question clearly we can't give a clear answer to how we should design the specific institutions, among them standardisation.

A modest proposal: Anyone wishing to create a file format will be forced to write out a 10 megabyte example of the format, by hand, without error, before the new format will be accepted.