The Logic of the Hypothesis Test: A Steel Man

The real bullet that is bitten to avoid induction is in step 1 (which is almost always a false dilemma). I see lots of other commenters see this too.

1. Either the null hypothesis or the alternative hypothesis is true.

Nope. This was a good point by Jaynes. The truth may not exist in your hypothesis space. It may be (and often is) something you haven't conceived of.

\4. Therefore under the null hypothesis, our sample is extremely unlikely.
\5. Therefore the null hypothesis is false.

Low likelihood of data under a hypothesis in no way implies rejection of that hypothesis.

\6. Therefore the alternative hypothesis is true.

Without also calculating the likelihood under the alternative hypothesis (it may be less), this is unjustified as well.

As a Bayesian, I'm very happy to see an attempted steelman of hypothesis testing. Too often I see Bayesian criticism of "frequentist" reasoning no frequentist statistician would ever actually apply. Unfortunately, this is a failed steelman (even granting the first premise) -- the description of the process of hypothesis testing is wrong, and as a result the the actual near-syllogism underlying hypothesis testing is not properly explained.

The first flaw with the description of the process is that it omits the need for some kind of ordering on the set of hypotheses, plus the need for a statistic -- a function of from the sample space to a totally ordered set -- such that more extreme statistic values are more probable (in some sense, e.g., ordered by median or ordered by expected value) the further an alternative is from the null. This is not too restrictive as a mathematical condition, but it often involves throwing away relevant information in the data (basically, any time there isn't a sufficient statistic).

The second flaw is that the third and fourth step of the syllogism should read something like "Under the null distribution, a statistic value as or more extreme than ours is extremely unlikely". Being able to say this is the point of the orderings discussed in the previous paragraph. Without the orderings, you're left talking about unlikely samples, which, as gjm pointed out, is not enough on its own to make the move from 4 to 5 even roughly truth-preserving. For example, that move would authorize the rejection of the null hypothesis "no miracle occurred" on these data.

As to the actual reasoning underlying the hypothesis testing procedure, it's helpful to think about the kinds of tests students are given in school. An idealized (i.e., impractical) test would deeply probe a student's understanding of the course material, such that a passing grade would be (in logical terms) both a necessary and a sufficient signifier of an adequate understanding of the material. In practice, it's only feasible to test a patchwork subset of the course material, which introduces an element of chance. A student whose understanding is just barely inadequate (by some arbitrary standard) might get lucky and be tested mostly on material she understands; and vice versa. The further the student's understanding lies from the threshold of bare adequacy, the less likely the test is to pass or fail in error.

In a closely analogous fashion, a hypothesis test is a probe for a certain kind of inadequacy in the statistical model. The statistic is the equivalent of the grade, and the threshold of statistical significance is equivalent of the standard of bare adequacy. And just as the standard of bare adequacy in the above metaphor is notional and arbitrary, the threshold of the hypothesis test need not be set in advance -- with the realized value of the statistic in hand, one can consider the entire class of hypothesis tests ex post facto. The p-value is one way of capturing this kind of information. For more on this line of reasoning, see the work of Deborah Mayo.

My understanding of standardised hypothesis tests was that they serve the purposes of

1. avoiding calculations dependent on details of the alternative hypothesis
2. providing objective criteria to decide under uncertainty

There are practical reasons for both purposes. (1) is useful because the alternative hypothesis is usually more complex than the null and can have lots of parameters, thus calculating probabilities under the alternative may become impossible, especially with limited computing power. As for (2), science is a social institution - journal editors need a tool to reject publishing unfounded hypotheses without risk of being accused of having "unfair" priors, or whatever.

However I don't understand how exactly hypothesis tests help to solve the philosophical problems with induction. Perhaps it would be helpful to list several different popular philosophical approaches to induction (not sure what are the major competing paradigms here - perhaps Bayesianism, falsificationism, "induction is impossible"?), present example of problems where the proponents of particular paradigms disagree about the conclusion, and show how a hypothesis test could resolve the disagreement?

I think a steel man for hypothesis testing should be focused on the types of problems that it can solve better than Bayesian methods can. After all, that's what the purpose of these tests is.

I hope you informed them about the fact that real people often don't satisfy condition (1). Instead they write things like "The patients in the treatment group showed dramatically decreased risk of stroke (p<0.01), indicating the efficacy of Progenitorivox."

it does seem to be a useful steel man.

Find a steel man in there somewhere, and then we can talk about the properties of said steel man.

What's interesting to me about this process is that it almost tries to avoid induction altogether. Only the move from step 4 to 5 seems anything like an inductive argument. The rest is purely deductive - though admittedly it takes a couple premises in order to quantify just how likely our sample was and that surely has something to do with induction. But it's still a bit like solving the problem of induction by sweeping it under the rug then putting a big heavy deduction table on top so no one notices the lumps underneath.

One interesting thing here is that you start with a null vs other hypothesis, but that's because you're doing a two-sample z/t-test. But what's going on when you do a one sample z/t-test and get out a confidence interval?

R> set.seed(12345); t.test(rnorm(20))

    One Sample t-test

data:  rnorm(20)
t = 0.4103, df = 19, p-value = 0.6861
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.3138  0.4668
sample estimates:
mean of x
  0.07652

t.test(data, alternative="greater", mu=c)

R> set.seed(12345); t.test(rnorm(20,100,15))

    One Sample t-test

data:  rnorm(20, 100, 15)
t = 36.16, df = 19, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
  95.29 107.00
sample estimates:
mean of x
    101.1
R>
R> 107-95.29
[1] 11.71
R> 107 - (11.71/2)
[1] 101.1