Insights from "All of Statistics": Statistical Inference

post by Rafael Harth (sil-ver) · 2021-04-08T17:49:16.270Z · LW · GW · 0 comments

Contents

  4. Fundamentals of Statistical Inference
  5. Bayesian Parametric Inference
    5.1. The concept
    5.2. A discrete Example
    5.3. A continuous Example
    5.4. Applications
  6. Frequentist Methods
    6.1. The concept
    6.2. Evaluations
    6.3. An Example
    6.4. Commonly used Estimators
      6.4.1. Maximum Likelihood Estimator
      6.4.2. Empirical Distribution Function
      6.4.3. Method of Moments
  7. Hypothesis Testing and p-Values
  8. Failures
    8.1. ... of p-values
      9.1.1. EXPERIMENT ONE
      8.1.2. EXPERIMENT TWO
      8.1.3. THE FAILURE
    8.2. ... of Bayesian inference
  9. Statistical Decision Theory
  10. Other Concepts
    10.1. Score and Fisher Information
    10.2. Bootstrapping
    10.3. The Jacknife
    10.4. Simulation
    10.5. Multiparameter Models for Parametric Inference
    10.6. Various tests
    10.7. Admissibility
      10.7.1. Stein's Paradox
    10.8. Plug-in Estimators
None
No comments

(This is the second of two posts on the textbook All of Statistics. Click here [LW · GW] for post I.)

4. Fundamentals of Statistical Inference

Probability theory is about using distributions to derive information about outcomes. Given the assumption that $X\sim \text{Poisson}\left(3\right)$, we can compute probabilities of outcomes the form $Pr(X=k)$. Statistics is about the opposite: using outcomes to derive information about distributions.

The book exclusively considers the setting where we observe independently sampled data points $x_1,...,x_n\sim X$ where the distribution of $X$ is unknown. It is often convenient to talk about the RV that generated the $k$-th sample point, the logical notation for which is $X_k$. The book uses upper-case letters for both (e.g., 'we observe data $X_1,...,X_n$"), not differentiating between the RV that generates the $k$-th point (i.e., $X_k$) and the k-th point itself ($x_k$).

On the highest level, one can divide all inference problems into two categories: parametric inference and non-parametric inference. In parametric Inference, we assume that we know the family of distributions that $X$ belongs to (Binomial, Poission, Normal, etc.). In this case, the problem reduces to inferring the parameters that characterize distributions in that family. For example, if we observe data on traffic accidents, we may assume that $X\sim \text{Poisson}\left(\lambda \right)$, and all that's left to estimate is $\lambda$. Conversely, in the context of non-parametric inference, one does not assume that $X$ belongs to any one family and thus has to estimate the distribution directly.

5. Bayesian Parametric Inference

5.1. The concept

Let $\Theta$ be the parameter we wish to learn about. Since we don't know its value, we treat it as a RV that ranges over several possible values. Furthermore, let $x=(x_1,...,x_n)$ be our observed data, and let $X=(X_1,...,X_n)$ be the RVs that generate this data.

Bayes' theorem is an equation relating ${Pr}_B\left(A\right)$ to ${Pr}_A\left(A\right)$. (With $P_Y\left(X\right):=P\left(X|Y\right).)$ In our case, we are interested in terms of the form $P_{[X=x]}(\Theta =\theta )$, and we know how to compute terms of the form $P_{[\Theta =\theta ]}(X=x)$ because once we fix a value of $\Theta$, the distribution is known and we can compute the required probabilities. Thus, Bayes' theorem gives us a way to reduce a problem of statistical inference to a problem of probability theory.

Alas, we have

$$P_{[X=x]}(\Theta =\theta )=P_{[\Theta =\theta ]}(X=x)\frac{P(\Theta =\theta )}{P(X=x)}$$

Our prior on $\theta$ will virtually always be continuous. Let $h$ be the pdf of this prior. The $X_i$ may be continuous or discrete. Let $f$ be their pdf or pmf. Our formula becomes

$$h_{[X=x]}\left(\theta \right)=f_{[\Theta =\theta ]}\left(x\right)\frac{h\left(\theta \right)}{f\left(x\right)}$$

Note that the denominator does not depend on $\theta$: once we observe the data, how likely it was to appear has no impact on how we weigh different values of $\Theta$ against each other. Thus, we have

$$h_{[X=x]}\left(\theta \right)\propto h\left(\theta \right)f_{[\Theta =\theta ]}\left(x\right)$$

Suppose we begin by computing the right side. If $f$ is a pmf, the result is an incorrectly scaled pdf (since we multiply a density with a product of masses, which is just a scalar). If $f$ is a pdf, the result will be the product of a density with a product of other densities. If you think of a density as a mass multiplied by an infinitesimal, then it's a term with an infinitesimal to the power of $n+1$. However, one can ignore this fact entirely, compute the integral of the [non-density-function obtained by evaluating the right side above], then scale that non-density-function by the inverse of that integral, and the result will be the correct density function, $h_{[X=x]}$ (I really need to study nonstandard analysis at some point). Since computing $f\left(x\right)$ is usually harder than computing this integral, this is the standard approach.

Using the fact that the $x_i$ are independently sampled, we can further rewrite the formula as

$$h_{[X=x]}\left(\theta \right)\propto h\left(\theta \right)\prod _{i=1}^n{f}_{[\Theta =\theta ]}\left(x_i\right)$$

The only open question is how to choose the prior distribution on $\theta$. The philosophically correct thing to do is to use ones prior beliefs about $\theta$, but this has the issue of making methods depend on the person carrying them out. There are various methods suggested in the literature that construct analytical priors, but there is no consensus as to which one is best.

5.2. A discrete Example

Let $x_1,...,x_n\sim X$ with $X\sim \text{Bernoulli}\left(\Theta \right)$. Note that here, $\Theta$ is a probability, so any reasonable prior should put zero density to values outside of $[0,1]$. For now, assume our prior on $\Theta$ is uniform, i.e., $h\left(\theta \right)=1$ for $\theta \in [0,1]$.

Define $k:={\sum }_{i=1}^n{x}_i$. We have

$$h_{[X=x]}\left(\theta \right)\propto h\left(\theta \right)\prod _{i=1}^n{f}_{[\Theta =p]}\left(x_i\right)={\theta }^k(1-\theta {)}^{n-k}$$

Usuually, we would now have to compute the integral ${\int }_0^1{\theta }^k(1-\theta {)}^{n-k}\text{d}p$. However, in this case, our posterior has a Beta distribution, $h_{[X=x]}\sim \text{Beta}(k,n-k+1)$. This means we already know that the integral sums up to $\frac{\Gamma (n+1)}{\Gamma \left(k\right)\Gamma (n-k+1)}$, which means that our posterior pdf is

$$h_{[X=x]}\left(\theta \right)=\frac{\Gamma \left(k\right)\Gamma (n-k+1)}{\Gamma (n+1)}{\theta }^k(1-\theta {)}^{n-k}$$

Here is a plot of $h$ (black) and $h_{[X=x]}$ (red) for $n=10$ and $k=7$:

5.3. A continuous Example

Let $X\sim \text{Uniform}(0,\theta )$, and let $x_1,x_2,x_3$ be samples of $X$. Assume we have prior $h\left(\theta \right)=\frac{1}{1+{\theta }^2}$ for $\theta \in R_{+}$. Let $y:=max\{x_1,x_2,x_3\}$ and suppose that $y=1$. Then,

$$h_{[X=x]}\left(\theta \right)\propto h\left(\theta \right)\prod _{i=1}^n{f}_{[\Theta =\theta ]}\left(x_i\right)=\frac{1}{1+{\theta }^2}\prod _{i=1}^n{f}_{[\Theta =\theta ]}\left(x_i\right)=\{\begin{array}{cc}\frac{1}{1+{\theta }^2}{\theta }^{-3}& ;\theta \ge 1\\ 0& ;\theta <1\end{array}$$

Therefore,

$${\int }_{-\infty }^{\infty }\text{not-}{h}_{[X=x]}\left(\theta \right)\text{d}\theta ={\int }_1^{\infty }\frac{{\theta }^{-3}}{1+{\theta }^2}\text{d}\theta ={\int }_1^{\infty }\frac{1}{{\theta }^3+{\theta }^5}\text{d}\theta =\frac{1-\ln \left(2\right)}{2}$$

which means we need to scale by $\frac{2}{1-\ln \left(2\right)}$, and our posterior pdf is

$$h_{[X=x]}\left(\theta \right)=\frac{2}{(1-\ln (2\left)\right)({\theta }^3+{\theta }^5)}$$

Here is a plot of $h$ (black) and $h_{[X=x]}$ (red):

5.4. Applications

The book mentions three distinct use cases for statistical inference:

• Point-estimation: provide a best guess for a specific object, like the parameter of interest.
• Confidence intervals: give an interval such that we can say with a given certainty that the parameter is in this interval. This is often reported as 'the margin of error'. If a pollster reports that 27% of people like something and the margin of error is 3 points, this probably means that $\theta \in [0.24,0.3]$ with 0.95% confidence. There is nothing special about $0.95$, but it's the convention.
• Hypothesis testing, which we'll cover in chapter 7.

To obtain a point estimate, one may take the mean, median, or mode of the posterior distribution. To obtain a confidence interval, suppose we want certainty $90$. Then, we need only find numbers $a,c$ such that

$${\int }_a^c{h}_{[X=x]}\left(\theta \right)\text{d}\theta =0.9$$

If we want, we can always choose $a,c$ such that our point estimate $\hat{\theta }$ of $\Theta$ is exactly in the middle of the interval $[a,c]$.

6. Frequentist Methods

6.1. The concept

The prevalent school of thought in statistics dictates that the unknown parameter $\theta$ is not a RV but a constant. Therefore, they reject Bayesian methods as they make probability statements about $\theta$. Instead, Frequentist methods deal in terms of what are called estimators. An estimator is an arbitrary function from the data that is meant to estimate a certain quantity (such as a parameter of interest). Then, there are rigorous analytical methods to evaluate estimators.

There are a bunch of commonly used estimators that are applicable to many problems. However, for any specific problem, one may always construct a nonstandard estimator.

In this chapter, we use $\Theta$ to denote a space of possible values for the unknown parameter, and we denote the parameter itself as $\theta$. (We don't need a symbol for the variable in the pdf of $\theta$ because it doesn't have a pdf.) Also, we can no longer condition on $\theta$ having a certain value, so we write $f_{\theta }$ instead of $f_{[\Theta =\theta ]}$, and think of $f$ as denoting a family of probability densities with $f_{\theta }$ being one of them, rather than as one density that we can update on the event $\Theta =\theta$.

6.2. Evaluations

Suppose we observe samples $x_1,...,x_n$ from some distribution parametrized by $\theta$. Let $f$ be an estimator of $\theta$. Since $f$ is a function of the data and we can view each data point $x_k$ as generated by a random variable $X_k$, we can view $f$ itself as a random variable that is some function of the $X_k$. For example, if we wish to estimate the mean of the distribution, then $f=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ is the obvious choice. This means we can compute

$$E\left(\right(f-\theta {)}^2)$$

which is called the Mean Squared Error. Note that computing the mean Squared Error will yield a term involving $\theta$, something like $\frac{\theta }{n+n^2}$.

If $E\left(f\right)=\theta$, the Mean Squared Error equals $V\left(f\right)$. In this case,

$$\text{bias}\left(f\right):=E\left(f\right)-\theta$$

is 0, and we call the estimator unbiased.

On first glance, being unbiased sounds like a good thing. It took me a while to understand exactly why, form a Bayesian perspective, bias is desirable:

• The obvious Bayesian point estimate for $\theta$ is the mean of the posterior distribution.
• For this estimate to be unbiased, it would have to be the case that, if we condition on $\Theta =5$, in expectation (that's expectation over the generated data given that $\Theta =5$) the posterior distribution of $\Theta$ centers around 5.
• However, a proper Bayesian update (even on data that is typical for the case that $\Theta =5$) only moves the distribution some portion of the way toward the evidence, not the entire way.
• Thus, the posterior distribution's mean will be closer to 5 than that of the prior distribution, but it won't be all the way there.

In other words, Bayes estimators are systematically biased toward the prior distribution, whereas Frequentist methods do not include a prior and hence can be unbiased. A recurring theme is that Frequentist methods tend to look like (or even be identical to) Bayesian methods with flat priors.

6.3. An Example

Consider again the case of a continuous RV $X\sim \text{Uniform}(0,\theta )$ with $\theta$ unknown, where we have samples $x_1,...,x_n$ of $X$. In this case, any function $f:R^n\to R_{+}$ that maps the data to an estimated value of $\theta$ is a valid estimator.

Here are two estimators that make intuitive sense:

• $f_1:(x_1,...,x_n)\mapsto max\{x_i:1\le i\le n\}$ (if $n$ is large, then the maximum should be close to the largest sample)
• $f_2:(x_1,...,x_n)\mapsto \frac{2}{n}{\sum }_{i=1}^n{x}_i$ (the maximum should be about twice the mean)

In this case, the first estimator is biased ($f_1$ is, in fact, guaranteed to be smaller than $\theta$), whereas the second estimator is unbiased. Which estimator is better?

At first, this seems completely unclear. You could now do two things. One is to compute the Mean Squared Error (this can be done exactly). The other is to take a look at what the Bayesian estimator was doing. In particular, we can see that the posterior distribution is solely determined by the maximum of the $x_i$. This suggests that $f_1$ is better, which is indeed true. (It can be scaled by $\frac{n+1}{n}$ to make it unbiased, but even the biased version is superior if $n$ is large.)

6.4. Commonly used Estimators

6.4.1. Maximum Likelihood Estimator

The Maximum Likelihood Estimator is the function outputting the value for $\theta$ such that the probability mass or density for the observed data is maximized, i.e.,

$${\text{argmax}}_{\theta \in \Theta }\prod _{i=1}^n{f}_{\theta }\left(x_i\right)$$

Thus, the Maximum Likelihood estimator outputs the mode of the posterior distribution obtained by doing a Bayesian Update with a flat prior. In the example from 6.3, the Maximum Likelihood estimator is just $f_1$.

In the case where $\theta$ denotes a probability, we have $\Theta =[0,1]$ and a flat prior is possible. In the case where $\Theta =R_{+}$, a flat prior is mathematically impossible in standard analysis because ${\int }_0^{\infty }c\text{d}\theta$ does not converge. It also "implies" that $Pr[\theta <{10}^{1000}]\approx 0$ since that interval is infinitely smaller than $R_{+}$ and our prior is uniform.

6.4.2. Empirical Distribution Function

In non-parametric inference, the empirical distribution function is an estimate for the distribution that generated the data. It simply puts probability mass $\frac{1}{n}$ on every data point $x_i$. Let $f_{\text{EDF}}$ be this function. It's pretty easy to see that, for any probability mass function $f$, we have

$$\prod _{i=1}^{n}f\left(x_i\right)\le \prod _{i=1}^n{f}_{\text{EDF}}\left(x_i\right)$$

Thus, the empirical distribution function is the maximum likelihood estimator applied to the probability distribution rather than to a parameter.

6.4.3. Method of Moments

The method of moments is a method for parametric inference that is not a special case of Bayesian Inference. Recall that a statistical functional provides a single number that encodes some property of a RV. Suppose that we can estimate this number for an observed data set. Then, we can estimate the value of a parameter $\theta$ in two steps:

• Estimate the statistical functional for the data set
• Choose $\theta$ such that the statistical functional of $f_{\theta }$ equals that number

If we have several parameters to estimate, just do the same with several statistical functionals. This leads to a system with $k$ equations and $k$ unknowns.

The statistical functionals we use are the moments, hence the name. Recall that the $k$-th moment is defined as

$${\mu }_k:={\int }_{-\infty }^{\infty }{x}^{k}f\left(x\right)\text{d}x$$

We estimate the $k$-th moment of a data set $(x_1,...,x_n)$ in the obvious way:

$${\hat{\mu }}_k:=\frac{1}{n}\sum _{i=1}^n{x}^k$$

Since the value of the ${\mu }_k$'s depends on the pdf $f$, which depends on our parameters, computing the ${\mu }_k$ yields a term that includes those parameters, whereas computing the ${\hat{\mu }}_k$ just yields numbers. Thus, if we have two parameters, then

$${\mu }_1={\hat{\mu }}_1{\mu }_2={\hat{\mu }}_2$$

is actually a 2 by 2 system of equations, with the variables being our two parameters.

The method of moments is not super accurate, but has the advantage of being computationally cheap.

In the example from 6.3, ${\mu }_1=E\left(f\right)=\frac{\theta }{2}$ and ${\hat{\mu }}_1=\frac{1}{n}{\sum }_{i=1}^n{x}_i$. Thus, the method of moments estimator is just $f_2$.

7. Hypothesis Testing and $p$-Values

Suppose we want to learn about some parameter $\theta$. In this case, think of $\theta$ as measuring some effect we care about, say the difference in [frequency of trait X among people with trait Y] and [frequency of trait X among all people]. Let $\Theta$ be the parameter space of $\theta$, and let ${\Theta }_0⊊\Theta$ be the subset in which "there is no effect". The claim that there is no effect, i.e., that $\theta \in {\Theta }_0$ is called the null hypothesis, and the goal of a test is to refute the null hypothesis.

Let $x_1,...,x_n\sim X$ be our data. Let $S$ be a function from the data to $R$. Formally, this is the same as an estimator, but here we call it the test statistic. $S$ should be such that it yields a high number if there is an effect and a low number if there is not (or vice versa).

Once we evaluate $S$, we get some number $x\in R$ that is somewhere on the less-effect-y $\leftrightarrow$ more-effect-y axis. Let $R⊊R$ be the subset of results that looks at least as effect-y as $x$. Here's a visualization:

We define the $p$-value of this test as the maximum chance that the result landed in $R$ even though the null hypothesis was true, i.e.,

$$p\text{-value}:=\underset{\theta \in {\Theta }_0}{sup}Pr(S\in R)$$

And that's it; that's a $p$-value. This is all well-defined since, once a $\theta$ is fixed, $S$ is just a RV, and since $R$ is a subset of the form $\{z\in R:z\ge z_0\}$ the condition $S\in R$ can be rewritten as $S\ge z_0$.

In the ideal case, a $p$-value of 0.002 means "under the assumption that the effect we are testing for doesn't exist (as defined by our choice of ${\Theta }_0⊊\Theta$), there was at most a 0.2% chance for the data to show as much of an effect or more as we have observed". Not that this is not a probability statement about $\theta$. (Frequentists don't want to make probability statements about $\theta$ since it's a constant.) The next chapter will work through an example of how to use $p$-values (which is also an example of where they 'fail', and it will illustrate why saying 'it's not a probability statement about $\theta$' is more than a technicality).

In classical Hypothesis Testing, one defines the region $R$ ahead of time, evaluates $S$ on the data, and reports a binary 'null-hypothesis rejected' if $S\in R$ and 'null-hypothesis not rejected' if $S\notin R$. This is much worse since now the choice of $R$ is arbitrary, and one can always find two points $y,z$ on $R$ such that $z$ suggests a 0.00000000000000001% stronger effect than $y$, yet $z\in R$ and $y\notin R$. Compared to this, reporting a $p$-value is amazingly informative.

8. Failures

8.1. ... of $p$-values

Suppose we have a not-necessarily-fair coin with unknown probability $p$ of getting heads. We want to refute the null hypothesis 'the coin is fair' defined by the one-point parameter subspace ${\Theta }_0:=\left\{0.5\right\}⊊[0,1]$.

Here are two possible experiments to test this hypothesis.

9.1.1. EXPERIMENT ONE

Experiment one is: "flip the coin twelve times". Our sample statistic is the RV $X$ that counts the number of heads. Obviously, $X\sim \text{Binom}(12,p)$. We obtain the result $X=3$. The region $R$ (of all outcomes that are at least as extreme as the observed one) is $\{0,1,2,3\}$. The $p$-value is thus

$$\underset{\theta \in {\Theta }_0}{sup}Pr(X_{\theta }\in R)=Pr(X_{0.5}\le 3)=\sum _{k=0}^3(\frac{12}{k}){0.5}^{12}\approx 0.073$$

8.1.2. EXPERIMENT TWO

Experiment two is: "flip the coin until it lands on heads three times". Our sample statistic is the RV $Y$ that counts the number of tosses that came up tails. We have $Y\sim \text{NegativeBinomial}(3,p)$. (The negative binomial distribution hasn't made it into the book or into my previous post; it's a generalization of the Geometric distribution that counts to $r$ misses rather than to 1.) We obtain the result $Y=9$ (i.e., nine tails, three heads). The region $R$ (of all outcomes that are at least as extreme as the observed one) is $\{9,10,11,12,13,14,15,...\}$. The $p$-value is thus

$$\underset{\theta \in {\Theta }_0}{sup}Pr(Y_{\theta }\in R)=Pr(Y_{0.5}\ge 9)=\sum _{k=9}^{\infty }(\frac{k+3-1}{k}){0.5}^3{0.5}^k\approx 0.0328$$

8.1.3. THE FAILURE

The first experiment yielded a $p$-value of 0.073, whereas the second one yielded a $p$-value of 0.0328. To see why this is an issue, consider a Bayesian update. Let $h$ be an arbitrary distribution over $\Theta$ (now uppercase since we treat it as a RV). In experiment one, we have twelve samples of $X$ with $3$ heads and $9$ tails, hence

$$h_{[X=x]}\left(\theta \right)\propto h\left(\theta \right)\prod _{i=1}^n{f}_{[\Theta =\theta ]}\left(x_i\right)=h\left(\theta \right){\theta }^3(1-\theta {)}^9$$

In experiment two, we have twelve samples of $X$ with $3$ heads and $9$ tails, hence

$$h_{[X=x]}\left(\theta \right)\propto h\left(\theta \right)\prod _{i=1}^n{f}_{[\Theta =\theta ]}\left(x_i\right)=h\left(\theta \right){\theta }^3(1-\theta {)}^9$$

Thus, since both experiments have observed the same data, the posterior distribution is exactly the same in both cases (provided that both use the same prior.) Nonetheless, the experiments have yielded different $p$-values -- and what's worse, the expected $p$-value is also different! Given a fixed $\theta$, the second experiment systematically produces lower $p$-values, both in terms of their mean and in terms of their median.

Despite having heard lots of people rail against $p$-values in the past, I was pretty shocked by this result. I had previously thought that $p$-values are bad because they provide bad incentives and give misleading impressions. I didn't know that they have an in-principle arbitrary component.

I don't think I understand why $p$-values are lower in the second experiment, though. What properties of the distribution are responsible? If anyone knows, please explain.

8.2. ... of Bayesian inference

Bayesian Inference relies on Bayes theorem, which is a theorem. Can it really fail? We shall see. The book mentions two cases meant to showcase weaknesses of Bayesian inference.

The first one is very silly. It comes down to the following experiment where we have a googol biased coins: "randomize a number $k$ between $1$ and ${10}^{100}$, report the result of a flip by the $k$-th biased coin; repeat $N$ times". Then, the book observes that, for any realistic $N$, using Bayesian inference on the ${10}^{100}$ unknown probabilities of the biased coins will fail since most of them are never observed. Of course, this has zero to do with Bayesian methods and everything with what we choose to update; if one estimates the total probability of getting heads, Bayesian inference does wonderfully.

The second one is much more interesting. Let $g:R_{+}\to R_{+}$ be a continuous, integrable function such that ${\int }_0^{\infty }g\left(x\right)\text{d}x=:c\in R$. In general, $g$ is not a pdf, but $f:=\frac{1}{c}g$ is.

Suppose we know $g$ and want to estimate $c$. One way to obtain $c$ is by computing the integral, but this may be difficult. Instead, suppose we are given points $x_1,...,x_n$ that are independently sampled according to $f$.

Let $h$ be any prior on $c$. We have

$$h_{[X=x]}\left(c\right)\propto h\left(c\right)\prod _{i=1}^n{f}_{[C=c]}\left(x_i\right)=h\left(c\right)\prod _{i=1}^n\frac{1}{c}g\left(x_i\right)$$

This expression with the prior excluded is larger the smaller $c$ is. Thus, this experiment will update $h$ toward lower values regardless of the observed data points $x_1,...,x_n$.

... wait, what? What happened?

Here is what I think happened. The function $g$ fully determines the value of $c$. Since we know $g$, one cannot model $c$ as a random variable. In fact $Pr(C=c)$ must be 1 since $1=1⟹C=c$, hence $Pr(C=c)\ge Pr(1=1)=1$. If anything $C$ is a logically uncertain variable à la Logical Induction. Since probability theory cannot handle logical uncertainty, the method fails.

However, it is interesting that there is an estimator that does a fine job approximating $f$. This estimator is $f(x_1,...,x_n):=\frac{1}{n}{\sum }_{i=1}^n\frac{g\left(x_i\right)}{\hat{f}\left(x_i\right)}$, where the $\hat{f}\left(x_i\right)$ are estimates of the probability distribution of the points based on the observed data $(x_1,...,x_n)$. (The book doesn't explain how this is done.) Thus, the concept of 'estimator' appears to include methods that deal with logical uncertainty.

Alas, it appears that the second example highlights a legitimate failure of Bayesian inference.

9. Statistical Decision Theory

So far, we've evaluated predictors by computing their Mean Squared Error. One can also do this for Bayesian methods. In the discrete example (5.2.), the posterior distribution $h_{[X=x]}$ has mean $\frac{k}{n+1}$ (no computations are needed to derive this result since the mean of a Beta distribution is known), so $f:(x_1,...,x_n)\to \frac{{\sum }_{i=1}^n{x}_i}{n+1}$ is just another estimator that happens to have been derived with Bayesian methods.

However, recall that estimating the Mean Squared Error yields a term involving the true parameter $\theta$. This can be considered a function of $\theta$, and thus, one can imagine two estimators whose functions cross paths, such that both of them yield lower errors for some values of $\theta$. In such a case, it's not immediately obvious which one is preferable. Statistical Decision Theory is about making this decision.

Let $f$ be an estimator. As mentioned, its mean squared error is a function of $\theta$. If we want to compare it to the mean squared error of other estimators, we need a one-point summary of this function. Here are two ways of doing this:

• $r_{\text{MAX}}\left(f\right):={sup}_{\theta \in \Theta }E\left(\right(f-\theta {)}^2)$
• $r_{\text{Bayes}}(f,h):=\int E\left(\right(f-\theta {)}^2\left)h\right(\theta )\text{d}\theta$

I.e., we can either optimize for the worst case and minimize the maximum error across all possible values of $\theta$, or we can weigh the errors depending on which $\theta$ is more probable. Of course, this again requires a prior $h$ on $\theta$, hence why it's called the Bayes risk.

One can prove that:

• If $h$ is a proper prior, the estimator $f$ that minimizes $r_{\text{Bayes}}(f,h)$ is the function that maps data to the mean of the posterior distribution, i.e., the estimator $f$ given by $f(x,h):=\int \theta h_{[X=x]}\left(\theta \right)\text{d}\theta$.
• If $h$ is a proper prior and one replaces the Mean Squared Error with the absolute loss, i.e., $E(|f-\theta |)$ instead of $E\left(\right(f-\theta {)}^2)$, then it's the median of the posterior distribution. (This was quite surprising to me. I always thought that, if one has values $1,2,3,4,90$, then, surely, $20$ (the mean) is a better summary of this data than 3 (the median). However, if one defines 'better' in terms of 'lower mean absolute distance to one of the elements', then $3$ is better! This quickly becomes obvious if you think about it; once you're at the median element, any step to either side increases the distance to more than half of all points and decreases the distance to fewer than half, by the same amount.)
• If $h$ is a proper prior and one replaces the Mean Squared Error with the binary error (1 if $\theta$ is hit exactly and $0$ otherwise), then it's the mode of the posterior distribution -- that is, provided the distribution of $\theta$ is discrete; if it's continuous, the loss is 1 with probability 1 regardless of the estimator.

Thus, Bayesian methods are optimal to minimize the Bayes risk.

Suppose a Bayes estimator $f$ for prior $h$ yields an error that does not depend on $\theta$, i.e., an error function that is a constant. In this case, the one-point summary is just that constant, which means that $r_{\text{Bayes}}(f,h)=r_{\text{MAX}}\left(f\right)$. Thus, one can also view minimization of $r_{\text{MAX}}$ as a special case of applying Bayesian methods. In this case, $h$ is called the least favorable prior.

10. Other Concepts

10.1. Score and Fisher Information

Consider a parametric inference problem. Let $\theta$ be the parameter and $f$ the density function. This time, we consider $f$ to be a function of two variables $\theta$ and $x$.

The score is defined as the derivative of $f$ with respect to $\theta$, i.e.,

$$s:=\frac{\partial f}{\partial \theta }$$

The score is again a function of two variables. However, if we fix a $\theta \in \Theta$, then the function $s_{\theta }$ defined by $s_{\theta }\left(x\right):=s(\theta ,x)$ is a RV. For any point $x$, $s_{\theta }\left(x\right)$ measures how increasing $\theta$ affects the density that $f$ assigns to $x$.

Suppose that $E\left(s_{\theta }\right)>0$. That would mean that increasing $\theta$ increases the overall density of $f$. However, this is impossible: $f_{\theta +ϵ}$ is still a valid pdf, so the overall density cannot change. Thus, the expectation of the score is always zero. This can also be proven formally.

On the other hand, consider the variance of the score, $V\left(s_{\theta }\right)$. We can imagine several cases:

• If $f$ does not depend on $\theta$ at all, then the score is zero everywhere. In this case, its variance is also zero.
• If $f$ can take drastically different forms depending on the value of $\theta$, then the score should be high in some places and low in others. In this case, the variance is high.

Thus, the variance of the score is a measure for how much the parameter impacts the distribution. Equivalently (I believe, I don't know information theory), it's a measure of how much information [samples of the distribution] reveal about $\theta$. Note that evaluating this variance for a specific $\theta$ yields a number; if $\theta$ is left as a parameter, it yields a term that includes $\theta$. The variance of the score is also called the Fisher Information, and it's denoted as a function $I$ (of $\theta$). Wikipedia lists the fisher information for all important distributions; for example, if $X\sim \text{Bernoulli}\left(p\right)$, then the fisher information is $\frac{1}{p(1-p)}$.

One can prove that

$$I\left(\theta \right)=V\left(s_{\theta }\right)=E(-\frac{\partial s}{\partial \theta }\left(\theta \right))=E(\frac{{\partial }^{2}f}{{\partial }^2\theta }\left(\theta \right))$$

If $f$ is the maximum likelihood estimator of $\theta$, then the Fisher Information takes a role similar to the variance of $f$ and one can use this to prove theorems about the limit distribution of $f$. Furthermore, in the context of Bayesian Inference, there is something called Jeffrey's Prior, which says that the prior on a parameter should be proportional to the root of its Fisher Information. This does not seem to make sense to me, but perhaps I don't understand it.

10.2. Bootstrapping

Which is a fairly complicated method for estimating the variance of statistics.

10.3. The Jacknife

Same as above.

10.4. Simulation

Which is a Monte Carlo method to estimate things that could in theory be computed but may be hard to compute, such as the marginal distribution of $\theta$ when the distribution of a vector $(\theta ,{\theta }_2,...,{\theta }_n)$ is known.

10.5. Multiparameter Models for Parametric Inference

In which (who'd have guessed) parametric inference with multiple parameters is studied.

10.6. Various tests

As in 'specific ways to do hypothesis testing'. There are also Bayesian methods here.

10.7. Admissibility

Which is a property of estimators. An estimator is inadmissible if there is another estimator that performs at least as well for all $\theta$, and better for at least one $\theta$. It's admissible if it's not inadmissible. Bayes' estimators with proper priors are always admissible.

10.7.1. Stein's Paradox

Suppose we sample $x_1\sim N({\theta }_1,1),...,x_n\sim N({\theta }_n,1)$ and want to estimate the ${\theta }_i$. Stein's paradox says that the estimator $f:(x_1,...,x_n)\mapsto (x_1,...,x_n)$ is inadmissible if $n>2$ (and admissible for $n=2$ and $n=1$).

10.8. Plug-in Estimators

The empirical distribution function gives a general way to estimate the value of any statistical functional on an unknown distribution, provided we have access to sample points $(x_1,...,x_n)$. This way is, of course, to simply compute the statistical functional of the empirical distribution function derived from $(x_1,...,x_n)$.

In the case of the mean, this leads to the obvious estimate $\hat{\mu }:=\frac{1}{n}{\sum }_{i=1}^n$ that is used in the method of moments. For the variance, it leads to the estimate $\hat{\sigma }:=\frac{1}{n}{\sum }_{i=1}^n(x_i-\hat{\mu }{)}^2$. Interestingly, this estimate is biased. The unbiased estimate is called the sample variance and it's defined by $S^2:=\frac{1}{n-1}{\sum }_{i=1}^n(x_i-\hat{\mu }{)}^2$. The reason for this is that any term of the form $(x_i-\hat{\mu }{)}^2$ is, in expectation, slightly lower than it should be because $x_i$ has contributed to the estimate $\hat{\mu }$, thus causing $\hat{\mu }$ to be closer to $x_i$ in expectation than the true mean. For example, suppose we have just two data points $x_1$ and $x_2$. If they are both below the mean, the estimate $\hat{\mu }$ will be too small, so the terms $(x_1-\hat{\mu }{)}^2$ and $(x_2-\hat{\mu }{)}^2$ will be too small as well. If they are both above the mean, the estimate $\hat{\mu }$ will be too large, and the terms $(x_1-\hat{\mu }{)}^2$ and $(x_2-\hat{\mu })$ will again be too small. Thus, regardless of the direction into which the noise skews the data points, the plug-in estimate for the variance will underestimate the true variance, making it biased. If one does the computation, it turns out that the effect of this bias is, on expectation, precisely the factor $\frac{n}{n-1}$.

0 comments

Comments sorted by top scores.