Complexity Penalties in Statistical Learning 2019-02-06T04:13:57.140Z

Comment by michael_h on Complexity Penalties in Statistical Learning · 2019-02-06T04:07:22.038Z · LW · GW

Good point. Thank you for bringing this up. I just had a closer look in my notes at how the complexity penalty is derived and there is a additional assumption that I left out.

The derivation uses a matrix $X$ with $p+1$ columns and $n$ rows which has entry $x_i^{j-1}$ in the $i^{th}$ row and $j^{th}$ column (where $x_1,x_2,\dots ,x_n$ is the training set). In the derivation it is assumed that $X$ has rank $p+1$ which true most of the time provided that $n\ge p+1$. For simplicity I won't add a mention of this matrix to original post but I will add the assumption $n\ge p+1$.