You've dropped a factor of $\alpha$ about half-way through your calculation. And then you've multiplied by $X^{-1}$ between two lines separated by "="; the idea is that both sides are zero so it kinda-sorta makes sense but it's super-misleading. If you restore the factor of $\alpha$ then your last equation ends up as $h=Y(X+\alpha X^{-1}{)}^{-1}$.

But even this is wrong, I'm afraid. You can't multiply by $X^{-1}$ there at all. There is no $X^{-1}$; $X$ is not (except by coincidence, and in an ML application if this coincidence happens then you don't have anything like enough data) a square matrix and in general it has no inverse.

There are problems earlier in the derivation, too, which I think are encouraged by some of your nonstandard notation. E.g., you write $h{x}_t$ rather than $h\cdot x_t$ or $x_t^{T}h$, and this has fooled you into writing down something wrong for what you write as $\frac{d}{dh}E$. That's also nonstandard notation; it's defensible but again it makes it easy to get things wrong by mixing up left and right multiplications. Let's do it with more standard and explicit notation, which will make it harder to make mistakes:

$0=\frac{\partial }{\partial h_j}(E+C)=\frac{\partial }{\partial h_j}\left[\right(Xh-Y{)}^T(Xh-Y)+\alpha h^{T}h]=\frac{\partial }{\partial h_j}[h^T{X}^{T}Xh-h^T{X}^{T}Y-Y^{T}Xh+Y^{T}Y+\alpha h^{T}h]$

The $Y^{T}Y$ is constant and its derivative is zero. The terms linear in $h$ are one another's transposes and readily yield $2(X^{T}Y{)}_j$. The second quadratic term is just $\alpha \sum h_k^2$ whose $\frac{\partial }{\partial h_j}$ is $2h_j$. The first quadratic term is similarly $\sum (Xh{)}_k^2$ which equals ${\sum }_k{\left({\sum }_l{X}_{kl}{h}_l\right)}^2$ whose $\frac{\partial }{\partial h_j}$ is ${\sum }_{k}2\left({\sum }_l{X}_{kl}{h}_l\right)\frac{\partial }{\partial h_j}\left({\sum }_l{X}_{kl}{h}_l\right)={\sum }_{k}2\left({\sum }_l{X}_{kl}{h}_l\right)X_{kj}=(2X^{T}Xh{)}_j$.

So what ends up being zero is the $j$th component of $2X^T(Xh-Y)+2\alpha h$ and if you like you can write $\frac{d}{dh}(E+C)=2\left[X^T\right(Xh-Y)+\alpha h]$. But again you need to be very clear about what you mean by that; $\frac{d}{dh}f\left(h\right)$ means "the $A$ such that to first order $f\left(h\right)=\text{const}+Ah$" and so actually the Right Thing to use for the "derivative" is the transpose of what I wrote down above.

Finishing off the correct derivation, we have

$X^T(Xh-Y)+\alpha h=0$ so $(X^{T}X+\alpha )h=Y$ so $h=(X^{T}X+\alpha {)}^{-1}Y$.

I think it genuinely doesn't make sense to say that $\alpha$ reflects our prior expectation of $\alpha$; the acolyte is correct. What $\alpha$ reflects is our prior on $h$; that regularization term corresponds exactly to a prior that makes $h$ (multivariate) normally distributed with mean zero and covariance $1/\alpha$ times the identity (i.e., components independent and each component having variance $1/\alpha$).