[Linkpost] Remarks on the Convergence in Distribution of Random Neural Networks to Gaussian Processes in the Infinite Width Limit

post by carboniferous_umbraculum (Spencer Becker-Kahn) · 2023-11-30T14:01:32.255Z · LW · GW · 0 comments

This is a link post for https://drive.google.com/file/d/1O_RbD45RjBWD4-a8tcjosvyXOxcD_q2V/view?usp=sharing

Contents

No comments

The linked note is something I "noticed"  while going through different versions of this result in the literature. I think that this sort of mathematical work on neural networks is worthwhile and worth doing to a high standard but I have no reason to think that this particular work is of much consequence beyond filling in a gap in the literature. It's the kind of nonsense that someone who has done too much measure theory would think about.

Abstract. We describe a direct proof of yet another version of the result that a sequence of fully-connected neural networks converges to a Gaussian process in the infinite-width limit.  The convergence in distribution that we establish is the weak convergence of probability measures on the non-separable,  non-metrizable product space $(R^{d^{\prime }}{)}^{R^d}$, i.e. the space of functions from $R^d$ to $R^{d^{\prime }}$ with the topology whose convergent sequences correspond to pointwise convergence.  The result itself is already implied by a stronger such theorem due to Boris Hanin, but the direct proof of our weaker result can afford to replace the more technical parts of Hanin's proof that are needed to establish tightness with a shorter and more abstract measure-theoretic argument.  

0 comments

Comments sorted by top scores.