Comment by emiddell on Mapping the semantic void: Strange goings-on in GPT embedding spaces · 2023-12-15T10:06:59.686Z · LW · GW

In higher dimensions most of the volume of a n-sphere is found close to its shell.

The volume of such a sphere is $V_n\left(R\right)\propto R^n$.  [1] 

The ratio of the volume of a shell to the rest of the ball is 

$\frac{V_n(R+\Delta )-V_n\left(R\right)}{V_n\left(R\right)}=(1+\frac{\Delta }{R}{)}^n-1$ 

which grows quickly with n. 

The embedding places the tokens in areas where there is the most space to accommodate them.

[1] https://en.wikipedia.org/wiki/Volume_of_an_n-ball