Comment by skosch on Re-Examining LayerNorm · 2022-12-02T01:19:12.693Z · LW · GW

That was my first thought as well. As far as I know, the most popular simple model used for this in the neuro literature, divisive normalization, uses similar but not quite identical formula. Different authors use different variations, but it's something shaped like

$$z_i=\frac{y_i^{\alpha }}{{\beta }^{\alpha }+{\sum }_j{\kappa }_{ij}{y}_j^{\alpha }}$$

where $y_i$ is the unit's activation before lateral inhibition, $\beta$ adds a shift/bias, ${\kappa }_{ij}$ are the respective inhibition coefficients, and the exponent $\alpha$ modulates the sharpness of the sigmoid (2 is a typical value). Here's an interactive desmos plot with just a single self-inhibiting unit. This function is asymmetric in the way you describe, if I understand you correctly, but to my knowledge it's never gained any popularity outside of its niche. The ML community seems to much prefer Softmax, LayerNorm et al. and I'm curious if anyone knows if there's a deep technical reason for these different choices.