You might enjoy this paper about a related idea.

If I understand correctly, this residual decomposition is equivalent to the edge / factorized view of a transformer described here [LW · GW].

Update: actually the residual decomposition is incorrect - see my other comment [LW(p) · GW(p)].

@Oliver Daniels-Koch [LW · GW]'s reply [LW(p) · GW(p)] to my comment made me read this post again more carefully and now I think that that your formulation of the residual expansion is incorrect.

Given $T=(MLP+Id)\circ (Att+Id)$  it does not follow that $T=(MLP\circ Att)+\left(MLP\right)+\left(Att\right)+Id$  because $MLP$ is a non-linear operation. It cannot be decomposed like this.

My understanding of your big summation (with $C$ representing any MLP or attention head):

again does not hold because the $C$s are non-linear.

There are two similar ideas which do hold, namely (1) the treeified / unraveled view and (2) the factorized view (both of which are illustrated in figure 1 here), but your residual expansion / big summation is not equivalent to either.

1. The treeified / unraveled view is the most similar. It separates each path from input to output, but the difference is that this does not claim that the output is the sum of all separate paths.

   

2. The factorized view follows from treeified view and is just the observation that any point in the residual stream can be decomposed into the outputs of all previous components.

   $T\left(x\right)=x+\sum _{i=0}^{n-1}\text{Output}\left(C_i\right)$

I think this makes sense.

I am not sure how new this approach is (for simplified Transformers, the original AMFOTC paper has several sections called "* Path Expansion *", which seem to do something very similar for a reduced set of transformations, and their formalism of "virtual attention heads" seems also to be in that spirit).