Comment by Sumeet S Singh (sumeet-s-singh) on An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 · 2024-12-23T17:33:05.185Z · LW · GW

Hi Neel, What do you think of this paper? I felt it was pretty novel and revealing when I wrote it, but never got the chance to publish it in a conference. I still think so. The paper essentially says that the embedding space after taining works like like a static landscape that is formed based on the training data. The model simply follows a predetermined path based on where it started and how much momentum it had been given initially (by the prompt), much like a ball rolling down a static landscape would follow a predetermined path after it had been set into motion based on where and with what momentum it started. Hoping you can review it and let me know if I'm correct or  whether I'm overestimating it's importance. 

https://arxiv.org/abs/2308.10874