An interactive introduction to grokking and mechanistic interpretability 2023-08-07T19:09:19.422Z

Comment by Adam Pearce (adam-pearce) on An interactive introduction to grokking and mechanistic interpretability · 2023-08-16T02:50:00.049Z · LW · GW

Lots of custom d3 https://github.com/PAIR-code/ai-explorables/tree/master/source/grokking

Comment by Adam Pearce (adam-pearce) on Growing Bonsai Networks with RNNs · 2023-08-07T21:28:03.520Z · LW · GW

• The optimization section of Learning Transformer Programs might work with your task/model
• You've probably seen David Ha's work, but something like https://es-clip.github.io/ could be a good starting point for dropping backprop.
• The exotic activation function almost feels like cheating? Like I want the model the model to discover these useful structures, then try to understand them. But trying to do everything at once may be too hard. 
• Incredibility minor, but changing from onchange to oninput and dropping the animation will make the slider feel much slicker.