Redundant Attention Heads in Large Language Models For In Context Learning 2024-09-01T20:08:48.963Z

Comment by skunnavakkam on Explore More: A Bag of Tricks to Keep Your Life on the Rails · 2024-12-04T22:35:07.368Z · LW · GW

I've found the part about applying random search to be the among the best takeaways I had from PAIR! Novelty for the sake of Novelty is not a terrible idea. Specifically, I've found that even if you don't like the things you do, it makes it much easier to then make progress towards the larger goal