Sparse Features Through Time

post by Rogan Inglis (rogan) · 2024-06-24T18:06:16.472Z · LW · GW · 1 comments

This is a link post for https://roganinglis.io/posts/Sparse%20Features%20Through%20Time

Contents

1 comment

This project explores the use of Sparse Autoencoders (SAEs) to track the development of features in large language models throughout their training. It investigates whether features can be reliably matched between different SAEs trained on various checkpoints of Pythia 70M and characterises the development of these features over the course of training. The findings show that features can successfully be matched between different SAEs. The results also support the distributional simplicity bias hypothesis, indicating that simpler features are learned early in training, with more complex features emerging later. While the focus was on a relatively small model, the results lay the groundwork for future research into larger models and the identification of potentially deceptive capabilities. This work aims to enhance the interpretability and safety of AI systems by providing a deeper understanding of feature development and the dynamics of neural network training, with the ultimate goal of making a general statement about the overall likelihood of dangerous deceptive alignment arising in practice.

 

1 comments

Comments sorted by top scores.