Transformer Debugger

post by Henk Tillman (henk-tillman) · 2024-03-12T19:08:56.280Z · LW · GW · 0 comments

This is a link post for https://github.com/openai/transformer-debugger

Contents

No comments

Transformer Debugger (TDB) is a tool developed by OpenAI's Superalignment team with the goal of supporting investigations into circuits underlying specific behaviors of small language models. The tool combines automated interpretability techniques with sparse autoencoders.

TDB enables rapid exploration before needing to write code, with the ability to intervene in the forward pass and see how it affects a particular behavior. It can be used to answer questions like, "Why does the model output token A instead of token B for this prompt?" or "Why does attention head H to attend to token T for this prompt?" It does so by identifying specific components (neurons, attention heads, autoencoder latents) that contribute to the behavior, showing automatically generated explanations of what causes those components to activate most strongly, and tracing connections between components to help discover circuits.

These videos give an overview of TDB and show how it can be used to investigate indirect object identification in GPT-2 small:

• Introduction
• Neuron viewer pages
• Example: Investigating name mover heads, part 1
• Example: Investigating name mover heads, part 2

 

Contributors: Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders.
Thanks to Johnny Lin for contributing to the explanation simulator design.

0 comments

Comments sorted by top scores.