Comment by Cheng-Han Chiang (d223302) on Domain-specific SAEs · 2025-01-05T06:47:24.809Z · LW · GW

Thanks for the fantastic post. I have a question on the Loss Recovered comparison. In the text, you calculate the ratio $\frac{L-L_{clean}}{L_{GSAE}-L_{clean}}$. Based on the interpretation, it seems that a higher ratio is better. However, it seems to me that if the domain-specific SAE is good enough and perfectly reconstructed, we should have  $L=L_{clean}$. In this case, the ratio from the above definition is zero, while this seems to be an optimal case. Can you point out where my interpretation may be wrong?