LessWrong 2.0 Reader

This is consistent with a model where AI alignment is heavily dependent on the data, and way less dependent on inductive biases/priors, so this is good news for alignment.

In one specific respect I'd like to challenge your point. I think fine-tuning models currently aligns them 'well-enough' to any target point of view. I think that the ethics shown by current LLMs are due to researchers actively putting them there. I've been doing red teaming exercises on LLMs for over a year now, and I find it quite easy to fine-tune them to be evil and murderous. Human texts help them understand morality, but don't make them care enough about it for it to be sticky in the face of fine-tuning.

It is an interesting paper (excerpts, related excerpts), but to temper expectations: like Peter Leeson's many papers 'explaining' this or that weird thing in history like trial-by-ordeal, this is more of a just-so story in which Allen comes up with a plausible-sounding model which he post hoc fits to some interesting historical details or anecdotes, and he doesn't present anything we would consider 'hard' evidence.

Stargate is not 2025, and going from $50bn to $80bn is in line with building a $25-40bn training system this year (an unusual expense distinct from other projects), a clue separate from SemiAnalysis claims.

It seems like the growth models already take much of that into account, the same way that they do crime or war: if new technologies create new crime (which of course they often do), then that simply slightly offsets the benefits of those technologies, and it is the net benefit which shows up in the long-term growth rather than some 'pure' benefit free of any drawbacks. And likewise for technologies as a whole: if you're inventing some unknown grabbag of technologies each time-period, then it's the net of all the good ideas being offset slightly by the bad ones that is getting measured or driving the growth in the next time-period etc. It would be like measuring the growth of an actual tumor: whatever growth you observe, well, that must be the net growth after the defectors inside the tumor have done their worst, by definition.

So you'd have to invoke some sort of non-constant or non-proportionality: "yes, the bad ideas are only an offset, up until some threshold value like 'inventing nuclear bombs'" (like Bostrom's 'black balls'). But then your results seems dangerously circular: if you assume some fat tail payoff from the bad ideas after a certain threshold or increasingly with time, you are building in your conclusions like "we should halt all technological progress forever".

Nice post! 

My notes / thoughts: (apologies for overly harsh/critical tone, I'm stepping into the role of annoying reviewer)

Summary

• Use residual stream SAEs spaced across layers, discovery node circuits with learned binary masking on templated data.
• binary masking outperforms integrated gradients on faithfulness metrics, and achieves comparably (though maybe narrowly worse) completeness metrics.
• demonstrated approach on code output prediction
   

Strengths: 

• to the best of my knowledge, first work to demonstrate learned binary masks for circuit discovery with SAEs
• to the best of my knowledge, first work to compute completeness metrics for binary mask circuit discovery

Weaknesses 

• no comparison of spaced residual stream SAEs to more finegrained SAEs
• theoretic arguments / justifications for coarse grained SAEs are weak. In particular, the claim that residual layers contain all the information needed for future layers seems kind of trivial
• no application to downstream tasks
• Finding Transformer Circuits with Edge Pruning introduces learned binary masks for circuit discovery, and is not cited. This also undermines the "core innovation claim" - the core innovation is applying learned binary masking to SAE circuits

More informally, this kind of work doesn't seem to be pushing on any of the core questions / open challenges in mech-interp, and instead remixes existing tools and applies them to toy-ish/narrow tasks.

Of the future research / ideas, I'm most excited about the non-templatic data / routing model 

 

I don't think you understand, in the example I gave they don't think they are 'average' they think their performance was not to the standard they hold themselves, and they believe that this was precipitated by their drinking which they regret. He is talking PAST the person after the show, not to them, almost like a soliloquy.

Do you think that every time you've ever felt shame it has always been primarily because of what others may think of you? You have never ever felt a solipsistic shame, a shame even though no one will know, no one will care, it has no negative influence on anyone other than yourself, and the only person you have to answer to is you? Never?

Yeah, the crux is I don't think the story ends before superintelligence

what i meant by "the story ends before what i meant by superintelligence has occurred" is that the written one ends there in 2035, but at that point there's still time to effect what the first long-term-decisive thing will be.

If you read this and, like I did, felt unfulfilled after reading it, it's worth noting that this paper (which was linked in the OP, but which I [and perhaps the unfulfilled reader] overlooked) goes into more detail:

https://projects.panickssery.com/docs/allen-2009-a_theory_of_the_pre-modern_british_aristocracy.pdf

Yes, I'm hoping that the better governance system is something that can be accomplished prior to superintelligence. I do agree that the short time frame for implementation seems like the biggest obstacle to success.