LessWrong 2.0 Reader

Thanks for writing this. I think this topic is generally a blind spot for LessWrong users, and it's kind of embarrassing how little thought this community (myself included) has given to the question of whether a typical future with human control over AI is good.

(This actually slightly broadens the question, compared to yours. Because you talk about "a human" taking over the world with AGI, and make guesses about the personality of such a human after conditioning on them deciding to do that. But I'm not even confident that AGI-enabled control of the world by e.g. the US government would be good.)

Concretely, I think that a common perspective people take is: "What would it take for the future to go really really well, by my lights", and the answer to that question probably involves human control of AGI. But that's not really the action-relevant question. The action-relevant question, for deciding whether you want to try to solve alignment, is how the average world with human-controlled AGI compares to the average AGI-controlled world. And... I don't know, in part for the reasons you suggest.

Yeah, on further thought I think you're right. This is pretty pessimistic then, AI companies will find it easy to align AIs to money interests, and the rest of us will be in a "natives vs the East India Company" situation. More time to spend on alignment then matters only if some companies actually try to align AIs to something good instead, and I'm not sure any companies will do that.

How do you know that humans and LLMs/current RL agents do optimize the reward? Are there any known theorems or papers on this, because this claim is at least a little bit important.

You may answer here:

https://www.lesswrong.com/posts/GDnRrSTvFkcpShm78/when-is-reward-ever-the-optimization-target [LW · GW]

This is consistent with a model where AI alignment is heavily dependent on the data, and way less dependent on inductive biases/priors, so this is good news for alignment.

In one specific respect I'd like to challenge your point. I think fine-tuning models currently aligns them 'well-enough' to any target point of view. I think that the ethics shown by current LLMs are due to researchers actively putting them there. I've been doing red teaming exercises on LLMs for over a year now, and I find it quite easy to fine-tune them to be evil and murderous. Human texts help them understand morality, but don't make them care enough about it for it to be sticky in the face of fine-tuning.

It is an interesting paper (excerpts, related excerpts), but to temper expectations: like Peter Leeson's many papers 'explaining' this or that weird thing in history like trial-by-ordeal, this is more of a just-so story in which Allen comes up with a plausible-sounding model which he post hoc fits to some interesting historical details or anecdotes, and he doesn't present anything we would consider 'hard' evidence.

Stargate is not 2025, and going from $50bn to $80bn is in line with building a $25-40bn training system this year (an unusual expense distinct from other projects), a clue separate from SemiAnalysis claims.

It seems like the growth models already take much of that into account, the same way that they do crime or war: if new technologies create new crime (which of course they often do), then that simply slightly offsets the benefits of those technologies, and it is the net benefit which shows up in the long-term growth rather than some 'pure' benefit free of any drawbacks. And likewise for technologies as a whole: if you're inventing some unknown grabbag of technologies each time-period, then it's the net of all the good ideas being offset slightly by the bad ones that is getting measured or driving the growth in the next time-period etc. It would be like measuring the growth of an actual tumor: whatever growth you observe, well, that must be the net growth after the defectors inside the tumor have done their worst, by definition.

So you'd have to invoke some sort of non-constant or non-proportionality: "yes, the bad ideas are only an offset, up until some threshold value like 'inventing nuclear bombs'" (like Bostrom's 'black balls'). But then your results seems dangerously circular: if you assume some fat tail payoff from the bad ideas after a certain threshold or increasingly with time, you are building in your conclusions like "we should halt all technological progress forever".

Nice post! 

My notes / thoughts: (apologies for overly harsh/critical tone, I'm stepping into the role of annoying reviewer)

Summary

• Use residual stream SAEs spaced across layers, discovery node circuits with learned binary masking on templated data.
• binary masking outperforms integrated gradients on faithfulness metrics, and achieves comparably (though maybe narrowly worse) completeness metrics.
• demonstrated approach on code output prediction
   

Strengths: 

• to the best of my knowledge, first work to demonstrate learned binary masks for circuit discovery with SAEs
• to the best of my knowledge, first work to compute completeness metrics for binary mask circuit discovery

Weaknesses 

• no comparison of spaced residual stream SAEs to more finegrained SAEs
• theoretic arguments / justifications for coarse grained SAEs are weak. In particular, the claim that residual layers contain all the information needed for future layers seems kind of trivial
• no application to downstream tasks
• Finding Transformer Circuits with Edge Pruning introduces learned binary masks for circuit discovery, and is not cited. This also undermines the "core innovation claim" - the core innovation is applying learned binary masking to SAE circuits

More informally, this kind of work doesn't seem to be pushing on any of the core questions / open challenges in mech-interp, and instead remixes existing tools and applies them to toy-ish/narrow tasks.

Of the future research / ideas, I'm most excited about the non-templatic data / routing model 

 

I don't think you understand, in the example I gave they don't think they are 'average' they think their performance was not to the standard they hold themselves, and they believe that this was precipitated by their drinking which they regret. He is talking PAST the person after the show, not to them, almost like a soliloquy.

Do you think that every time you've ever felt shame it has always been primarily because of what others may think of you? You have never ever felt a solipsistic shame, a shame even though no one will know, no one will care, it has no negative influence on anyone other than yourself, and the only person you have to answer to is you? Never?