LessWrong 2.0 Reader

That's an error. Thank you for pointing it out!

Mathematically, the Taylor expansion is:

$L\left(x_{test}\right)=L\left(x_{train}\right)+(x_{test}-x_{train})\cdot \nabla L\left(x_{train}\right)+\frac{1}{2}(x_{test}-x_{train}{)}^T{H}_L(x_{train}\left)\right(x_{test}-x_{train})+...$

And then we have $L\left(x_{train}\right)=0$ and also $\nabla L\left(x_{train}\right)=0$. (This does assume a "sufficiently nice" loss function, that is satisfied by most loss functions used in practice.)

I agree $\nabla L\left(x_{test}\right)$ is not zero. I also agree if you take some point in between $x_{train}$ and $x_{test}$ it can have non-zero loss, e.g. $L\left(\frac{x_{train}+x_{test}}{2}\right)$ need not be zero. I'm not sure if either of these are what you're trying to say, but in any case they aren't relevant to the quoted sentence.

If you are claiming $\nabla L\left(x_{train}\right)\ne 0$, then I disagree and am unclear on how your arguments are supposed to establish that.

And while flattering to Brockman, there is nothing about Murati - free tip to all my VC & DL startup acquaintances, there's a highly competent AI manager who's looking for exciting new opportunities, even if she doesn't realize it yet.

Heh, here it is: https://x.com/miramurati/status/1839025700009030027 

If a simple philosophical argument can cut the expected odds of AI doom by an order of magnitude, we might not change our current plans, but it suggests that we have a lot of confusion on the topic that further research might alleviate.

And more generally, "the world where we almost certainly get killed by ASI" and "The world where we have an 80% chance of getting killed by ASI" are different worlds, and, ignoring motives to lie for propaganda purposes, if we actually live in the latter we should not say we live in the former.

Being ignorant, I can't respond in detail. It makes sense that there'd be variation between ideologies, and that many people would have versions that are less, or differently, bad (according to me, on this dimension). But I would also guess that I'd find deep disagreements in more strands, if I knew more about them, that are related to motive dismantling.

For example, I'd expect many strands to incorporate something like the negation

• of "Reality bites back." or
• of "Reality is (or rather, includes quite a lot of) that which, when you stop believing in it, doesn't go away." or
• of " We live in the world beyond the reach of God.".

As another example, I would expect most Buddhists to say that you move toward unity with God (however you want to phrase that) by in some manner becoming less {involved with / reliant on / constituted by / enthralled by / ...} symbolic experience/reasoning, but I would fairly strongly negate this, and say that you can only constitute God via much more symbolic experience/reasoning.

Rather than doubling down on a single single-layered decomposition for all activations, why not go with a multi-layered decomposition (ie: some combination of SAE and metaSAE, preferably as unsupervised as possible).  Or alternatively, maybe the decomposition that is most useful in each case changes and what we really need is lots of different (somewhat) interpretable decompositions and an ability to quickly work out which is useful in context. 
 

Definitely seems like multiple ways to interpret this work, as also described in SAE feature geometry is outside the superposition hypothesis [LW · GW]. Either we need to find other methods and theory that somehow finds more atomic features, or we need to get a more complete picture of what the SAEs are learning at different levels of abstraction and composition.

Both seem important and interesting lines of work to me!

 

Ilya Sutskever and Mira Murati will leave OA or otherwise take on some sort of clearly diminished role by year-end (90%, 75%; cf. Murati's desperate-sounding internal note)

Mira Murati announced today she is resigning from OA. (I have also, incidentally, won a $1k bet with an AI researcher on this prediction.)

That all makes sense, thanks. I'm really looking forward to seeing where this line of research goes from here!

This distinction might be important in some particular cases. If it looks like an AGI might ascend to power with no real chance of being stopped by humanity, its decision about humanity might be swayed by just such abstract factors.

That consideration of being in a test might be the difference between our extinction, and our survival and flourishing by current standards.

This would also apply to the analagous consideration that alien ASIs might consider any new ASI that extincted its creators to be untrustworthy and therefore kill-on-sight.

None of this has anything to do with "niceness", just selfish logic, so I don't think it's a response to the main topic of that post.

My big question about this post is when you claim that a massively superhuman AI could outwit control measures if it wanted to, how much more intelligent are you usually imagining when you think of the best control measures that you think could be applied to the AI?

I'd ideally like the amount of compute that we could control, and I'd want a more quantitative upper-bound than you gave here.