LessWrong 2.0 Reader

I agree.

Ah, I didn't read the paper, only the LW post.

As I emphasized in a different comment, this work is not about mechanistic anomalies or how the model spells, it's about measurement error in the SAE method.

I understand that, I more meant my suggestion as an idea for if you want to go beyond poking holes in SAE to instead solve interpretability.

We asked whether Gemma-2-2b could spell / do the first letter identification task. We then asked which latents causally mediated spelling performance, comparing SAE latents to probes.

One downside to this is that spelling is a fairly simple task for LLMs.

We found that we couldn't find a set of 26 SAE latents that causally mediated spelling because the relationship between the latents and the character information, "exogenous factors", if I understand your meaning, wasn't as clear as it should have been.

I expect that:

• Objects in real-world tasks will be spread over many tokens, so they will not be identifiable within individual tokens.
• Objects in real-world tasks will be massively heterogenous, so they will not be identifiable with a small number of dimensions.

Implications:

• SAE latents will not be relevant at all, because they are limited to individual tokens.
• The value of interpretability will less be about finding a small fixed set of mediators and more about developing a taxonomy of root causes and tools that can be used to identify those root causes.
  • SAEs would be an example of such a tool, except I don't expect they will end up working.

A half-baked thought on a practical use-case would be, LLMs are often used for making chatbot assistants. If one had a taxonomy for different kinds of users of chatbots, and how they influence the chatbots, one could maybe create a tool for debugging cases where the language model does something weird, by looking at chat logs and extracting the LLM's model for what kind of user it is dealing with.

But I guess part of the long-term goal of mechanistic interpretability is people are worried about x-risk from learned optimization, and they want to identify fragments of that ahead of time so they can ring the fire alarm. I guess upon reflection I'm especially bearish about this strategy because I think x-risk will occur at a higher level than individual LLMs and that whatever happens when we're diminished all the way down to a forward propagation is going to look indistinguishable for safe and unsafe AIs.

That's just my opinion though.

That sounds like a terrible strategy. Your threat won't be credible because your goal is to make the world better, not destroy it. And anything you do to make the threat credible (like some sort of precomitment mechanism) will risk the world actually getting destroyed.

It doesn't seem to me that, unless we're putting humanity's self-interest in conflict with some other hypothetical alien civilization, that the rest of existence has an opinion on humanity's continued existence. The universe, insofar as we know, is pretty indifferent to the state it's in, and the only frame of reference we have in terms of values to decide whether or not humanity should continue is... well, humanity's own myriad value systems, which, unless fully misanthropic, generally have a group of humans, or humanity overall's continued prosperity as a good as a fundamental of the value system. 

You only set aside occasional low-value fragments for national parks, mostly for your own pleasure and convenience, when it didn't cost too much?

Earth as a proportion of the solar system's planetary mass is probably comparable to national parks as a proportion of the Earth's land, if not lower.

Maybe I've misunderstood your point, but if it's that humanity's willingness to preserve a fraction of Earth for national parks is a reason for hopefulness that ASI may be willing to preserve an even smaller fraction of the solar system (namely, Earth) for humanity, I think this is addressed here:

it seems like for Our research purposes simulations would be just as good. In fact, far better, because We can optimize the hell out of them, running it on the equivalent of a few square kilometers of solar diameter

"research purposes" involving simulations can be a stand-in for any preference-oriented activity. Unless ASI would have a preference for letting us, in particular, do what we want with some fraction of available resources, no fraction of available resources would be better left in our hands than put to good use.

Yes we agree. 👍🌻

I think I mention this in the essay too. 

If we are just changing at the drop of a hat, not for truth, but for convenience or any old reason, like most people are, ... 

or even under very strenuous dire circumstances, like we're about to die or in excruciating pain or something...

then that is a compromised mind. You're working with a compromised, undisciplined mind that will change its answers as soon as externals change.

Views change. Even our "robust" principles can go out the window under extreme circumstances. (So what's going on with those people who stay principled under extreme circumstances? This is worth looking into.) 

Of course views are usually what we have, so we should use them to the best extent we are able. Make good views. Make better views. Use truth-tracking views. Use ethical views. Great. 

AND there is this spiritual path out of views altogether, and this is even more reliable than solely relying on views or principles or commandments. 

I will try out a metaphor. Perhaps you've read The Inner Game of Tennis.

In tennis, at first, you need to practice the right move deliberately over and over again, and it feels awkward. It goes against your default movements. This is "using more ethical views" over default habits. Using principles. 

But somehow, as you fall into the moves, you realize: This is actually more natural than what I was doing before. The body naturally wants to move this way. I was crooked. I was bent. I was tense and tight. I was weak or clumsy. Now that the body is healthier, more aligned, stronger... these movements are obviously more natural, correct, and right. And that was true all along. I was lacking the right training and conditioning. I was lacking good posture. 

It wasn't just that I acquired different habits and got used to them. The new patterns are less effortful to maintain, better for for me, and they're somehow clearly more correct. I had to unlearn, and now I somehow "know" less. I'm holding "less" patterning in favor of what doesn't need anything "held onto." 

This is also true for the mind. 

We first have to learn the good habits, and they go against our default patterning and conditioning. We use rules, norms, principles. We train ourselves to do the right thing more often and avoid the wrong thing. This is important.

Through training the mind, we realize the mind naturally wishes to do good, be caring, be courageous, be steadfast, be reliable. The body is not naturally inclined to sit around eating potato chips. We can find this out just by actually feeling what it does to the body. And so neither is the mind naturally inclined to think hateful thoughts, lie to itself and others, or be fed mentally addictive substances (e.g. certain kinds of information). 

To be clear:

"More natural" does not mean more in line with our biology or evo-psych. It does not mean lazier or more complacent. It does not mean less energetic, and in some way it doesn't even mean less "effort". But it does mean less holding on, less tension, less agitation, less drowsiness, less stuckness, less hinderance.

"More natural" is even more natural than biology. And that's the thing that's probably going to trip up materialists. Because there's a big assumption that biology is more or less what's at the bottom of this human-stack. 

Well it isn't. 

There isn't a "the bottom."

It's like a banana tree. 

When you peel everything away, what is actually left? 

Well it turns out if you were able to PEEL SOMETHING AWAY, it wasn't the Truth of You. So discard it. And you keep going.

And that's the path. 

This is a difficult topic (in more ways than one). I'll try to do a better job of addressing it in a future post.

I don't see why either expecting or not-expecting to meet other ASIs would make it instrumental to be nice to humans.

I think that's exactly what we did? Though to be fair we de-emphasized this version of the narrative in the paper: We asked whether Gemma-2-2b could spell / do the first letter identification task. We then asked which latents causally mediated spelling performance, comparing SAE latents to probes. We found that we couldn't find a set of 26 SAE latents that causally mediated spelling because the relationship between the latents and the character information, "exogenous factors", if I understand your meaning, wasn't as clear as it should have been. As I emphasized in a different comment, this work is not about mechanistic anomalies or how the model spells, it's about measurement error in the SAE method.

I wouldn't assume that Hitchen's writings are a complete record of his views. I remember him being a regular (and fiery!) TV guest during this period, often arguing in defense of military intervention on the basis that radical Islam is an evil worth fighting against. It's possible that he argued in favor of waterboarding in one of these many appearances.