LessWrong 2.0 Reader

Why you should try degrading NN behavior in experiments.

I got some feedback on the post I wrote yesterday [LW · GW] that seems right. The post is trying to do too many things, and not properly explaining what it is doing, why this is reasonable, and how the different parts are related. I think the main piece of advice in this post is potentially important. This is: experimentalists should in many cases run an experiment on multiple neural nets with a variable complexity dial that allows some "natural" degradations of the NN's performance, and certain dials are better than others depending on context. I am eventually planning splitting out the post into a few parts (and only discussion of the "koan" itself in the post: i.e., nitpicks about work that isn't careful about thinking about the scale at which it is performing interpretability.) For now I want to give a quick reductive take on why I think "interpretability on degraded networks" is important. Basically: when ML experiments modify a neural net to identify or induce a particular behavior, this always degrades performance. Now there are two hypotheses for what is going on:

1. You are messily pulling your NN in the direction of a particular behavior
2. You are messily pulling your NN in the direction of a particular behavior, but also singling out a few "real" internal circuits of the NN that are carrying out this behavior.

Because of how many parameters you have to play with and the polysemanticity of everything in a NN, it's genuinely hard to tell these two behaviors apart. You might find stuff that "looks" like a core circuit, but actually is just bits of other circuits that your circuit-fitting experiment makes look like a coherent behavior.

Now the idea behind running this experiment at "natural" degradations of network performance is to try to separate out these two possibilities more cleanly. Namely, an ideal outcome is that in running your experiment on some class of natural degradation of your neural net, you find a regime such that

• the intervention you are running no longer significantly affects the (naturally degraded) performance
• the observed effect still takes place. Then what you've done is effectively "cleaned up" your experiment such that you are still probably finding interpretable behaviors in the original neural net (since a good degradation is likely to contain a subset of circuits/behaviors of your original net and not many "new behaviors), in a way that sufficiently reduced the complexity that the behavior you're seeking is no longer "entangled" with a bunch of other behaviors; this should significantly update you that the behavior is indeed "natural" and not spurious.

This is of course a very small, idealized sketch: but the basic idea behind looking at neural nets with degraded performance is to "squeeze" the complexity in a controlled way to suitably match the complexity of the circuit (and how it's embedded in the rest of the network/how it interacts with other circuits). If you then have a circuit of "the correct complexity" that explains a behavior, there is in some sense no "complexity room" for other sneaky phenomena to confound it.

In the post, the natural degradation I suggested is the physics-inspired "SLGD sampling" process which in some sense tries to add a maximal amount of noise to your NN while only having a limited impact on performance (measured by loss); this has a bias of keeping "generally useful" circuits and interactions and noising more inessential/ memorize-y circuits. Other interventions that have different properties are "just adding random noise" (either to weights or activations) to suitable reduce performance, or looking at earlier training checkpoints. I suspect that different degradations (or combinations thereof) are appropriate to isolate the relevant complexity of different experiments.

Analogously: "I am claiming that people when informed will want horses to continue being the primary mode of transportation. I also think that most people when informed will not really care that much about economic growth, will continue to believe that you're more responsible for changing things than for maintaining the status quo, etc. And that this is a coherent view that will add up to a large set of people wanting things in cities to remain conservatively the same. I separately claim that if this is true, then other people should just respect this preference, and go find new continents / planets on which to build cars that people in the cities don't care about."

Sometimes it's good to be conservative when you're changing things, like if you're changing lots of social norms or social institutions, but I don't get it at all in this case. The sun is not a complicated social institution, it's primarily a source of heat and light and much of what we need can be easily replicated especially when you have nanobots. I am much more likely to grant that we should be slow to change things like democracy and the legal system than I am that we should change exactly how and where we should get heat and light. Would you have wanted conservatism around moving from candles to lightbulbs? Installing heaters and cookers in the house instead of fire pits? I don't think so.

Sure, but one can assess it at any point. I'm not asking about whether he will end up being net-positive or net-negative overall in the long run.

I'd agree. But he certainly does not seem to even be trying anymore to have positive impact on solving alignment, no?

Hi, thanks.

I don't see how what you say contradicts that the reach of his actions and opinions have increased. Did you maybe quote the wrong sentence?

Thanks so much for engaging, Nathan!

The pivotal act was defined by Yudkowsky, I'm just borrowing the definition. The idea is that even after you've built a perfectly aligned superintelligent AI, you only have about 6 months before someone else builds an unaligned superintelligent AI. That's probably not enough time to convince the entire world to adopt a better governance system before getting atomized by nanobots. So your aligned AI would have to take over the world and forcefully implement this better governance system within a span of a few months.

Claude Sonnet 3.6 is worthy of sainthood!

But as I mention in my other comment I'm concerned that such an AI's internal mental state would tend to become cynical or discordant as intelligence increases.

I'm going to give an argument in favor of computationalism as a theory of identity, but first I'm going to set up some of the stuff that I take as background, but others may not, and will hopefully illuminate related issues.

I believe computationalism is a very general way to look at effectively everything, because a model of computation can be extremely expressive, and a big reason for why philosophical debates go nowhere is that they don't realize that expressive enough models of computation can essentially trivialize any problem they set up, solely through the expressiveness of the model, so it's very important to realize that philosophical problems end up trivial if we allow ourselves to be unconstrained.

More here:

http://www.amirrorclear.net/academic/ideas/simulation/index.html

http://www.amirrorclear.net/academic/research-topics/other-topics/hypercomputation.html

https://arxiv.org/abs/1806.08747

This also answers andeslodes's point around physicalism, as the physicalist ontology is recoverable as a special case of the computationalist ontology, where we care about each instantiation, and we also care about low-level things that are always simulatable by powerful models of computation.

However, I do think there's an actually non-trivial question here, about why can we be confident that the brain is copyable by a realistic classical computer with reasonable fidelity, when in principle it could be using quantum mechanics for a non-trivial portion of the brain's operations, which would complicate mind uploading massively?

My general answer here is that quantum computation in warm, wet environments is very hard to error-correct at scale, and the biologically plausible ways to do such a thing fundamentally cannot scale to make a quantum computer in our brains, and anything that is fragile in the presence of noise/error is basically unevolvable by default, because there's no incremental way to make a quantum computer inside a human brain.

That's why we can reasonably use classical models of the brain without it leaving a lot out.

As far as my own theory of identity/consciousness goes, I think the gooder regulator theorem gives am answer on why we would want to have a self-model of ourselves, and I'd argue this is the seed from which consciousness/identity grows, so computations meeting the requirements in the gooder regulator theorem can certainly have identities, which means that I do think Rob Bensinger is more or less right about this topic, modulo caveats.

The caveats are given below:

https://www.lesswrong.com/posts/Dx9LoqsEh3gHNJMDk/fixing-the-good-regulator-theorem#Takeaway [LW(p) · GW(p)]

The main functions of consciousness/identity is to give a consistent abstraction across time so that cooperation and deal making become possible, at least for causal decision theories, at least without stuff like provability logic/program equilibrium, which isn't really evolvable (for other theories like logical decision theory, cooperation in one-shot PD is very possible, but that comes about because they view their identities as closer to an isomorphism class of a program, and consider all instances combined as 1 program, so long as they are functionally equivalent), combined with the necessity of self-modeling yourself in order to get a good outcome, though the self-model here is the more important part for me on consciousness, and the abstraction to a consistent identity comes later.

So for my purposes, I think Rob Bensinger is mostly right about the philosophical underpinnings of identity/consciousness, but I don't blame TAG, @andeslodes [LW · GW] and @sunwillrise [LW · GW] for questioning the consensus, since the consensus was frankly poorly argued, and @Rob Bensinger [LW · GW] was acting far too much like a soldier rather than a scout in the comments section.

Mostly fair, but tiers did have a slight other impact in that they were used to bias the final room: Clay Golem and Hag were equally more-likely to be in the final room, both less so than Dragon and Steel Golem but more so than Orcs and Boulder Trap.

I am claiming that people when informed will want the sun to continuing being the sun. I also think that most people when informed will not really care that much about creating new people, will continue to believe in the act-omission distinction, etc. And that this is a coherent view that will add up to a large set of people wanting things in the solar system to remain conservatively the same. I seperately claim that if this is true, then other people should just respect this preference, and use the other stars that people don't care about for energy.