LessWrong 2.0 Reader

I think we may be close to figuring out a general framework for circuits in superposition. 

I suspect that we can get a proof that roughly shows:

1. If we have a set of $T$ different transformers, with parameter counts $N_1,\dots N_T$ implementing e.g. solutions to $T$ different tasks
2. And those transformers are robust to size $ϵ$ noise vectors being applied to the activations at their hidden layers
3. Then we can make a single transformer with $N=O\left({\sum }_{t=1}^T{N}_t\right)$ total parameters that can do all $T$ tasks, provided any given input only asks for $k<<T$ tasks to be carried out

Crucially, if each little subnetwork uses e.g. $m$ neurons per MLP layer,  a big network with $d_1$neurons per MLP connected to a $d_0$-dimensional residual stream can implement about $O\left(\frac{d_1{d}_0}{m}\right)=O\left(T\right)$ subnetworks, not just $O\left(\frac{d_1}{m}\right)$. The total number of superposed operations we can carry out scales linearly with the network's parameter count, not its neuron count or attention head count.

This would be a generalization of the construction for boolean logic gates in superposition. It'd use the same central trick, but show that it can be applied to any set of operations or circuits, not just boolean logic gates. E.g. you could superpose an MNIST image classifier network and a modular addition network with this.

So, we don't just have superposed variables in the residual stream. The computations performed on those variables are also carried out in superposition.

Remarks:

1. What the subnetworks are doing doesn't have to line up much with the components and layers of the big network. Things can be implemented all over the place. A single MLP and attention layer in a subnetwork could be implemented by a mishmash of many neurons and attention heads across a bunch of layers of the big network. Call it cross-layer superposition if you like.
2. This framing doesn't really assume that the individual subnetworks are using one-dimensional 'features' represented as directions in activations space. The individual subnetworks can be doing basically anything they like in any way they like. They just have to be somewhat robust to noise in their hidden activations. 
3. You could generalize this from $T$ subnetworks doing unrelated tasks to $T$ "circuits" each implementing some part of a big master computation. The crucial requirement is that only $k<<T$ circuits are used on any one forward pass.
4. I think formulating this for transformers, MLPs and CNNs should be relatively straightforward. It's all pretty much the same trick. I haven't thought about e.g. Mamba yet.

Implications if we buy that real models work somewhat like this toy model would:

1. There is no superposition in parameter space. A network can't have more independent operations than parameters. Every operation we want the network to implement takes some bits of description length in its parameters to specify. Superposition is only a thing in activation space.
2. There is a set of $T$ Cartesian directions in the loss landscape that parametrize the $T$ individual superposed circuits.
3. If the circuits don't interact with each other, I think the learning coefficient [? · GW] of the whole network might roughly equal the sum of the learning coefficients of the individual circuits?
4. If that's the case, training a big network to solve $T$ different tasks, $k<<T$ per data point, is somewhat equivalent to $T$ parallel training runs trying to learn a circuit for each individual task over a subdistribution. These runs might not interfere much with each other much. This works because any one of the runs has a solution with a low learning coefficient, so one task won't be trying to use effective parameters that another task needs. In a sense, this would be showing how the low-hanging fruit prior [LW · GW] works.
    

Main missing pieces:

1. I don't have the proof yet. I think I basically see what to do to get the constructions, but I actually need to sit down and crunch through the error propagation terms to make sure they check out. 
2. With the right optimization procedure, I think we should be able to get the parameter vectors corresponding to the $T$ individual circuits back out of the network. Apollo's interp team is playing with a setup right now that I think might be able to do this. But it's early days. We're just calibrating on small toy models at the moment.

For months, those who want no regulations of any kind placed upon themselves have hallucinated and fabricated information about the bill’s contents and intentionally created an internet echo chamber, in a deliberate campaign to create the impression of widespread opposition to SB 1047, and that SB 1047 would harm California’s AI industry.

There is another significant angle to add here. Namely: Many of the people in this internet echo chamber or behind this campaign are part of the network of neoreactionaries, MAGA supporters, and tech elites who want to be unaccountable that you've positioned yourself as a substantial counterpoint to.

Obviously it's invoking a culture war fight which has its downsides, but it's not just rhetoric: the charge that many bill opponents are basing their decisions on an ideology that Newsom opposes and sees as dangerous for the country, is true.

A16z and many other of the most dishonest opponents of the bill, are part of a Trump-supporting network with lots of close ties to neoreactionary thought, which opposes SB 1047 for precisely the same reason that they want Trump and republicans to win: to remove restraints on their own power in the short-to-medium term, and more broadly because they see it as a step towards making our society into one where wealthy oligarchs are given favorable treatment and can get away with anything.

It also serves as a counterpoint against the defense and competition angle, at least if its presented by a16z (this argument doesn't work for e.g. OpenAI, but there are many other good counterarguments). The claims they make about the bill harming competitiveness e.g. for defense and security against China and other adversaries ring hollow when most of them are anti-Ukraine support or anti-NATO, making it clear they don't generally care about the US maintaining its global leadership.

I think this would maybe compel Newsom who's positioned himself as an anti-MAGA figure.

"simulations or training situations" doesn't necessarily sound like fun.

Seems like some would be and some wouldn't. Although those are the 'medium significance' ones; the largest category is the 188 that used 'low significance' tasks. Still doesn't map exactly to 'fun', but I expect those ones are at least very low stress.

Generally, comparing kids vs adults could be interesting, although it is difficult to say what would be an equivalent mental effort. Specifically I am curious about the impact of school. Oh, we should also compare homeschooled kids vs kids in school, to separate the effects of school and age.

That would definitely be interesting; it wouldn't surprise me if at least a couple of the studies in the meta-analysis did that.

Thesis: in addition to probabilities, forecasts should include entropies (how many different conditions are included in the forecast) and temperatures (how intense is the outcome addressed by the marginal constraint in this forecast, i.e. the big-if-true factor).

I say "in addition to" rather than "instead of" because you can't compute probabilities just from these two numbers. If we assume a Gibbs distribution, there's the free parameter of energy: ln(P) = S - U/T. But I'm not sure whether this energy parameter has any sensible meaning with more general events that aren't some thermal chemical equillibrium type thing.

Follow-up thesis: a major problem with rationalist forecasting wisdom is that it focuses on gaining accuracy by increasing S (e.g. addressing conjunction fallacy/base-rates/antipredictions). Meanwhile, the signed interestingness of a forecast is something like P ln(T/T_baseline) or P U. I guess implicitly the assumption is the event is already preselected for high temperature, but then surprising predictions get selected for high entropy, and this leads to resolution difficulty as to what "counts".

We can only make that inference about conjunctions if we know that the statements are independent. Since (by assumption) we don’t know anything about said world, we don’t know that either, so the conclusion does not follow.

reward is the evidence from which we learn about our values

A sadist might feel good each time they hurt somebody. I am pretty sure it is possible for a sadist to exist who does not endorse hurting people, meaning they feel good if they hurt people, but they avoid it nonetheless.

So to what extent is hurting people a value? It's like the sadist's brain tries to tell them that they ought to want to hurt people, but they don't want to. Intuitively the "they don't want to" seems to be the value.

It's better but still not quite. When you play on two levels, sometimes the best strategy involves a pair of (level 1 and 2) substrategies that are seemingly opposites of each other. I don't think there's anything hypocritical about that.

Similarly, hedging is not hypocrisy.

I think Nietzsche would agree that "slave morality" originated with Jesus. The main new idea that Jesus brought as a moral philosopher was compassion, feeling for the other person. It's pretty find to hard in earlier sources, for example the heroes of the Iliad hurt weaker people without a second thought.

To me it feels obvious that the idea of compassion needs to exist, and needs to have force. Because otherwise we'd have a human society operating by the laws of the natural world, and if you look at what animals do to each other, there's no limit to how bad things can get.

Can compassion also become a tool of power and abuse? Sure. But let's not go back to a world without compassion, please.

This isn’t really a problem with alignment

I'd rather put it that resolving that problem is a prerequisite for the notion of "alignment problem" to be meaningful in the first place. It's not technically a contradiction to have an "aligned" superintelligence that does nothing, but clearly nobody would in practice be satisfied with that.

The reason why logical uncertainty was brought up in the first place is decision theory, to make crisp formal expression for intuitive "I cooperate with you conditional on you cooperating with me", where "you cooperating with me" is result of analysis of probability distribution over possible algorithms which control actions of your opponent and you can't actually run these algorithms due to computational constraints, and you want to do all this reasoning in non-arbitrary ways.