LessWrong 2.0 Reader

Can we make the robots.txt programmatic by page, and then have a tag we can add to exclude a post from the robots.txt? That feels like the 80/20

I broadly agree with this, though I'll state 2 things:

1. Limited steering ability doesn't equal 0 steering ability, and while there's an argument to be made that people overestimate how much you can do with pure social engineering, I do still think there can be multiple equilibrium points, even if a lot of what happens is ultimately controlled by incentives.

2. AIs probably have a much easier time coordinating on what to do, and importantly can route around a lot of the bottlenecks that exist in human societies solely due to copying, merging and scaling, so assuming alignment is achieved, it's very possible for single humans to do large scale social change by controlling the economy and military, and working your way from there.

I’m still curious about how you’d answer my question above [LW(p) · GW(p)]. Right now we don’t know how to build ASI. Sometime in the future, we will. So there has to be some improvement to AI technology that will happen between now and then. My opinion is that this improvement will involve AI becoming (what you describe as) “better at extrapolating”.

If that’s true, then however we feel about getting AIs that are “better at extrapolating”—its costs and its benefits—it doesn’t much matter, because we’re bound to get those costs and benefits sooner or later on the road to ASI. So we might as well sit tight and find other useful things to do, until such time as the AI capabilities researchers figure it out.

…Furthermore, I don’t think the number of months or years between “AIs that are ‘better at extrapolating’” and ASI is appreciably larger if the “AIs that are ‘better at extrapolating’” arrive tomorrow, versus if they arrive in 20 years. In order to believe that, I think you would need to expect some second bottleneck standing between “AIs that are ‘better at extrapolating’”, and ASI, such that that second bottleneck is present today, but will not be present (as much) in 20 years, and such that the second bottleneck is not related to “extrapolation”.

I suppose that one could argue that availability of compute will be that second bottleneck. But I happen to disagree. IMO [LW · GW] we already have an absurdly large amount of compute overhang with respect to ASI, and adding even more compute overhang in the coming decades won’t much change the overall picture. Certainly plenty of people would disagree with me here. …Although those same people would probably say that “just add more compute” is actually the only way to make AIs that are “better at extrapolation”, in which case my point would still stand.

I don’t see any other plausible candidates for the second bottleneck. Do you? Or do you disagree with some other part of that? Like, do you think it’s possible to get all the way to ASI without ever making AIs “better at extrapolating”? IMO it would hardly be worthy of the name “ASI” if it were “bad at extrapolating”  :)

Belief propagation seems too much of a core of AI capability to me. I'd rather place my hope on GPT7 not being all that good yet at accelerating AI research and us having significantly more time.

This just seems doomed to me. The training runs will be even more expensive, the difficulty of doing anything significant as an outsider ever-higher. If the eventual plan is to get big labs to listen to your research, then isn't it better to start early? (If you have anything significant to say, of course.)

I'd imagine it not too hard to get >1OOM efficiency improvement which one can demonstrate in smaller AI and one might use this to get a lab to listen. If the labs are sufficiently uninterested in alignment it's pretty doomy anyway even if they adopted a better paradigm.

Also government interventions might still happen (perhaps more likely because of AI-caused unemployment than x-risk, and it won't buy amazingly much time, but still).

Also the strategy of "maybe if AIs are more rational they will solve alignment or at least realize that they cannot" seems also very unlikely to me to work on the current DL paradigm, though still slightly helpful.

(Also maybe some supergenius or my future self or some other group can figure something out.)

I thought these were pretty... let's say "exciting"... reads, but I'd be interested to hear more people's opinion of this as a trustworthy source.

I don’t think that. See the bottom part of the comment you’re replying to [LW(p) · GW(p)]. (The part after “Here’s what I would say instead:”)

Sry my comment was sloppy.

Right, my point is, I don’t see any difference between “AIs that produce slop” and “weak AIs” (a.k.a. “dumb AIs”).

(I agree the way I used sloppy in my comment mostly meant "weak". But some other thoughts:)

So I think there are some dimensions of intelligence which are more important for solving alignment than for creating ASI. If you read planecrash, WIS and rationality training seem to me more important in that way than INT.
I don't really have much hope for DL-like systems solving alignment but a similar case might be if an early transformative AI recognizes and says "no I cannot solve the alignment problem. the way my intelligence is shaped is not well suited to avoiding value drift. we should stop scaling and take more time where I work with very smart people like Eliezer etc for some years to solve alignment". And depending on the intelligence profile of the AI it might be more or less likely that this will happen (currently seems quite unlikely).
But overall those "better" intelligence dimensions still seem to me too central for AI capabilities, so I wouldn't publish stuff.

(Btw the way I read John's post was more like "fake alignment proposals are a main failure mode" rather than also "... and therefore we should work on making AIs more rational/sane whatever". So given that I maybe would defend John's framing, but not sure.)

Btw tbc, sth that I think slightly speeds up AI capability but is good to publish is e.g. producing rationality content for helping humans think more effectively (and AIs might be able to adopt the techniques as well). Creating a language for rationalists to reason in more Bayesian ways would probably also be good to publish.

Yeah, basically everything I'm saying is an extension of this (but obviously, I'm extending it much further than you are). We don't exactly care whether the increased rationality is in humans or AI, when the two are interacting a lot. (That is, so long as we're assuming scheming is not the failure mode to worry about in the shorter-term.) So, improved rationality for AIs seems similarly good. The claim I'm considering is that even improving rationality of AIs by a lot could be good, if we could do it.

An obvious caveat here is that the intervention should not dramatically increase the probability of AI scheming!

Belief propagation seems too much of a core of AI capability to me. I'd rather place my hope on GPT7 not being all that good yet at accelerating AI research and us having significantly more time.

This just seems doomed to me. The training runs will be even more expensive, the difficulty of doing anything significant as an outsider ever-higher. If the eventual plan is to get big labs to listen to your research, then isn't it better to start early? (If you have anything significant to say, of course.)

A point in favor of evals being helpful for advancing AI capabilities: https://x.com/polynoamial/status/1887561611046756740

Noam Brown @polynoamial A lot of grad students have asked me how they can best contribute to the field of AI when they are short on GPUs and making better evals is one thing I consistently point to.

TL,DR: K-means explains about as much (or more) variance in the activations as SAEs do.

SAEs explain most of the variance in activations. Is this alone a sign that activations are structured in an SAE-friendly way, i.e. that activations are indeed a composition of sparse features like the superposition hypothesis suggests?

I'm asking myself this questions since I initially considered this as pretty solid evidence: SAEs do a pretty impressive job compressing 512 dimensions into ~100 latents, this ought to mean something, right?

But maybe all SAEs are doing is "dataset clustering" [LW(p) · GW(p)] (the data is cluster-y and SAEs exploit this)---then a different sensible clustering method should also be able do perform similarly well!

I took this SAE graph from Neuronpedia, and added a K-means clustering baseline (with K=1). Think of this as pretty equivalent to a top-k SAE (with k=1). In fact I use the K-means algorithm to find "feature vectors" and then encode / decode the activations just like I would in an SAE (I'm not using the (non-linear) "predict" method of K-means).

It turns out that even a K=1 clustering explains up to 90% of the variance in activations, being matched only by SAEs with K>100. This isn't an entirely fair comparison, since SAEs are optimised for the large-K regime, while I haven't found a K>1 operationalisation of clustering that meaningfully improves over K=1. To have some comparison I'm adding a PCA + Clustering baseline where I apply a PCA before doing the clustering. It does roughly as well as expected, exceeding the SAE reconstruction for most L0 values. The SAEBench upcoming paper also does a PCA baseline so I won't discuss PCA in detail here.

Here's the result for layer 4, 16k latents. See below for plots with layer 3, and/or with 4k latents. (These were the 4 SAEBench suites available on Neuronpedia.)

What about interpretability? Clusters seem "monosemantic" on a skim. In an informal investigation I looked at max-activating dataset examples, and they seem to correspond to related contexts / words like monosemantic SAE features tend to do. I haven't spent much time looking into this though.

Here's the code used to get the clustering & PCA below; the SAE numbers are taken straight from Neuronpedia. Both my code and SAEBench/Neuronpedia use OpenWebText with 128 tokens context length so I hope the numbers are comparable, but there's a risk I missed something and we're comparing apples to oranges.

A final caveat I want to mention is that I think the SAEs I'm comparing here (SAEBench suite for Pythia-70M) are maybe weak. They're only using 4k and 16k latents, for 512 embedding dimensions, using expansion ratios of 8 and 32, respectively (the best SAEs I could find for a ~100M model). But I also limit the number of clusters to the same numbers, so I don't necessarily expect the balance to change qualitatively at higher expansion ratios.

I want to thank @Adam Karvonen [LW · GW], @Lucius Bushnaq [LW · GW], and @Patrick Leask [LW · GW] for feedback on early results, and @Johnny Lin [LW · GW] for implementing an export feature on Neuronpedia for me! I also learned that @scasper [LW · GW] proposed something similar here (though I didn't know about it), I'm excited for follow-ups implementing some of Stephen's advanced ideas (HAC, a probabilistic alg, ...).

Right, my point is, I don’t see any difference between “AIs that produce slop” and “weak AIs” (a.k.a. “dumb AIs”). So from my perspective, the above is similar to : “…Because weak AIs can speed up AI capabilities much easier than they can produce actually good alignment ideas.”

I want to explicitly call out my cliff vs gentle slope [LW(p) · GW(p)] picture from another recent comment. Sloppy AIs can have a very large set of tasks at which they perform very well, but they have sudden drops in their abilities due to failure to extrapolate well outside of that.