LessWrong 2.0 Reader

There are ~two broad ways of thinking about RG. 'HEP-like' is structural; it helps roughly map out the space of theories at different scales/couplings, even if it does not capture all of physics at all times (for example, the standard model is a good description up to some energy scale, but each sector is often treated independently). Different aspects of training and inference seems to take the shape of renormalization, although as we pointed out there is a lot of work to be done to understand the various scales and couplings inherent to NNs. A goal of this opportunity space is not to make RG 'go backwards', but to correctly map this renormalization picture onto a 'space of representations' of a NN theory. I don't expect this to be easy or simple, but am hopeful that it will shed more theoretical insight onto different training and inference regimes and point to insightful conceptual gaps. 

In contrast, interpreting NNs to a certain degree of (human-specified) abstraction is more CMT-like. I suspect you are focusing on this perspective overall. Again, as we pointed out, this formulation of RG is not invertible, there are likely many ways to coarse grain a representation (just as there are many RG schemes in condensed matter), but that doesn't mean you can't reframe something like a sparse autoencoder as doing RG. 

Sure, there are 'lots of phase transitions', but that doesn't mean that it's pointless to try and classify or describe some interesting one (like memorization -> generalization). Similarly, just because lots of interesting physics happens far from phase transitions doesn't mean they aren't useful 'tethers' in the space of theories (phi-4 theory has been pretty useful).

Regarding attention, I agree that a theory of representations here would be non-local with respect to inputs. That's fine. Long range dependencies just change the flavor of RG. A lot depends on how we measure locality, since there's no a priori natural way to do this in NNs (although there are several options that likely pick up on different relationships between features). 

 

I previously posted Was the K-T event a Great Filter? [LW · GW] as a pushback against the notion that different lineages of life on Earth evolving intelligence is really "independent evidence" in any meaningful sense. Intelligence can evolve only if there's selective pressure favoring it, and a large part of that pressure likely comes from the presence of other intelligent creatures competing for resources. Therefore mammals and birds together really should only count as one data point.

(It's more plausible that octopus intelligence is independent, since the marine biome is largely separate from the terrestrial, although of course not totally.)

If we remap main actor in AGI27 like this:

Human civ (in the paper)  -> evolution (on earth)

Then it strikes how

Agent 4 (in the paper) -> Human civ (or the part of it that is involved in AI dev)

Fits perfectly - I hear the clink thinking about it.

I first heard this idea from Joscha Bach, and it is my favorite explanation of free will. I have not heard it called as a 'predictive-generative gap' before though, which is very well formulated imo

That was a very useful answer, thank you! I'm going to try to repeat back the model in my own words and relate back to the frame of the post and rest of this comment thread. Please let me know if it still sounds like I'm missing something.

Model: the cases where a woman has a decent idea of what she wants aren't the central use-case of subtle cues in the first place. The central use case is when the guy seems maybe interesting, and therefore she mostly just wants to spend more time around him, not in an explicitly romantic or sexual way yet. The "subtle signals" mostly just look like "being friendly", and that's a feature because in the main use case "being friendly" is in fact basically what the girl wants; she actually does just want to spend a bit more time together in a friendly way, with romantic/sexual interaction as a possibility on the horizon rather than an immediate interest.

Relating that back to the post and comment thread: subtle signals still seem like a pretty stupid choice once a woman is confident she's romantically/sexually interested in a guy. So "send subtle signals" remains terrible advice for women in that situation. But if one is going to give the opposite advice - i.e. advise a policy of asking out men she's interested in and/or sending extremely unsubtle signals - then it should probably come alongside the disclaimer that if she's highly uncertain and mostly wants to gather more information (which may be most of the time) then subtle signals are a sensible move. If the thing she actually wants right now is to just spend some more time together in a friendly way, with romantic/sexual interaction as a possibility on the horizon rather than an immediate thing, then subtle signals are not a bad choice. The subtle signals are mostly not distinguishable from "being friendly" because "being friendly" is in fact the (immediate) goal.

What I really like about this model is that it answers the question "under what circumstances should one apply this advice vs apply its opposite?", which is something which most advice should answer but doesn't.

(Sorry, hard to differentiate quotes from you vs quotes from the paper in this format)

(I agree it is assuming that the judge has that goal, but I don't see why that's a terrible assumption.)

If the judge is human, sure. If the judge is another AI, it seems like a wild assumption to me. The section on judge safety in your paper does a good job of listing many of the problems. On thing I want to call out as something I more strongly disagree with is:

One natural approach to judge safety is bootstrapping: when aligning a new AI system, use the previous generation of AI (that has already been aligned) as the judge.

I don't think we have any fully aligned models, and won't any time soon. I would expect bootstrapping will at most align a model as thoroughly as its predecessor was aligned (but probably less), and goodhart's law definitely applies here.

You don't have to stop the agent, you can just do it afterwards.

Maybe this is just using the same terms for two different systems, but the paper also talks about using judging for monitoring deployed systems:

This approach is especially well-suited to monitoring deployed systems, due to the sheer scale required at deployment.

Is this intended only as a auditing mechanism, not a prevention mechanism (eg. we noticed in the wild the AI is failing debates, time to roll back the release)? Or are we trusting the AI judges enough at this point that we don't need to stop and involve a human? I also worry the "cheap system with high recall but low precision" will be too easy to fool for the system to be functional past a certain capability level.
 

From a 20000ft perspective, this all just looks like RLHF with extra steps that make it harder to reason about whether it'll work or not. If we had evidence that RLHF worked great, but the only flaw with it was the AI was starting to get too complex for humans to give feedback, then I would be more interested in all of the fiddly details of amplified oversight and working through them. The problem is RLHF already doesn't work, and I think it'll only work less well as AI becomes smarter and more agentic.

“It is probably a lot more efficient to show that by modifying the LAION database and slapping some sort of image search on it, so people can see that their pictures were used to train the model.”

Sounds great, bohaska! Yes, it should be possible to have an image search so people can find their personal photos or their faces (or something very similar to their faces). The art gallery idea just makes it more appealing to YouTubers and millions of viewers, they like haunted houses in 3D.

Wolfram mentioned “bridges between concepts”: you can have a picture of a human face that morphs into a dog. Crazy stuff like that will potentially make people outraged and/or will inspire them to become AI interpretability researchers, the more people pay attention and try to understand what’s inside the better. We want to democratize AI interpretability. 

Ideally people will make a fully open 3D game-like LLM where you can walk of fly inside. We can have millions of gamers do a simple sort of interpretability research/get inspired to do it for real. 

The most important thing about Llama 4 is that the 100K H100s run that was promised got canceled, and its flagship model Behemoth will be a 5e25 FLOPs compute optimal model^[1] rather than a ~3e26 FLOPs model that a 100K H100s training system should be able to produce. This is merely 35% more compute than Llama-3-405B from last year, while GPT-4.5, Grok 3 and Gemini 2.5 Pro are probably around 3e26 FLOPs or a bit more. They even explicitly mention that it was trained on 32K GPUs (which must be H100s). Since Behemoth is the flagship model, a bigger model got pushed back to Llama 5, which will only come out much later, possibly not even this year.

In contrast, capabilities of Maverick are unsurprising and prompt no updates. It's merely a 2e24 FLOPs ~7x overtrained model^[2], which is 2x less compute than DeepSeek-V3 and 100x less than the recent frontier models, and also it's not a reasoning model for now. So of course it's not very good. If it was very good with this little compute, that would be a feat on the level of Anthropic or DeepSeek, which would be a positive update about Meta's model training competence, but this unexpected thing merely didn't happen, so nothing to see here, what are people even surprised about (except some benchmarking shenanigans).

To the extent Llamas 1-3 were important open weights releases that could be run by normal people locally, Llama 4 does seem disappointing, because there are no small models (in total params), though as Llama 3.2 demonstrated this might change shortly. Even the smallest Scout model still has 109B total params, meaning a 4 bit quantized version might fit on high end consumer hardware, but all the rest is only practical with datacenter hardware.

• Like Wolfram present a diffusion model as a world of concepts. But remove the noise, make the generated concepts like pictures in an art galley (so make 2D pictures stand upright like pictures in this 3D simulated art gallery), this way gamers and YouTubers will see how dreadful those model really are inside. There is a new monster every month on YT, they get millions of views. We want the public to know that AI companies make real-life Frankenstein monsters with some very crazy stuff inside of their electronic "brains" (inside of AI models). It can help to spread the outrage, if people also see their personal photos are inside of those models. If they used the whole output of humanity to train their models, those models should benefit the whole humanity, not cost $200/month like paid ChatGPT. People should be able to see what's in the model, right now a chatbot is like a librarian that spits quotes at you but doesn't let you enter the library (the AI model).

Okay, so you propose a mechanistic interpretability program where you create a virtual gallery of AI concepts extracted from Stable Diffusion, represented as images. I am slightly skeptical that this would move the needle on AI safety significantly, we already have databases like LAION which are open-source databases of scraped images used to train AI models, and I don't see that much outrage over it. I mean, there is some outrage, but not a significantly large amount to be a cornerstone of an AI safety plan.

gamers and YouTubers will see how dreadful those model really are inside. There is a new monster every month on YT, they get millions of views. We want the public to know that AI companies make real-life Frankenstein monsters with some very crazy stuff inside of their electronic "brains" (inside of AI models).

What exactly do you envision that is being hidden inside these Stable Diffusion concepts? What "crazy stuff" is in it? I'm currently not aware of anything about their inner representations that is especially concerning.

It can help to spread the outrage, if people also see their personal photos are inside of those models.

It is probably a lot more efficient to show that by modifying the LAION database and slapping some sort of image search on it, so people can see that their pictures were used to train the model. 

I'd distinguish 'imparting information that happens to induce guilt' from 'guilting', based on intent to cooperatively inform vs. psychologically attack.

Mhm, and in practice no one who accuses of guilt tripping actually cares about that distinction; if someone is being made to look bad then they basically never wonder if it's right. I'm not objecting to the 'guilt-tripping' framing for no reason; it's a thought-terminating cliche in 99.99% of cases where it's used.

[reading what I actually wrote here] ... And anyways 'inducing guilt' is what the most relevant informing-act looks like; if you're doing something wrong then you don't necessarily change it without attending to the exact details which would induce guilt. I never even said anything about 'guilting'; OP explicitly discouraged a correct thing to do without even mentioning 'guilting'.