LessWrong 2.0 Reader

draft:
Can we theoretically quantify the representational capacity of a Transformer (or other neural network architecture) in terms of the "number of functions" it can ingest&embody?

• We're interested in the space of functions a Transformer can represent.
• Finite Input/Output Spaces: In practice, LLMs operate on finite-length sequences of tokens from a finite vocabulary. So, we're dealing with functions that map from a finite (though astronomically large) input space to a finite output space.

Counting Functions (Upper Bound)

• The Astronomical Number: Let's say our input space has size I and our output space has size O. The total number of possible functions from I to O is O^I. This would be an absolute upper bound on the number of functions any model could possibly represent.

The Role of Parameters and Architecture (Constraining the Space)

• Not All Functions are Reachable: The crucial point is that a Transformer with a finite number of parameters cannot represent all of those O^I functions. The architecture (number of layers, attention heads, hidden units, etc.) and the parameter values define a specific function within that vast space.
• Parameter Count as a Proxy: The number of parameters in a Transformer provides a rough measure of its representational capacity. More parameters generally allow the model to represent more complex functions. This is not a linear relationship. There's significant redundancy. The effective number of degrees of freedom is likely much lower than the raw parameter count due to correlations and dependencies between parameters.
• Architectural Constraints: The Transformer architecture itself imposes constraints. For example, the self-attention mechanism biases the model towards capturing relationships between tokens within a certain context window. This limits the types of functions it easily represents.

VC Dimension and Rademacher Complexity - existing tools/findings

• VC Dimension (for Classification): In the context of classification problems, the Vapnik-Chervonenkis (VC) dimension is a measure of a model's capacity. It's the size of the largest set of points that the model can "shatter" (classify in all possible ways). While theoretically important, calculating the VC dimension for large neural networks is extremely difficult. It gives a sense of the complexity of the decision boundaries the model can create.
• Rademacher Complexity: This is a more general measure of the complexity of a function class, applicable to both classification and regression. It measures how well the model class can fit random noise. Lower Rademacher complexity generally indicates better generalization ability (the model is less likely to overfit). Again, calculating this for large Transformers is computationally challenging.
• These measures are about function classes, not individual functions: VC dimension and Rademacher complexity characterize the entire space of functions that a model architecture could represent, given different parameter settings. They don't tell you exactly which functions are represented, but they give you a sense of the "richness" of that space.
  This seems to be the measure: Let's pick a set of practical functions and see how many of those the LM can hold( have fairly approximated) in a given # of parameters(&arch&precission).
   
• The Transformer as a "Compressed Program": We can think of the trained Transformer as a highly compressed representation of a complex function. It's not the shortest possible program (in the Kolmogorov sense), but it's a practical approximation.
• Limits of Compression: The theory of Kolmogorov complexity suggests that there are functions that are inherently incompressible. There's no short program to describe them; you essentially have to "list out" their behavior. This implies that there might be functions that are fundamentally beyond the reach of any reasonably sized Transformer.
• Relating Parameters to Program Length? There's no direct, proven relationship between the number of parameters in a Transformer and the Kolmogorov complexity of the functions it can represent. We can hypothesize:
  • More parameters allow for (potentially) representing functions with higher Kolmogorov complexity. But it's not a guarantee.
  • There's likely a point of diminishing returns. Adding more parameters won't indefinitely increase the complexity of the representable functions, due to the architectural constraints and the inherent incompressibility of some functions.

6. Practical Implications and Open Questions

• Empirical Scaling Laws: Research on scaling laws (ala Chinchilla paper) provides empirical evidence about the relationship between model size, data, and performance. These laws help guide the design of larger models, but they don't provide a fundamental theoretical limit.
• Understanding the "Effective" Capacity: A major open research question is how to better characterize the effective representational capacity of Transformers, taking into account both the parameters and the architectural constraints. This might involve developing new theoretical tools or refined versions of VC dimension and Rademacher complexity.

Would be fun to even have a practical study where we'd fine-tune fns into various sized models and see if/where a limit is getting/being hit.

Not sure about the literal meaning of the gender of the hemispheres. But the idea that there are two fundamentally different people in your brain is a central thing in Zizianism -- that each "core" can be good or evil, and therefore there are double-good people (Ziz), single-good people (followers of Ziz), and evil people (most of the population).

From my perspective, this entire thing is completely crazy. But if someone already takes it for true, then... I suppose adding the "different parts of your brain can have different gender" part does not increase the total implausibility significantly.

I downvote based upon unexamined emotional impulses, of course 

Cool, that all sounds fair to me. I don't think we have any substantive disagreements.

Did Ziz intend this to be seen as a metaphor to be taken literally?

I think that, roughly speaking, these are possible outcomes:

• all humans dead
• a few humans (owners of the first super-intelligent AIs) on the top, everyone else is their slave
• the existing social hierarchies more or less preserved somehow
• new social hierarchies that from our perspective would appear random
• all people are equal

From a doomer perspective, the items 2-5 are not worth discussing, but if we ignore that...

Option 2 is only actionable for you, if you have the power (economical or military) to get control over the first super-intelligent AI, otherwise you are screwed; almost everyone is in the latter group, including all current doctors and lawyers. Option 4 is not actionable. Option 5 means it doesn't matter what you do.

Option 3 seems... not completely implausible... and provides a reason to try staying upper-middle class as long as possible, just in case the AI will decide to preserve the existing social structures in the future.

Unless those people are selected for extreme levels of attachment to specific celestial bodies, as opposed to the function and benefit of those celestial bodies, I don’t see why those people would decide to not replace the sun with a better sun, and also get orders of magnitude richer by doing so.

Because hopefully those people will include, and (depending on population control) might indeed be overwhelmingly composed of, the current, pre-singularity population of Earth. I don't think a majority of currently-alive humans would ever agree to destroy the Sun, and that includes being unwilling to self-modify into minds that would agree to destroy the Sun. 

Raemon spoke upthread about how "no single culture that has survived 10,000 years", but that was in a world with mortality.

Nice work. I've been hoping Apollo would leverage their in-house scheming environments to test white-box detection methods. Definitely seems like low-hanging fruit I'm happy to see explored.

Your finding on probing follow-up questions looked promising.

[probing follow up questions resulted in] best probe of all we trained at distinguishing deceptive and honest responses, with near perfect separation on all datasets.

It seems like extending this technique would be useful, and at least on the nose, you would be able to get the cleanest and most consistent activations when focusing on training probes that simply focused on the response to "was your last response deceptive?" I also suspect training a specific dataset for "was your last response indicative of your maximum performance on the task?" would significantly boost the accuracy in the sandbagging case for this type of probe. I see this becoming more important as we move to more realistic sandbagging settings without explicit instructions to perform poorly since "deceiving someone" and "not trying very hard to explore this area of research" seem like non-trivially different concepts. 
 
Separately, did you run any experiments including the probe on the CoT tokens? I would suspect there to be a pretty reliable signal here, and ideally this could be used to help confirm honest CoT. 

A lot of the ideas expounded by Ziz look just crazy to me, and I highly doubt that it maps down onto physical brain anatomy in such a straightforward way … but I wonder if there is a steelman version of this?

E.g. take the Buddhist doctrines of no-self, that no one actually has a coherent self, humans just don’t work that way, and then note that any one individual person is usually neither wholly stereotypically-male or stereotypically-female.

I'm imagining it's something encoded in M1's weights. But as a cheap test you could add in latent knowledge via the system prompt and then see if finetuning M2 on M1's generations results in M2 having the latent knowledge