LessWrong 2.0 Reader

Yeah, this seems probably a good idea. Though some of these would be best on existing resource pages, like the funders list.

Thank you. Yes, our claim isn't that SAEs only find composed features. Simple counterexample: Make a product space of two spaces with $9$ dictionary elements each, with an average of $3$ features active at a time in each factor space. Then the dictionary of $81$ composed features has an $L_0$ of $9$, whereas the dictionary of $18$ factored features has an $L_0$ of $6$, so a well-tuned SAE will learn the factored set of features. Note however that just because the dictionary of $18$ factored features is sparser doesn't mean that those are the features of the model. The model could be using the $81$ composed features instead, because that's more convenient for the downstream computations somehow, or for some other reason.

Our claim is that an SAE trained on the activations at a single layer cannot tell whether the features of the model are in composed representation or factored representation, because the representation the model uses need not be the representation with the lowest $L_0$. 

Collection of how-to guides

• Research soft skills
  • How to make research slides [LW · GW] by James Chua and John Hughes
  • How to manage up by Henry Sleight
  • How to ML series by Tim rocktaschel and Jakob Foerster
    • How to write an ML paper
    • How to write a rebuttal
    • How to review
• Procedural expertise
  • "How to become an expert at a thing" by Karpathy
  • Mastery, by Robert Greene
• Working sustainably
  • Slow Productivity by Cal Newport
  • Feel-good Productivity by Ali Abdaal

Some other guides I'd be interested in 

• How to write a survey / position paper
• "How to think better" - the Sequences probably do this, but I'd like to read a highly distilled 80/20 version of the Sequences

To be clear, though, it's not the mere fact of modelling attention, but that attention is modelled in a way that makes it seem mysterious or unphysical, and that's what explains our intuitions about phenomwnal consciousness

A big part of the issue is self-reference is hard to reason around, and not just for humans, and it took until the 20th century to get a formal justification for how self-reference could work at all, combined with consciousness being a polarizing and conflationary term since it determines in a lot of people's ethical systems whether uploading is desirable, and whether they should have rights.

Graziano thinks mammals, birds and reptiles are conscious, is 50-50 on octopuses and skeptical of fish and arthropods.

I'd also guess this as a good first guess, though it really depends on the self-modeling ability of animals in general, and in particular the more general the scenarios they have to end up in, the more they are conscious.

I'd weakly guess consciousness is closest to a continuous property, rather than a discrete one in general

And yes, I think Graziano's Attention Schema theory combined with Global Workspace Theory with it's associated Global Neuronal Workspace is a key component in why human consciousness has the weird properties that it does (aside from the fact of being conscious, which a lot of animals are to some degree.)

Ah yep, that's a good clarification.

Excellent work and I think you raise a lot of really good points, which help clarify for me why this research agenda is running into issues, and I think ties in to my concerns about activation space work engendered by recent success in latent obfuscation (https://arxiv.org/abs/2412.09565v1). 

In a way that does not affect the larger point, I think that your framing of the problem of extracting composed features may be slightly too strong: in a subset of cases, e.g. if there is a hierarchical relationship between features (https://www.lesswrong.com/posts/XHpta8X85TzugNNn2/broken-latents-studying-saes-and-feature-co-occurrence-in) SAEs might be able to pull out groups of latents that act compositionally (https://www.lesswrong.com/posts/WNoqEivcCSg8gJe5h/compositionality-and-ambiguity-latent-co-occurrence-and). The relationship to any underlying model compositional encoding is unclear, this probably only works in a few cases, and generally does not seem like a scalable approach, but I think that SAEs may be doing something more complex/weirder than only finding composed features. 

Interesting! Graziano's Attention Schema Theory is also basically the same: he proposes consciousness to be found in our models of our own attention, and that these models evolved to help control attention. To be clear, though, it's not the mere fact of modelling or controlling attention, but that attention is modelled in a way that makes it seem mysterious or unphysical, and that's what explains our intuitions about phenomenal consciousness.^[1]

Graziano thinks mammals, birds and reptiles are conscious, is 50-50 on octopuses and very skeptical of fish and arthropods.

1. ^{^}

   Graziano, 2021:

   In the attention schema theory (AST), having an automatically constructed self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. The reason why such a self-model evolved in the brains of complex animals is that it serves the useful role of modeling, and thus helping to control, the powerful and subtle process of attention, by which the brain seizes on and deeply processes information.

    

   Suppose the machine has a much richer model of attention. Somehow, attention is depicted by the model as a Moray eel darting around the world. Maybe the machine already had need for a depiction of Moray eels, and it coapted that model for monitoring its own attention. Now we plug in the speech engine. Does the machine claim to have consciousness? No. It claims to have an external Moray eel.

   Suppose the machine has no attention, and no attention schema either. But it does have a self-model, and the self-model richly depicts a subtle, powerful, nonphysical essence, with all the properties we humans attribute to consciousness. Now we plug in the speech engine. Does the machine claim to have consciousness? Yes. The machine knows only what it knows. It is constrained by its own internal information.

   AST does not posit that having an attention schema makes one conscious. Instead, first, having an automatic self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. Second, the reason why such a self-model evolved in the brains of complex animals, is that it serves the useful role of modeling attention.

Something interesting here is that a part of why AI companies won't want to use agents is because their capabilities are good enough that being very reckless with them might actually cause small-scale misalignment issues, and if that's truly a big part of the problem in getting companies to adopt AI agents, this is good news for our future:

https://www.lesswrong.com/posts/K2D45BNxnZjdpSX2j/?commentId=qEkRqHtSJfoDA7zJX [LW · GW]

Diplomacy AI by Meta [LW · GW] is a clear example of how adding LLMs can improve narrow game playing. Most multiplayer games with communication will benefit in the same way.

I don't understand your perspective. Many forms of warfighting required soldiers to move towards danger. Your basic Roman infantry charge is running towards people who want to stab you. Some video games can be seen as a safe super-stimulus for warfighting and require moving towards danger. If an innate preference for warfighting stopped being reproductively fit in the 1800s that won't have a large genetic effect by the 2000s.

The bible is large and includes both pro-war and anti-war passages. I agree that a preference for warfighting is not universal, but innate preferences need not be universal. Some are attracted to men, some are attracted to women, some both, some neither; this does not mean that sexual attraction is not an innate preference. The naive evo-psych perspective implies that warfighting is more often an innate preference for young males, and not for women, very young children, and elders. Innate preferences can be deactivated or inverted by reality, as in the turn against warfighting in post-WW1 Europe, or post-Vietnam USA. Or they can be activated.