LessWrong 2.0 Reader

To be clear, though, it's not the mere fact of modelling attention, but that attention is modelled in a way that makes it seem mysterious or unphysical, and that's what explains our intuitions about phenomwnal consciousness

A big part of the issue is self-reference is hard to reason around, and not just for humans, and it took until the 20th century to get a formal justification for how self-reference could work at all, combined with consciousness being a polarizing and conflationary term since it determines in a lot of people's ethical systems whether uploading is desirable, and whether they should have rights.

Graziano thinks mammals, birds and reptiles are conscious, is 50-50 on octopuses and skeptical of fish and arthropods.

I'd also guess this as a good first guess, though it really depends on the self-modeling ability of animals in general, and in particular the more general the scenarios they have to end up in, the more they are conscious.

I'd weakly guess consciousness is closest to a continuous property, rather than a discrete one in general

And yes, I think Graziano's Attention Schema theory combined with Global Workspace Theory with it's associated Global Neuronal Workspace is a key component in why human consciousness has the weird properties that it does (aside from the fact of being conscious, which a lot of animals are to some degree.)

Ah yep, that's a good clarification.

Excellent work and I think you raise a lot of really good points, which help clarify for me why this research agenda is running into issues, and I think ties in to my concerns about activation space work engendered by recent success in latent obfuscation (https://arxiv.org/abs/2412.09565v1). 

In a way that does not affect the larger point, I think that your framing of the problem of extracting composed features may be slightly too strong: in a subset of cases, e.g. if there is a hierarchical relationship between features (https://www.lesswrong.com/posts/XHpta8X85TzugNNn2/broken-latents-studying-saes-and-feature-co-occurrence-in) SAEs might be able to pull out groups of latents that act compositionally (https://www.lesswrong.com/posts/WNoqEivcCSg8gJe5h/compositionality-and-ambiguity-latent-co-occurrence-and). The relationship to any underlying model compositional encoding is unclear, this probably only works in a few cases, and generally does not seem like a scalable approach, but I think that SAEs may be doing something more complex/weirder than only finding composed features. 

Interesting! Graziano's Attention Schema Theory is also basically the same: he proposes consciousness to be found in our models of our own attention, and that these models evolved to help control attention. To be clear, though, it's not the mere fact of modelling or controlling attention, but that attention is modelled in a way that makes it seem mysterious or unphysical, and that's what explains our intuitions about phenomenal consciousness.^[1]

Graziano thinks mammals, birds and reptiles are conscious, is 50-50 on octopuses and very skeptical of fish and arthropods.

1. ^{^}

   Graziano, 2021:

   In the attention schema theory (AST), having an automatically constructed self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. The reason why such a self-model evolved in the brains of complex animals is that it serves the useful role of modeling, and thus helping to control, the powerful and subtle process of attention, by which the brain seizes on and deeply processes information.

    

   Suppose the machine has a much richer model of attention. Somehow, attention is depicted by the model as a Moray eel darting around the world. Maybe the machine already had need for a depiction of Moray eels, and it coapted that model for monitoring its own attention. Now we plug in the speech engine. Does the machine claim to have consciousness? No. It claims to have an external Moray eel.

   Suppose the machine has no attention, and no attention schema either. But it does have a self-model, and the self-model richly depicts a subtle, powerful, nonphysical essence, with all the properties we humans attribute to consciousness. Now we plug in the speech engine. Does the machine claim to have consciousness? Yes. The machine knows only what it knows. It is constrained by its own internal information.

   AST does not posit that having an attention schema makes one conscious. Instead, first, having an automatic self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. Second, the reason why such a self-model evolved in the brains of complex animals, is that it serves the useful role of modeling attention.

Something interesting here is that a part of why AI companies won't want to use agents is because their capabilities are good enough that being very reckless with them might actually cause small-scale misalignment issues, and if that's truly a big part of the problem in getting companies to adopt AI agents, this is good news for our future:

https://www.lesswrong.com/posts/K2D45BNxnZjdpSX2j/?commentId=qEkRqHtSJfoDA7zJX [LW · GW]

Diplomacy AI by Meta [LW · GW] is a clear example of how adding LLMs can improve narrow game playing. Most multiplayer games with communication will benefit in the same way.

I don't understand your perspective. Many forms of warfighting required soldiers to move towards danger. Your basic Roman infantry charge is running towards people who want to stab you. Some video games can be seen as a safe super-stimulus for warfighting and require moving towards danger. If an innate preference for warfighting stopped being reproductively fit in the 1800s that won't have a large genetic effect by the 2000s.

The bible is large and includes both pro-war and anti-war passages. I agree that a preference for warfighting is not universal, but innate preferences need not be universal. Some are attracted to men, some are attracted to women, some both, some neither; this does not mean that sexual attraction is not an innate preference. The naive evo-psych perspective implies that warfighting is more often an innate preference for young males, and not for women, very young children, and elders. Innate preferences can be deactivated or inverted by reality, as in the turn against warfighting in post-WW1 Europe, or post-Vietnam USA. Or they can be activated.

Without making any normative arguments: if you're in a position (industrially and technologically) to disassemble the sun at all, or build something like a Dyson swarm, then it's probably not too difficult to build an artificial system to light the Earth in such a way as to mimic the sun, and make it look and feel nearly identical to biological humans living on the surface, using less than a billionth of the sun's normal total light output. The details of tides might be tricky, but probably not out of reach.

Nice post! I think these are good criticisms that don't justify the title. Points 1 through 4 are all (specific, plausible) examples of ways we may interpret the activation space incorrectly. This is worth keeping in mind, and I agree that just looking at the activation space of a single layer isn't enough, but it still seems like a very good place to start. 

A layer's activation is a relatively simple space, constructed by the model, that contains all the information that the model needs to make its prediction. This makes it a great place to look if you're trying to understand how the model's thinking.

Yes, many people will have problems with the Archimedes' axiom because it implies that everything has a price (that any good option can be probability-diluted enough that a mediocre is chosen instead), and people don't take it kindly when you tell "you absolutely must have a trade-off between value A and value B"  - especially if they really don't have a trade-off, but also if they don't want to admit or consciously estimate it.

Thankfully, that VNM property is not that critical for rational decision-making because we can simply use surreal numbers instead.

One possible real-world example (with integer-valued $U_1$ for deterministic outcomes) would be a parent whose top priority is minimizing the number of their children who die within the parent's lifetime, with the rest of their utility function being secondary.

Wouldn't work well since in real world outcomes are non-deterministic; given that, minimizing expected number is accomplished by simply having zero children.