LessWrong 2.0 Reader

Oh and sorry just to be clear, does this mean you do think that recurrent connections in the cortex are essential for forming intuitive self-models / the algorithm modelling properties of itself?

Wait, but we know that people sometimes have happy moments. Is the idea that such moments are always outweighed by suffering elsewhere? It seems more likely that increasing the proportion of happy moments is doable, an engineering problem. So basically I'd be very happy to see a world such as in the first half of your story, and don't think it would lead to the second half.

Thank you for the response! I am embarrassed that I didn't realise that the lack of recurrent connections referenced in the sources were referring to regions outside of their cortex-equivalent, should've read through more thoroughly :) I am pretty up-to-date in terms of those things.

Can I additionally ask why do you think some invertebrates likely have intuitive self models as well? Would you restrict this possibility to basically just cephalopods and the like (as many do, being the most intelligent invertebrates), or would you likely extend to it to creatures like arthropods as well? (what's your fuzzy estimate that an ant could model itself as having awareness?)

This is really cool! How much computational burden does this add compared to training without the SAEs? 

I could possibly get access to an H100 node at my school's HPC to try this on GPT-2 small.

Some brief reference definitions for clarifying conversations.

Consciousness:

1. The state of being awake and aware of one's environment and existence
2. The capacity for subjective experience and inner mental states
3. The integrated system of all mental processes, both conscious and unconscious
4. The "what it's like" to experience something from a first-person perspective
5. The global workspace where different mental processes come together into awareness

Sentient:

1. Able to have subjective sensory experiences and feelings. Having the capacity for basic emotional responses.
2. Capable of experiencing pleasure and pain, valenced experience, preferences.
3. Able to process and respond to sensory information. Having fundamental awareness of environmental stimuli. Having behavior that is shaped by this perception of external world.

Sapient:

1. Possessing wisdom or the ability to think deeply
2. Capable of complex reasoning and problem-solving
3. Having human-like intelligence and rational thought
4. Able to understand abstract concepts and their relationships
5. Possessing higher-order cognitive abilities like planning and metacognition

Self-aware:

1. Recognizing oneself as distinct from the environment and others. Ability to recognize oneself in a mirror or similar test. Having a concept of "I" or self-model.
2. Capable of introspection. Perceiving and understanding one's own mental states and processes.
3. Capable of abstract self-reflection, modeling potential future behaviors of oneself, awareness of how one has changed over time and may change in the future.

Qualia:

1. The subjective, qualitative aspects of conscious experience. The ineffable, private nature of conscious experiences
2. The phenomenal character of sensory experiences. The raw feel or sensation of an experience (like the redness of red)

The next international PauseAI protest is taking place in one week in London, New York, Stockholm (Sunday 9th Feb), Paris (Mon 10 Feb) and many other cities around the world.

We are calling for AI Safety to be the focus of the upcoming Paris AI Action Summit. If you're on the fence, take a look at Why I'm doing PauseAI [LW · GW].

I'm sure some of people's ignorance of these threat models comes from the reasons. But my intuition is that most of it comes from "these are vaguer threat models that seem very up in the air, and other ones seem more obviously real and more shovel-ready" (this is similar to your "Flinch", but I think more conscious and endorsed).

Thus, I think the best way to converge on whether these threat models are real/likely/actionable is to work through as-detailed-as-possible example trajectories. Someone objects that the state will handle it? Let's actually think through how the state might look like in 5 years! Someone objects that democracy will prevent it? Let's actually think through the actual consequences of cheap cognitive labor in democracy!
This is analogous to what pessimists about single-single alignment have gone through. They have some abstract arguments, people don't buy them, so they start working through them in more detail or provide example failures. I buy some parts of them, but not others. And if you did the same for this threat model, I'm uncertain how much I'd buy!

Of course, the paper might have been your way of doing that. I enjoyed it, but still would have preferred more fully detailed examples, on top of the abstract arguments. You do use examples (both past and hypothetical), but they are more like "small, local examples that embody one of the abstract arguments", rather than "an ambitious (if incomplete) and partly arbitrary picture of how these abstract arguments might actually pan out in practice". And I would like to know the messy details of how you envision these abstract arguments coming into contact with reality. This is why I liked TASRA, and indeed I was more looking forward to an expanded, updated and more detailed version of TASRA.

I don't really have an empirical basis for this, but: If you trained something otherwise comparable to current or near-future reasoning models without any mention of angular momentum, and gave it a context with several different problems to which angular momentum was applicable, I'd be surprised if it couldn't notice that $\overrightarrow{r}\times \overrightarrow{p}$ was a common interesting quantity, and then, in an extension of that context, correctly answer questions about it. If you gave it successive problem sets where the sum of that quantity was applicable, the integral, maybe other things, I'd be surprised if a (maybe more powerful) reasoning model couldn't build something worth calling the ability to correctly answer questions about angular momentum. Do you expect otherwise, and/or is this not what you had in mind?

the extent human civilization is human-aligned, most of the reason for the alignment is that humans are extremely useful to various social systems like the economy, and states, or as substrate of cultural evolution. When human cognition ceases to be useful, we should expect these systems to become less aligned, leading to human disempowerment.

oh good, I've been thinking this basically word for word for a while and had it in my backlog. Glad this is written up nicely, far better than I would likely have done :)

The one thing I'm not a big fan of: I'd bet "Gradual Disempowerment" sounds like a "this might take many decades or longer" to most readers, whereas with capabilities curves this could be a few months to single digit years thing.

I don't think that most people, upon learning that Anthropic's justification was "other companies were already putting everyone's lives at risk, so our relative contribution to the omnicide was low" would then want to abstain from rioting. Common ethical intuitions are often more deontological than that, more like "it's not okay to risk extinction, period." That Anthropic aims to reduce the risk of omnicide on the margin is not, I suspect, the point people would focus on if they truly grokked the stakes; I think they'd overwhelmingly focus on the threat to their lives that all AGI companies (including Anthropic) are imposing.