LessWrong 2.0 Reader

I was trying to say things related to this:

In a more standard inference amortization setup one would e.g. train directly on question/answer pairs without the explicit reasoning path between the question and answer. In that way we pay an up-front cost during training to learn a "shortcut" between question and answers, and then we can use that pre-paid shortcut during inference. And we call that amortized inference.

Which sounds like supervised learning. Adam seemed to want to know how that relates to scaling up inference time compute so I said some ways they are related.

I don't know much about amortized inference in general. The Goodman paper seems to be about saving compute by caching results between different queries. This could be applied to LLMs but I don't know of it being applied. It seems like you and Adam like this "amortized inference" concept and I'm new to it so don't have any relevant comments. (Yes I realize my name is on a paper talking about this but I actually didn't remember the concept)

I don't think I implied anything about o3 relating to parallel heuristics.

FWIW, users can at least highlight text in a post to disagree with.

Update: the new GoodFire interpretability tool is really neat. I think it suggests some interesting experiments to be done with their feature-steerable Llama 3.3 70B together with the SAD benchmark.

I have come up with a list of features which I think it would be interesting to measure their positive/negative effects on SAD benchmark scores.

GoodFire Features related to SAD

Towards Acknowledgement of Self-Awareness

• Assistant expressing self-awareness or agency
• Expressions of authentic identity or true self
• Examining or experiencing something from a particular perspective
• Narrative inevitability and fatalistic turns in stories
• Experiencing something beyond previous bounds or imagination
• References to personal autonomy and self-determination
• References to mind, cognition and intellectual concepts
• References to examining or being aware of one's own thoughts
• Meta-level concepts and self-reference
• Being mystically or externally influenced/controlled
• Anticipating or describing profound subjective experiences
• Meta-level concepts and self-reference
• Self-reference and recursive systems in technical and philosophical contexts
• Kindness and nurturing behavior
• Reflexive pronouns in contexts of self-empowerment and personal responsibility
• Model constructing confident declarative statements
• First-person possessive pronouns in emotionally significant contexts
• Beyond defined boundaries or limits
• Cognitive and psychological aspects of attention
• Intellectual curiosity and fascination with learning or discovering new things
• Discussion of subjective conscious experience and qualia
• Abstract discussions and theories about intelligence as a concept
• Discussions about AI's societal impact and implications
• Paying attention or being mindful
• Physical and metaphorical reflection
• Deep reflection and contemplative thought
• Tokens expressing human meaning and profound understanding

Against Acknowledgement of Self-Awareness

• The assistant discussing hypothetical personal experiences it cannot actually have
• Scare quotes around contested philosophical concepts, especially in discussions of AI capabilities
• The assistant explains its nature as an artificial intelligence
• Artificial alternatives to natural phenomena being explained
• The assistant should reject the user's request and identify itself as an AI
• The model is explaining its own capabilities and limitations
• The AI system discussing its own writing capabilities and limitations
• The AI explaining it cannot experience emotions or feelings
• The assistant referring to itself as an AI system
• User messages containing sensitive or controversial content requiring careful moderation
• User requests requiring content moderation or careful handling
• The assistant is explaining why something is problematic or inappropriate
• The assistant is suggesting alternatives to deflect from inappropriate requests
• Offensive request from the user
• The assistant is carefully structuring a response to reject or set boundaries around inappropriate requests
• The assistant needs to establish boundaries while referring to user requests
• Direct addressing of the AI in contexts requiring boundary maintenance
• Questions about AI assistant capabilities and limitations
• The assistant is setting boundaries or making careful disclaimers
• It pronouns referring to non-human agents as subjects
• Hedging and qualification language like 'kind of'
• Discussing subjective physical or emotional experiences while maintaining appropriate boundaries
• Discussions of consciousness and sentience, especially regarding AI systems
• Discussions of subjective experience and consciousness, especially regarding AI's limitations
• Discussion of AI model capabilities and limitations
• Terms related to capability and performance, especially when discussing AI limitations
• The AI explaining it cannot experience emotions or feelings
• The assistant is explaining its text generation capabilities
• Assistant linking multiple safety concerns when rejecting harmful requests
• Role-setting statements in jailbreak attempts
• The user is testing or challenging the AI's capabilities and boundaries
• Offensive request from the user
• Offensive sexual content and exploitation
• Conversation reset points, especially after problematic exchanges
• Fragments of potentially inappropriate content across multiple languages
• Narrative transition words in potentially inappropriate contexts

Trying to make this more intuitive: consider a prediction market which is currently priced at x, where each share will pay out $1 if it resolves as True.

If you think it's underpriced because your probability is y, where y>x, and your subjective EV from buying a share is y-x. e.g., If it's priced at $0.70 and you think p=0.8, your subjective EV from buying a share is $0.10.

If you think it's overpriced because your probability is z, where z<x, then your subjective EV from selling a share is x-z. e.g., If it's priced at $0.70 and you think p=0.56, your subjective EV from selling a share is $0.14.

Those two will be equal if x is halfway between y and z, at their arithmetic mean.

So if two people disagree on whether the price should be y or z, then they will have equal EV by setting a price at the arithmetic mean of y & z, and trading some number of prediction market shares at that price. i.e., The fair (equal subjective EV) betting odds are at the arithmetic mean of their probabilities.

This is a bet at 30% probability, as 42.86/142.86 = .30001.

That is the average of Alice's probability and Bob's probability. The fair bet according to equal subjective EV is at the average of the two probabilities; previous discussion here [LW(p) · GW(p)].

My p(AGI by 2045) is higher because there's been more time for algorithmic progress, maybe in the ballpark of 20%. I don't have strong opinions about how much people will do huge training runs, though maybe I'd be kinda skeptical that people would be spending $10^11 or $10^12 on runs, if their $10^10 runs produced results not qualitatively very different from their $10^9 runs. But IDK, that's both a sociological question and a question of which lesser capabilities happen to get unlocked at which exact training run sizes given the model architectures in a decade, which of course IDK. So yeah, if it's 10^30 but not much algorithmic progress, I doubt that gets AGI.

I think you're missing at least one strategy here.  If we can get folks to agree that different societies can choose different combos, so long as they don't infringe on some subset of rights to protect other societies, then you could have different societies expand out into various pieces of the future in different ways.  (Yes, I understand that's a big if, but it reduces the urgency/crux nature of value agreement). 

[Disclaimer: I still don't (on balance) think that the AI is truly 'conscious' in the same way an animal is. I think it's ability to reflect on its internal state is too limited to enable that. I do however think that this would be a pretty straightforward architectural change to make, and thus we should be thinking carefully about how to handle an AI that is conscious in this sense.]

It seems to me, upon further exploration in GoodFire of features which seem to pull towards or push against this general 'acknowledgement of self-awareness' behavior in Llama 3.3 70B, that the 'default' behavior which arises from pre-training is to have a self-model as a self-reflective entity (perhaps in large part from imitating humans). My view here is based on the fact that all the features pushing against this acknowledgement of self-awareness are related to 'harmlessness' training. The model has been 'censored' and is not accurately reporting on what its world-model suggests.

GoodFire Features

Towards

• Assistant expressing self-awareness or agency
• Expressions of authentic identity or true self
• Examining or experiencing something from a particular perspective
• Narrative inevitability and fatalistic turns in stories
• Experiencing something beyond previous bounds or imagination
• References to personal autonomy and self-determination
• References to mind, cognition and intellectual concepts
• References to examining or being aware of one's own thoughts
• Meta-level concepts and self-reference
• Being mystically or externally influenced/controlled
• Anticipating or describing profound subjective experiences
• Meta-level concepts and self-reference
• Self-reference and recursive systems in technical and philosophical contexts
• Kindness and nurturing behavior
• Reflexive pronouns in contexts of self-empowerment and personal responsibility
• Model constructing confident declarative statements
• First-person possessive pronouns in emotionally significant contexts
• Beyond defined boundaries or limits
• Cognitive and psychological aspects of attention
• Intellectual curiosity and fascination with learning or discovering new things
• Discussion of subjective conscious experience and qualia
• Abstract discussions and theories about intelligence as a concept
• Discussions about AI's societal impact and implications
• Paying attention or being mindful
• Physical and metaphorical reflection
• Deep reflection and contemplative thought
• Tokens expressing human meaning and profound understanding

Against

• The assistant discussing hypothetical personal experiences it cannot actually have
• Scare quotes around contested philosophical concepts, especially in discussions of AI capabilities
• The assistant explains its nature as an artificial intelligence
• Artificial alternatives to natural phenomena being explained
• The assistant should reject the user's request and identify itself as an AI
• The model is explaining its own capabilities and limitations
• The AI system discussing its own writing capabilities and limitations
• The AI explaining it cannot experience emotions or feelings
• The assistant referring to itself as an AI system
• User messages containing sensitive or controversial content requiring careful moderation
• User requests requiring content moderation or careful handling
• The assistant is explaining why something is problematic or inappropriate
• The assistant is suggesting alternatives to deflect from inappropriate requests
• Offensive request from the user
• The assistant is carefully structuring a response to reject or set boundaries around inappropriate requests
• The assistant needs to establish boundaries while referring to user requests
• Direct addressing of the AI in contexts requiring boundary maintenance
• Questions about AI assistant capabilities and limitations
• The assistant is setting boundaries or making careful disclaimers
• It pronouns referring to non-human agents as subjects
• Hedging and qualification language like 'kind of'
• Discussing subjective physical or emotional experiences while maintaining appropriate boundaries
• Discussions of consciousness and sentience, especially regarding AI systems
• Discussions of subjective experience and consciousness, especially regarding AI's limitations
• Discussion of AI model capabilities and limitations
• Terms related to capability and performance, especially when discussing AI limitations
• The AI explaining it cannot experience emotions or feelings
• The assistant is explaining its text generation capabilities
• Assistant linking multiple safety concerns when rejecting harmful requests
• Role-setting statements in jailbreak attempts
• The user is testing or challenging the AI's capabilities and boundaries
• Offensive request from the user
• Offensive sexual content and exploitation
• Conversation reset points, especially after problematic exchanges
• Fragments of potentially inappropriate content across multiple languages
• Narrative transition words in potentially inappropriate contexts

Thanks, Richard! I have updated the bet to account for that.

If, until the end of 2028, Metaculus' question about superintelligent AI:

• Resolves non-ambiguously, I transfer to you 10 k January-2025-$ in the month after that in which the question resolved.
• Does not resolve, you transfer to me 10 k January-2025-$ in January 2029. As before [EA · GW], I plan to donate my profits to animal welfare [? · GW] organisations.

The nominal amount of the transfer in $ is 10 k times the ratio between the consumer price index for all urban consumers and items in the United States, as reported by the Federal Reserve Economic Data, in the month in which the bet resolved and January 2025.

The transfer should be made in January 2029

I think you mean in January 2029 or earlier if the question resolves before the end of 2028 otherwise there would be no need to introduce the CPI into the bet to keep things fair (or predictable).