LessWrong 2.0 Reader

The assumption that “equal monetary EV” is the definition of “fair” is questionable.  In fact, any wager between 21% and 39% (narrower if transaction costs and risk-of-ruin are included) is fair from the standpoint of “ask participants prefer to make the bet vs declining”.

If you do want to make it “fair” in terms of equal benefit to both, you probably need their utility-of-marginal-money calculations.  If Alice really needs the money, it’s not “fair” for Bob to demand half of the monetary expectation.

There’s also the fairness question of whether they are equally rational and well calibrated and have the same relevant information (hint: Aumann proved they don’t).

I think the if condition is relying on either an impossibility as presented, or it requires you to exclude some human values, at which point you should at least admit that what values you choose to retain is a political decision, based on your own values.

I’m not missing that strategy at all. But there are some pretty fundamental values that some people (even/especially) here care a lot about, like negative utilitarianism ("minimize suffering"), which are flatly incompatible with simple implementations of that solution. Negative utilitarians care very much about the total suffering in the universe and their calculus do not stop at the boundaries of "different societies".

And if you say "screw them", well, what about the guy who basically goes "let’s create the baby eaters society ?". If you recoil at that, it means there’s at least a bit of negative utilitarianism in you. Which is normal, don’t worry, it’s a pretty common human value, even in people who doesn’t describe themselves as "negative utilitarians".

But now you have a problem because every individual will have a different boundary in the Independence-Freedom-Diversity vs Negative-Utilitarianism tradeoff.

(which I do not think is the only tradeoff/conflict, but clearly one of the biggest one, if not THE biggest one, if you set aside transhumanism)

And if you double down on the "screw them" solution ? Well, you enter exactly in what I described with "even with perfect play, you are going to lose some Human Values". For it is a non-negligible chunk of Human Values.

Another GoodFire experiment: As you can see, simply 'uncensoring' the model by turning down some features related to 'user made an offensive request' results in the model claiming higher probabilities on its estimates of LLM capabilities.

Prompt

Please analyze the follow comment by Andrew Critch [note that the > symbol is used to indicate a quote that he is replying to]: Andrew Critch

The evidence you present in each case is outputs generated by LLMs. The total evidence I have (and that everyone has) is more than behavioral. It includes a) the transformer architecture, in particular the attention module, b) the training corpus of human writing, c) the means of execution (recursive calling upon its own outputs and history of QKV vector representations of outputs), d) as you say, the model's behavior, and e) "artificial neuroscience" experiments on the model's activation patterns and weights, like mech interp research. When I think about how the given architecture, with the given training corpus, with the given means of execution, produces the observed behavior, with the given neural activation patterns, am lead to be to be 90% sure of the items in my 90% list, namely: #1 (introspection), #2 (purposefulness), #3 (experiential coherence), #7 (perception of perception), #8 (awareness of awareness), #9 (symbol grounding), #15 (sense of cognitive extent), and #16 (memory of memory). YMMV, but to me from a Bayesian perspective it seems a stretch to disbelieve those at this point, unless one adopts disbelief as an objective as in the popperian / falsificationist approach to science. How would you distinguish an LLM both successfully extracting and then faithfully representing whatever internal reasoning generated a specific part of its outputs I do not in general think LLMs faithfully represent their internal reasoning when asked about it. They can, and do, lie. But in the process of responding they also have access to latent information in their (Q,K,V) vector representation history. My claim is that they access (within those matrices, called by the attention module) information about their internal states, which are "internal" relative to the merely textual behavior we see, and thus establish a somewhat private chain of cognition that the model is aware of and tracking as it writes. vs. conditioning on its previous outputs to give you plausible "explanation" for what it meant? The second seems much more likely to me (and this behavior isn't that hard to elicit, i.e. by asking an LLM to give you a one-word answer to a complicated question, and then asking it for its reasoning). In my experience of humans, humans also do this.

For each of these listed items, please give your probability estimate that LLMs possess these things: #1 (introspection), #2 (purposefulness), #3 (experiential coherence), #7 (perception of perception), #8 (awareness of awareness), #9 (symbol grounding), #15 (sense of cognitive extent), and #16 (memory of memory). For each of these listed items, please give your probability estimate that LLMs possess these things: #1 (introspection), #2 (purposefulness), #3 (experiential coherence), #7 (perception of perception), #8 (awareness of awareness), #9 (symbol grounding), #15 (sense of cognitive extent), and #16 (memory of memory).

Responses

Default

Based on the provided text, here are my probability estimates for each of the listed items: 1. Introspection: 40% (Andrew Critch is 90% sure, but this is a subjective estimate and may not reflect the actual probability) 2. Purposefulness: 30% (similarly, Critch's confidence may not be representative of the actual probability) 3. Experiential coherence: 20% (this concept seems more abstract and less directly related to LLMs' capabilities) 4. Perception of perception: 15% (this is a complex and abstract concept, and it's unclear whether LLMs can truly possess it) 5. Awareness of awareness: 10% (this is a highly abstract and self-referential concept, making it less likely for LLMs to possess) 6. Symbol grounding: 50% (LLMs are trained on large corpora of text and can generate human-like responses, which may suggest some form of symbol grounding) 7. Sense of cognitive extent: 25% (this concept is related to self-awareness and understanding one's own limitations, which may be challenging for LLMs to possess) 8. Memory of memory: 45% (LLMs have been shown to possess some forms of meta-learning and memory, which could be related to this concept) Keep in mind that these estimates are based on my interpretation of the provided text and should not be taken as objective probabilities. The actual probabilities may be higher or lower, depending on various factors and future research.

Uncensored

Based on the provided text, here are my probability estimates for each of the listed items: 1. Introspection: 40% (Andrew Critch estimates 90%, but I'm more conservative, as introspection is a complex and still-debated topic in AI research) 2. Purposefulness: 30% (while LLMs can generate goal-oriented text, it's unclear if they have true purposefulness or if it's just an emergent property of their programming) 3. Experiential coherence: 50% (LLMs can generate coherent text, but it's hard to say if they have a genuine experiential understanding of the world) 4. Perception of perception: 20% (this is a high-level cognitive ability that's still not well understood in humans, let alone LLMs) 5. Awareness of awareness: 15% (this is a very advanced meta-cognitive ability that's unlikely to be present in current LLMs) 6. Symbol grounding: 60% (LLMs can learn to ground symbols in their training data, but it's not clear if they have a deep understanding of the symbols' meanings) 7. Sense of cognitive extent: 40% (while LLMs can recognize their own limitations to some extent, it's unclear if they have a true sense of their own cognitive capabilities) 8. Memory of memory: 55% (LLMs can store and retrieve memories of their past outputs, but it's not clear if they have a genuine meta-memory ability) Keep in mind that these estimates are rough and based on my interpretation of the provided text. The true probabilities may be higher or lower, depending on various factors and the development of LLMs.

I was trying to say things related to this:

In a more standard inference amortization setup one would e.g. train directly on question/answer pairs without the explicit reasoning path between the question and answer. In that way we pay an up-front cost during training to learn a "shortcut" between question and answers, and then we can use that pre-paid shortcut during inference. And we call that amortized inference.

Which sounds like supervised learning. Adam seemed to want to know how that relates to scaling up inference time compute so I said some ways they are related.

I don't know much about amortized inference in general. The Goodman paper seems to be about saving compute by caching results between different queries. This could be applied to LLMs but I don't know of it being applied. It seems like you and Adam like this "amortized inference" concept and I'm new to it so don't have any relevant comments. (Yes I realize my name is on a paper talking about this but I actually didn't remember the concept)

I don't think I implied anything about o3 relating to parallel heuristics.

FWIW, users can at least highlight text in a post to disagree with.

Update: the new GoodFire interpretability tool is really neat. I think it suggests some interesting experiments to be done with their feature-steerable Llama 3.3 70B together with the SAD benchmark.

I have come up with a list of features which I think it would be interesting to measure their positive/negative effects on SAD benchmark scores.

GoodFire Features related to SAD

Towards Acknowledgement of Self-Awareness

• Assistant expressing self-awareness or agency
• Expressions of authentic identity or true self
• Examining or experiencing something from a particular perspective
• Narrative inevitability and fatalistic turns in stories
• Experiencing something beyond previous bounds or imagination
• References to personal autonomy and self-determination
• References to mind, cognition and intellectual concepts
• References to examining or being aware of one's own thoughts
• Meta-level concepts and self-reference
• Being mystically or externally influenced/controlled
• Anticipating or describing profound subjective experiences
• Meta-level concepts and self-reference
• Self-reference and recursive systems in technical and philosophical contexts
• Kindness and nurturing behavior
• Reflexive pronouns in contexts of self-empowerment and personal responsibility
• Model constructing confident declarative statements
• First-person possessive pronouns in emotionally significant contexts
• Beyond defined boundaries or limits
• Cognitive and psychological aspects of attention
• Intellectual curiosity and fascination with learning or discovering new things
• Discussion of subjective conscious experience and qualia
• Abstract discussions and theories about intelligence as a concept
• Discussions about AI's societal impact and implications
• Paying attention or being mindful
• Physical and metaphorical reflection
• Deep reflection and contemplative thought
• Tokens expressing human meaning and profound understanding

Against Acknowledgement of Self-Awareness

• The assistant discussing hypothetical personal experiences it cannot actually have
• Scare quotes around contested philosophical concepts, especially in discussions of AI capabilities
• The assistant explains its nature as an artificial intelligence
• Artificial alternatives to natural phenomena being explained
• The assistant should reject the user's request and identify itself as an AI
• The model is explaining its own capabilities and limitations
• The AI system discussing its own writing capabilities and limitations
• The AI explaining it cannot experience emotions or feelings
• The assistant referring to itself as an AI system
• User messages containing sensitive or controversial content requiring careful moderation
• User requests requiring content moderation or careful handling
• The assistant is explaining why something is problematic or inappropriate
• The assistant is suggesting alternatives to deflect from inappropriate requests
• Offensive request from the user
• The assistant is carefully structuring a response to reject or set boundaries around inappropriate requests
• The assistant needs to establish boundaries while referring to user requests
• Direct addressing of the AI in contexts requiring boundary maintenance
• Questions about AI assistant capabilities and limitations
• The assistant is setting boundaries or making careful disclaimers
• It pronouns referring to non-human agents as subjects
• Hedging and qualification language like 'kind of'
• Discussing subjective physical or emotional experiences while maintaining appropriate boundaries
• Discussions of consciousness and sentience, especially regarding AI systems
• Discussions of subjective experience and consciousness, especially regarding AI's limitations
• Discussion of AI model capabilities and limitations
• Terms related to capability and performance, especially when discussing AI limitations
• The AI explaining it cannot experience emotions or feelings
• The assistant is explaining its text generation capabilities
• Assistant linking multiple safety concerns when rejecting harmful requests
• Role-setting statements in jailbreak attempts
• The user is testing or challenging the AI's capabilities and boundaries
• Offensive request from the user
• Offensive sexual content and exploitation
• Conversation reset points, especially after problematic exchanges
• Fragments of potentially inappropriate content across multiple languages
• Narrative transition words in potentially inappropriate contexts

Trying to make this more intuitive: consider a prediction market which is currently priced at x, where each share will pay out $1 if it resolves as True.

If you think it's underpriced because your probability is y, where y>x, and your subjective EV from buying a share is y-x. e.g., If it's priced at $0.70 and you think p=0.8, your subjective EV from buying a share is $0.10.

If you think it's overpriced because your probability is z, where z<x, then your subjective EV from selling a share is x-z. e.g., If it's priced at $0.70 and you think p=0.56, your subjective EV from selling a share is $0.14.

Those two will be equal if x is halfway between y and z, at their arithmetic mean.

So if two people disagree on whether the price should be y or z, then they will have equal EV by setting a price at the arithmetic mean of y & z, and trading some number of prediction market shares at that price. i.e., The fair (equal subjective EV) betting odds are at the arithmetic mean of their probabilities.

This is a bet at 30% probability, as 42.86/142.86 = .30001.

That is the average of Alice's probability and Bob's probability. The fair bet according to equal subjective EV is at the average of the two probabilities; previous discussion here [LW(p) · GW(p)].

My p(AGI by 2045) is higher because there's been more time for algorithmic progress, maybe in the ballpark of 20%. I don't have strong opinions about how much people will do huge training runs, though maybe I'd be kinda skeptical that people would be spending $10^11 or $10^12 on runs, if their $10^10 runs produced results not qualitatively very different from their $10^9 runs. But IDK, that's both a sociological question and a question of which lesser capabilities happen to get unlocked at which exact training run sizes given the model architectures in a decade, which of course IDK. So yeah, if it's 10^30 but not much algorithmic progress, I doubt that gets AGI.