LessWrong 2.0 Reader

How does language model introspection work? What mechanisms could be at play? 

'Introspection': When we ask a  language model about its own capabilities, a lot of times this turns out to be a calibrated estimate of the actual capabilities. E.g. models 'know what they know', i.e. can predict whether they know answers to factual questions. Furthermore this estimate gets better when models have access to their previous (question, answer) pairs. 

One simple hypothesis is that a language model simply infers the general level of capability from the previous text. Then we'd expect that more powerful language models are better at this than weaker ones. However, Owain's work on introspection finds evidence to the contrary. This implies there must be 'privileged information'. 

Another possibility is that model simulates itself answering the inner question, and then uses that information  to answer the outer question, similar to latent reasoning. If models really do this two-step computation, then it should be possible to recover a 'bridge entity' at some point in the representations. 

It's plausible that the privileged information is something 'more abstract', not amounting to a full simulation of the language model's own forward pass, but nonetheless carrying useful information about it's own level of capability. 

I've made a new "quick take" explaining it. Please let me know.

P.S. Anybody can purchase any domain for $10, I don't see why domains should be more trustworthy than IP addresses. Anyway, I've added it to my domain now.

http://tokensfortokens.samuelshadrach.com/sender.html

Pay for OpenAI API usage using cryptocurrency.

Currently supported: OpenAI o1 model, USDC on Optimism Rollup on ethereum.

Why use this?

 - You want anonymity

 - You want to use AI for cheaper than the rate OpenAI charges

How to use this?

 - You have to purchase a few dollars of USDC and ETH on Optimism Rollup, and install Metamask browser extension. Then you can visit the website.

More info:

 - o1 by OpenAI is the best AI model in the world as of Jan 2025. It is good for reasoning especially on problems involving math and code. OpenAI is partially owned by Microsoft and is currently valued above $100 billion.

 - Optimism is the second largest rollup on top of ethereum blockchain. Ethereum is the second largest blockchain in terms of market capitalisation. (Bitcoin is the largest. Bitcoin has very limited functionality, and it is difficult to build apps using it.) People use rollups to avoid the large transaction fees charged by blockchains, while still getting similar level of security. As of 2025 users have trusted Optimism with around $7 billion in assets. Optimism is funded by Paradigm, one of top VCs in the cryptocurrency space.

 - USDC is a stablecoin issued by Circle, a registered financial company in the US. A stablecoin is a cryptocurrency token issued by a financial company where the company holds one dollar (or euro etc) in their bank account for every token they issue. This ensures the value of the token remains $1. As of 2025, USDC is the world's second largest stablecoin with $45 billion in reserves.

Do you have any opinion on bupropion vs SSRIs/SNRIs?

I don't know about depression. But anecdotally they seem to be highly effective (even overly effective) against anxiety. They also tend to have undesirable effects like reduced sex drive & inappropriate or reduced motivation -- the latter possibly a downstream effect of reduced anxiety. So the fact that they would help some people but hurt others seems very likely true.

Was this idea or post AI-assisted in any way? If it was, it seems a cut above the usual AI-generated material, and I would be interested to know your methods. 

Is there a good Russian-language introduction to AI alignment?

Excellent! This has always been my guess about depressive realism, but I never did the careful lit review to see if it lined up.

A bit of overconfidence is, curiously, rational. That's if you're also burdened with some unfortunate cognitive limitations that prevent your brain from perfectly telling times when you should err on the side of optimism from times you shouldn't.

And here we all are trying to think with monkey brains.

I've been familiar with this issue for quite some time as it was misleading some relatively smart people in the context of infectious disease research. My initial take was also to view it as an extreme example of over fitting. But I think it's more helpful to think of it as some thing inherent to random walks. In particular the phenomena has very little to do with d>>T & persists even with T>>d. I believe you need multiple independent random walks for PCA to behave as expected.

I didn't watch the whole video (it's 2 hours long), but like I remember the process by which the books were written was described in the book. Mostly that they were written as a result of taped conversations between Feynman and a friend of his. This makes it pretty unlikely the books say anything that Feynman didn't actually say (since the conversations were taped), and also is pretty transparent in the book. It also seems kind of more likely to be true than an autobiography, since the impressions have filtered through at least one other person.