LessWrong 2.0 Reader

Ok. It seems you are arguing that anything that presents like it is conscious implies that it is conscious. You are not arguing whether or not the structure of LLMs can give rise to consciousness.

But then your argument is a social argument. I'm fine with a social definition of consciousness - after all, our actions depend to a large degree on social feedback and morals (about what beings have value) at different times have been very different and thus been socially construed.  

But then why are you making a structural argument about LLMs in the end?

PS. In fact, I commented on the filler symbol paper when Xixidu posted about it and I don't think that's a good comparison.

I agree with all of that. Even being sceptical that LLMs plus search will reach AGI. The lack of constraint satisfaction as the human brain does it could be a real stumbling block.

But LLMs have copied a good bit of our reasoning and therefore our semantic search. So they can do something like constraint satisfaction.

Put the constraints into a query, and the answer will satisfy those constraints. The process used is different than a human brain, but for every problem I can think of, the results are the same.

Now, that's partly because every problem I can think of is one I've already seen solved. But my ability to do truly novel problem solving is rarely used and pretty limitted. So I'm not sure the LLM can't do just as good a job if it had a scaffolded script to explore its knowledge base from a few different angles.

Vanessa Kosoy has a list specifically for her alignment agenda but is probably applicable to agent foundations in general: https://www.alignmentforum.org/posts/fsGEyCYhqs7AWwdCe/learning-theoretic-agenda-reading-list [AF · GW]

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

Altman made a Twitter-edit joke about 'gpt-2 i mean gpt2', so at this point, I think it's just a funny troll-name related to the 'v2 personality' which makes it a successor to the ChatGPT 'v1', presumably, 'personality'. See, it's gptv2 geddit not gpt-2? very funny, everyone lol at troll

If you can code, build a small AI with the fast.ai course. This will (hopefully) be fun while also showing you particular holes in your knowledge to improve, rather than a vague feeling of "learn more". 

If you want to follow along with more technical papers, you need to know the math of machine learning: linear algebra, multivariable calculus, and probability theory. For Agent Foundations work, you'll need more logic and set theory type stuff. 

MIRI has some recommendations for textbooks here. There's also the Study Guide [LW · GW] and this sequence on leveling up [? · GW].

3blue1brown's Youtube has good videos for a lot of this, if that's the medium you like. 

If you like non-standard fiction, some people like Project Lawful.
 

At the end of the day, it's not a super well-defined field that has clear on-ramps into the deeper ends. You just gotta start somewhere, and follow your curiosity. Have fun!

Have you compared this method (finding vectors that change downstream activations as much as possible based on my understanding) with just using random vectors? (I didn't see this in the post, but I might have just missed this.)

In particular, does that yield qualitatively similar results?

Naively, I would expect that this would be qualitatively similar for some norm of random vector. So, I'd be interested in some ablations of the technique.

If random vectors work, that would simplify the story somewhat: you can see salient and qualitatively distinct behaviors via randomly perturbing activations.

(Probably random vectors have to somewhat higher norm to yield qualitatively as large results to vectors which are optimized for changing downstream activations. However, I current don't see a particular a priori (non-empirical) reason to think that there doesn't exist some norm at which the results are similar.)

I want a word that's like "capable" but clearly means the things you have the knowledge or skill to do. I'm clearly not capable of running a hundred miles an hour or catching a bullet in my bare hand. I'm not capable of bench pressing 200lbs either; that's pretty likely in the range of what I could do if I worked out and trained at it for a few years, but right this second I'm not in that kind of shape. In some senses, I'm capable of logging into someone else's LessWrong account- my fingers are physically capable of typing their password- but I don't have the knowledge of what to type.

This comes up in places where a thing is obviously possible to achieve, and wouldn't require, say, the kinds of built up physical changes to my body that lifting 200lbs would require, but I still don't expect to be able to pull it off. If I had someone experienced, who did know how to do it, sitting at my shoulder giving me advice the whole way it would be obviously possible. 

Nevertheless, acting like I'm capable of flying an airplane or resolving a messy divorce or interpreting a blood test if I had to do it right now is going to result in some problems. I'd like to be able to say I'm not able to accomplish that without a bunch of disclaimers that yes, I could learn, or I could follow someone's close directions and manage it, but that's different.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

I'm really excited about Andrew's discovery here. With it, maybe we can get a more complete picture of what these models can do, and how. This feels like the most promising new idea I've seen in a while. I expect it to open up a few new affordances and research directions. Time will tell how reliable and scalable this technique is. I sure hope this technique gets the attention and investigation it (IMO) deserves. 

More technically, his discovery unlocks the possibility of unsupervised capability elicitation, whereby we can automatically discover a subset of "nearby" abilities and behavioral "modes", without the intervention itself "teaching" the model the elicited ability or information.