LessWrong 2.0 Reader

I'm a 20 year old who perceives myself as the kind of young founder you're probably talking to in this post. And I've noticed a lot of older guys have similar sentiments to you about younger guys and the perspective often annoys me. I do everything I can to learn from other people, but in the context of giving and receiving advice I believe that a lot of information is typically not considered. For example, you talk about a lot of mistakes younger people make that could be easily avoided if they had the older generation's wisdom, but as conveyed by this post, your knowledge/skillset of communicated complex ideas in a way that can be understood by someone with different knowledge/experiences appears limited to me. Additionally, I've written over a thousand documents and am thinking every hour of every day from the perspectives that I value and want to improve in. Someone who doesn't have any context with which to understand me or even what I want can seldom give good advice aimed at observed circumstances in my opinion. The best advise I've received tends to be advice not directed at me because of reasons like this.


I don't believe burnout is real. I have theories on why people think it's real, but I think the phenomenon people label as burnout is more complex than people understand, and advice about burnout is consequently not very helpful. I know you've said you don't typically try to argue/explain yourself around this belief, but if I'm wrong and you have some insight that can only be gained with experience I'm not privy to, I would be thankful if you would correct my false belief. 
 

I agree with most of what I've read in this post, I mainly disagree with some of the perspectives you take.
 

I read through most of the beginning of this post. Then started skimming by my perception of header relevance to the original idea of the post. I'm not really sure what goal you were trying to achieve by branching off into so many different topics in a single post instead of creating separate posts, but I'm still pretty green to the LessWrong community and the norms here, so maybe you can enlighten me. I'm also very biased towards practical texts, so I believe I just am not the target audience for most of this post. I liked the statements about trigger#1 and trigger #2 in the practical section, and its given me some insightful tips I'd never thought of on how to be a better listener. I recently made a note to myself that people say what they mean, so take them literally instead of translating their words into something you already understand. Admittedly, I have not been following my own advice. And this post has served as a valuable reminder for me. I'm really interested in communicated and learning about complex, soulful ideas so maybe you could direct me to some good practical posts on the topic. I've had to navigate this skillset entirely alone, so I'd rather not reinvent the wheel if someone has already publicized their work around this.
 

I had a similar reflection yesterday regarding these inference-time techniques (post-training, unhobbling, whatever you want to call it) being in the very early days. Would it be too much of a stretch to draw parallels here between how such unhobbling methods lead to an explosion of human capabilities over the past ~10000 years? The human DNA has undergone roughly the same number of 'gradient updates' (evolutionary cycles) as our predecessors from a few millenia ago. I see it as having an equivalent amount of training compute. Yet through an efficient use of tools, language, writing, coordination and similar, we have completely outdone what our ancestors were able to do.

There is a difference in that for us, these abilities arose naturally through evolution. We are now manually engineering them into AI systems. I would not be surprised to see a real capability explosion soon (much faster than what we are observing now) - not because of the continued scaling up of pre-training, but because of these post-training enhancements.

Indeed, we know about those posts! Lmk if you have a recommendation for a better textbook-level treatment of any of it (modern papers etc). So far the grey book feels pretty standard in terms of pedagogical quality.

I don't completely disagree but there is also some danger of being systematically misleading.

I think your last 4 bullet points are really quite good & they probably apply to a number of organizations not just the World Bank. I'm inclined to view this as an illustration of organizational failure more than an evaluation of the World Bank. (Assuming of course that the book is accurate)!

I will say tho that my opinion of development economics is quite low…

Re steganography for chain-of-thought: I've been working on a project related to this for a while, looking at whether RL for concise and correct answers might teach models to stenographically encode their CoT for benign reasons. There's an early write-up here: https://ac.felixbinder.net/research/2023/10/27/steganography-eval.html\

Currently, I'm working with two BASIS fellows on actually training models to see if we can elicit steganography this way. I'm definitely happy to chat more/set up a call about this topic

That's interesting. One underlying consideration is that the object-level choices of reasoning steps are relative to a reasoner: differently abled agents need to decompose problems differently, know different things and might benefit from certain ways of thinking in different ways. Therefore, a model plausibly chooses CoT that works well for it "on the object level", without any steganography or other hidden information necessary. If that is true, then we would expect to see models benefit from their own CoT over that of others for basic, non-steganography reasons.

Consider a grade schooler and a grad student thinking out loud. Each benefits from having access to their own CoT, and wouldn't get much from the others for obvious reasons.

I think the questions of whether models actually choose their CoT with respect to their own needs, knowledge and ability is a very interesting one that is closely related to introspection. 

Nobody has a deal where they'll pay you to not take an offer from an AI lab right? I realize that would be weird incentives, just curious.

Mod note: I've put this on Personal rather than Frontapge. I imagine the content of these talks will be frontpage content, but event announcements in general are not.

This came from a Facebook thread where I argued that many of the main ways AI was described as failing fall into few categories (John disagreed). 

I appreciated this list, but they strike me as fitting into a few clusters.
 

...I would flag that much of that is unsurprising to me, and I think categorization can be pretty fine.

In order:

1) If an agent is unwittingly deceptive in ways that are clearly catastrophic, and that could be understood by a regular person, I'd probably put that under the "naive" or "idiot savant" category. As in, it has severe gaps in its abilities that a human or reasonable agent wouldn't. If the issue is that all reasonable agents wouldn't catch the downsides of a certain plan, I'd probably put that under the "we made a pretty good bet given the intelligence that we had" category.

2) I think that "What Failure Looks Like" is less Accident risk, more "Systemic" risk. I'm also just really unsure what to think about this story. It feels to me like it's a situation where actors are just not able to regulate externalities or similar.

3) The "fusion power generator scenario" seems like just a bad analyst to me. A lot of the job of an analyst is to flag important considerations. This seems like a pretty basic ask. For this itself to be the catastrophic part, I think we'd have to be seriously bad at this. ("i.e. Idiot Savant")

4) STEM-AGI -> I'd also put this in the naive or "idiot savant" category.

5) "that plan totally fails to align more-powerful next-gen AGI at all" -> This seems orthogonal to "categorizing the types of unalignment". This describes how incentives would create an unaligned agent, not what the specific alignment problem is. I do think it would be good to have better terminology here, but would probably consider it a bit adjacent to the specific topic of "AI alignment" - more like "AI alignment strategy/policy" or something.

6) "AGIs act much like a colonizing civilization" -> This sounds like either unalignment has already happened, or humans just gave AIs their own power+rights for some reason. I agree that's bad, but it seems like a different issue than what I think of as the alignment problem. More like, "Yea, if unaligned AIs have a lot of power and agency and different goals, that would be suboptimal"

7) "but at some point a particular subagent starts self-improving, goes supercritical, and takes over the rest of the system overnight." -> This sounds like a traditional mesa-agent failure. I expect a lot of "alignment" with a system made of a bunch of subcomponents is "making sure no subcomponents do anything terrible." Also, still leaves open the specific way this subsystem becomes/is unaligned.

8 ) "using an LLM to simulate a whole society. " -> Sorry, I don't quite follow this one.

Personally, I like the focus "scheming" has. At the same time, I imagine there are another 5 to 20 clean concerns we should also focus on (some of which have been getting attention).

While I realize there's a lot we can't predict, I think we could do a much better just making lists of different risk factors and allocating research amongst them.