LessWrong 2.0 Reader

I encourage responses even if they feel like obvious things to you.

In many places.

same here, even 5 months later.

this comment from the time [LW(p) · GW(p)] says the word counter is still there but i can't find it anywhere. (edit: nvm, it's visible on the frontpage hoverover.)

( @habryka [LW · GW] because comments on others' comments don't notify by default. while i'm at it, also your username doesn't show when i enter "@habryka")

Superforecasters can beat domain experts, as shown in Phil Tetlock's work comparing superforecasters to intelligence analysts.

This isn't accurate, see this post [EA · GW]: especially (3a), (3b), and https://docs.google.com/document/d/1ZEEaVP_HVSwyz8VApYJij5RjEiw3mI7d-j6vWAKaGQ8/edit?tab=t.0#heading=h.mma60cenrfmh Goldstein et al (2015)

I'm a 20 year old who perceives myself as the kind of young founder you're probably talking to in this post. And I've noticed a lot of older guys have similar sentiments to you about younger guys and the perspective often annoys me. I do everything I can to learn from other people, but in the context of giving and receiving advice I believe that a lot of information is typically not considered. For example, you talk about a lot of mistakes younger people make that could be easily avoided if they had the older generation's wisdom, but as conveyed by this post, your knowledge/skillset of communicated complex ideas in a way that can be understood by someone with different knowledge/experiences appears limited to me. Additionally, I've written over a thousand documents and am thinking every hour of every day from the perspectives that I value and want to improve in. Someone who doesn't have any context with which to understand me or even what I want can seldom give good advice aimed at observed circumstances in my opinion. The best advise I've received tends to be advice not directed at me because of reasons like this.


I don't believe burnout is real. I have theories on why people think it's real, but I think the phenomenon people label as burnout is more complex than people understand, and advice about burnout is consequently not very helpful. I know you've said you don't typically try to argue/explain yourself around this belief, but if I'm wrong and you have some insight that can only be gained with experience I'm not privy to, I would be thankful if you would correct my false belief. 
 

I agree with most of what I've read in this post, I mainly disagree with some of the perspectives you take.
 

I read through most of the beginning of this post. Then started skimming by my perception of header relevance to the original idea of the post. I'm not really sure what goal you were trying to achieve by branching off into so many different topics in a single post instead of creating separate posts, but I'm still pretty green to the LessWrong community and the norms here, so maybe you can enlighten me. I'm also very biased towards practical texts, so I believe I just am not the target audience for most of this post. I liked the statements about trigger#1 and trigger #2 in the practical section, and its given me some insightful tips I'd never thought of on how to be a better listener. I recently made a note to myself that people say what they mean, so take them literally instead of translating their words into something you already understand. Admittedly, I have not been following my own advice. And this post has served as a valuable reminder for me. I'm really interested in communicated and learning about complex, soulful ideas so maybe you could direct me to some good practical posts on the topic. I've had to navigate this skillset entirely alone, so I'd rather not reinvent the wheel if someone has already publicized their work around this.
 

I had a similar reflection yesterday regarding these inference-time techniques (post-training, unhobbling, whatever you want to call it) being in the very early days. Would it be too much of a stretch to draw parallels here between how such unhobbling methods lead to an explosion of human capabilities over the past ~10000 years? The human DNA has undergone roughly the same number of 'gradient updates' (evolutionary cycles) as our predecessors from a few millenia ago. I see it as having an equivalent amount of training compute. Yet through an efficient use of tools, language, writing, coordination and similar, we have completely outdone what our ancestors were able to do.

There is a difference in that for us, these abilities arose naturally through evolution. We are now manually engineering them into AI systems. I would not be surprised to see a real capability explosion soon (much faster than what we are observing now) - not because of the continued scaling up of pre-training, but because of these post-training enhancements.

Indeed, we know about those posts! Lmk if you have a recommendation for a better textbook-level treatment of any of it (modern papers etc). So far the grey book feels pretty standard in terms of pedagogical quality.

I don't completely disagree but there is also some danger of being systematically misleading.

I think your last 4 bullet points are really quite good & they probably apply to a number of organizations not just the World Bank. I'm inclined to view this as an illustration of organizational failure more than an evaluation of the World Bank. (Assuming of course that the book is accurate)!

I will say tho that my opinion of development economics is quite low…

Re steganography for chain-of-thought: I've been working on a project related to this for a while, looking at whether RL for concise and correct answers might teach models to stenographically encode their CoT for benign reasons. There's an early write-up here: https://ac.felixbinder.net/research/2023/10/27/steganography-eval.html\

Currently, I'm working with two BASIS fellows on actually training models to see if we can elicit steganography this way. I'm definitely happy to chat more/set up a call about this topic

That's interesting. One underlying consideration is that the object-level choices of reasoning steps are relative to a reasoner: differently abled agents need to decompose problems differently, know different things and might benefit from certain ways of thinking in different ways. Therefore, a model plausibly chooses CoT that works well for it "on the object level", without any steganography or other hidden information necessary. If that is true, then we would expect to see models benefit from their own CoT over that of others for basic, non-steganography reasons.

Consider a grade schooler and a grad student thinking out loud. Each benefits from having access to their own CoT, and wouldn't get much from the others for obvious reasons.

I think the questions of whether models actually choose their CoT with respect to their own needs, knowledge and ability is a very interesting one that is closely related to introspection.