This seems right to me, but the discussion of "scaling will plateau" feels like it usually comes bundled with "and the default expectation is that this means LLM-centric-AI will plateau", which seems like the wrong-belief-to-have, to me.

Noting, this doesn't really engage with any of the particular other claims in the previous comment's link, just makes a general assertion. 

Curated. This was one of the more inspiring things I read this year (in a year that had a moderate number of inspiring things!)

I really like how Sarah lays out the problem and desiderata for neutrality in our public/civic institutional spaces.

LessWrong's strength is being a fairly opinionated ”university^[1]” about how to do epistemics, which the rest of the world isn't necessarily bought into. Trying to make LW a civic institution would fail. But, this post has me more excited to revisit "what would be necessary to build good, civic infrastructure" (where "good" requires both "be 'good' in some kind of deep sense," but also "be memetically fit enough to compete with Twitter et all." One solution might be convincing Musk of specific policies rather than building a competitor)

1. ^{^}

   I.e. A gated community with epistemic standards, a process for teaching people, and a process for some of those people going on to do more research. 

You can make a post or shortform discussing it and see what people think. I recommend front loading the main arguments, evidence or takeaways so people can easily get a sense of it - people often bounce off long worldview posts from newcomers

Fwiw I didn't find the post hostile. 

I'm assuming "natural abstraction" is also a scalar property. Reading this paragraph, I refactored the concept in my mind to "some abstractions tend to be cheaper to abstract than others. agents will converge to using cheaper abstractions. Many cheapness properties generalize reasonably well across agents/observation-systems/environments, but, all of those could in theory come apart."

And the Strong NAH would be "cheap-to-abstract-ness will be very punctuated, or something" (i.e. you might expect less of a smooth gradient of cheapnesses across abstractions)

How would you solve the example legal situation you gave?

Thanks, this gave me the context I needed.

Put another way: this post seems like it’s arguing with someone but I’m not sure who.

I think I care a bunch about the subject matter of this post, but something about the way this post is written leaves me feeling confused and ungrounded.

Before reading this post, my background beliefs were:

1. Rationality doesn't (quite) equal Systemized Winning. Or, rather, that focusing on this seems to lead people astray more than helps them.
2. There's probably some laws of cognition to be discovered, about what sort of cognition will have various good properties, in idealized situations.
3. There's probably some messier laws of cognition that apply to humans (but those laws are maybe more complicated).
4. Neither sets of laws necessarily have a simple unifying framework that accomplishes All the Things (although I think the search for simplicity/elegance/all-inclusiveness is probably a productive search, i.e. it tends to yield good stuff along the way. "More elegance" is usually achievable on the margin.
5. There might be heuristics that work moderately well for humans much of the time, which approximate those laws.
   1. there are probably Very Rough heuristics you can tell an average person without lots of dependencies, and somewhat better heuristics you can give to people who are willing to learn lots of subskills.

Given all that... is there anything in-particular I am meant to take from this post? (I have right now only skimmed it, it felt effortful to comb for the novel bits). I can't tell whether the few concrete bits are particularly important, or just illustrative examples.

This is not very practically useful to me but dayumn it is cool

An individual Social Psychology lab (or lose collection of labs) can choose who to let in.

Frontier Lab AI companies can decide who to hire, and what sort of standards they want internally (and maybe, in a lose alliance with other Frontier Lab companies).

The Immoral Mazes outlines some reasons that you might think large institutions are dramatically worse than smaller ones (see: Recursive Middle Manager Hell for a shorter intro, although I don't spell out the part argument about how mazes are sort of "contagious" between large institutions)

But the simpler argument is "the fewer people you have, the easier it is for a few leaders to basically make personal choices based on their goals and values," rather than selection effects resulting in the largest institutions being better modeled as "following incentives" rather than "pursuing goals on purpose." (If an organization didn't follow the incentives, they'd be outcompeted by one that does)

This claim looks like it's implying that research communities can build better-than-median selection pressures but, can they? And if so why have we hypothesized that scientific fields don't?

I'm a bit surprised this is the crux for you. Smaller communities have a lot more control over their gatekeeping because, like, they control it themselves, whereas the larger field's gatekeeping is determined via openended incentives in the broader world that thousands (maybe millions?) of people have influence over. (There's also things you could do in addition to gatekeeping. See Selective, Corrective, Structural: Three Ways of Making Social Systems Work)

(This doesn't mean smaller research communities automatically have good gatekeeping or other mechanisms, but it doesn't feel like a very confusing or mysterious problem on how to do better)

Curated. This was a practically useful post. A lot of the advice here resonated with stuff I've tried and found valuable, so insofar as you were like "well I'm glad this worked for Shoshannah but I dunno if it'd work for me", well, I personally also have found it useful to:

• have a direction more than a goal
• do what I love but always tie it back
• try random things and see what affordances they give me

Yeah, I didn't read this post and come away with "and this is why LessWrong works great", I came away with a crisper model of "here are some reasons LW performs well sometimes", but more importantly "here is an important gear for what LW needs to work great."

Nod. 

One of the things we've had a bunch of internal debate about is "how noticeable should this be at all, by default?" (with opinions ranging from "it should be about as visible as the current green links are" to "it'd be basically fine if it jargon-terms weren't noticeable at all by default."

Another problem is just variety in monitor and/or "your biological eyes." When I do this:

Turn your screen brightness up a bunch and the article looks a bit like Swiss cheese (because the contrast between the white background and the black text increases, the relative contrast between the white background and the gray text decreases).

What happens to me when I turn my macbook brightness to the max is that I stop being able to distinguish the grey and the black (rather than the contrast between white and grey seeming to decrease). I... am a bit surprised you had the opposite experience (I'm on a ~modern M3 macbook. What are you using?)

I will mock up a few options soon and post them here.

For now, here are a couple random options that I'm not currently thrilled with:

1. the words are just black, not particularly noticeable, but use the same little ° that we use for links.

2. Same, but the circle is green:

This feels like you have some way of thinking about responsibility that I'm not sure I'm tracking all the pieces of.

1. Who literally meant the individuals? No one (or, some random alien mind).
2. Who should take actions if someone flags that an unapproved term is wrong? The author, if they want to be involved, and site-admins (or me-in-particular), if they author does not want to be involved.
3. Who should be complained to if this overall system is having bad consequences? Site admins, me-in-particular or habryka-in-particular (Habryka has more final authority, I have more context on this feature. You can start with me and then escalate, or tag both of us, or whatever)
4. Who should have Some Kind of Social Pressure Leveraged At them if reasonable complaints seem to be falling on deaf ears and there are multiple people worried? Also the site admins, and habryka-and-me-in-particular. 

It seems like you want #1 to have a better answer, but I don't really know why.

Part of the uncertainties we're aiming to reduce here are "can we make thinking tools or writing tools that are actually good, instead of bad?" and our experiments so far suggest "maybe". We're also designing with "six months from now" in mind – the current level of capabilities and quality won't be static.

Our theory of "secret sauce" is "most of the corporate Tech World in fact has bad taste in writing, and the LLM fine-tunings and RLHF data is generated by people with bad taste. Getting good output requires both good taste and prompting skill, and you're mostly just not seeing people try."

We've experimented with jailbroken Base Claude which does a decent job of actually having different styles. It's harder to get to work reliably, but, not so much harder that it feels intractable.

The JargonHovers currently use regular Claude, not jailbroken claude. I have guesses of how to eventually get them to write it in something like the author's original style, although it's a harder problem so we haven't tried that hard yet.

it becomes just another purveyor of AI “extruded writing product”.

If it happened here the way it happened on the rest of the internet, (in terms of what the written content was like) I'd agree it'd be straightforwardly bad. 

For things like jargon-hoverovers, the questions IMO are:

• is the explanation accurate?
• is the explanation helpful for explaining complex posts, esp. with many technical terms?
• does the explanation feel like soulless slop that makes you feel ughy the way a lot of the internet is making you feel ughy these days?

If the answer to the first two is "yep", and the third one is "alas, also yep", then I think an ideal state is for the terms to be hidden-by-default but easily accessible for people who are trying to learn effectively, and are willing to put up with somewhat AI-slop-sounding but clear/accurate explanations.

If the answer to the first two is "yep", and the third one is "no, actually is just reads pretty well (maybe even in the author's own style, if they want that)", then IMO there's not really a problem.

I am interested in your actual honest opinion of, say, the glossary I just generated for Unifying Bargaining Notions (1/2) (you'll have to click option-shift-G to enable the glossary on lesswrong.com). That seems like a post where you will probably know most of the terms to judge them on accuracy, while it still being technical enough you can imagine being a person unfamiliar with game theory trying to understand the post, and having a sense of both how useful they'd be and how aesthetically they feel.

My personal take is that they aren't quite as clear as I'd like and not quite as alive-feeling as I'd like, but over the threshold of both that I much rather having them than not having them, esp. if I knew less game theory than I currently do.

The most important thing is "There is a small number of individuals who are paying attention, who you can argue with, and if you don't like what they're doing, I encourage you to write blogposts or comments complaining about it. And if your arguments make sense to me/us, we might change our mind. If they don't make sense, but there seems to be some consensus that the arguments are true, we might lose the Mandate of Heaven or something."

I will personally be using my best judgment to guide my decisionmaking. Habryka is the one actually making final calls about what gets shipped to the site, insofar as I update that we're doing a wrong thing, I'll argue about it."

It happening at all already constitutes “going wrong”.

This particular sort of comment doesn't particularly move me. I'm more likely to be moved by "I predict that if AI used in such and such a way it'll have such and such effects, and those effects are bad." Which I won't necessarily automatically believe, but, I might update on if it's argued well or seems intuitively obvious once it's pointed out.

I'll be generally tracking a lot of potential negative effects and if it seems like it's turning out "the effects were more likely" or "the effects were worse than I thought", I'll try to update swiftly.

Whoops, should be fixed now.

Prolly less than 60%. I think you're overestimating how LLM-pilled the overall LW userbase is (even filtering for people who publish posts). But, my guess is like 25-45% tho.

Somewhat following this up: I think not using LLMs is going to be fairly similar to "not using google." Google results are not automatically true – you have to use your judgment. But, like, it's kinda silly to not use it as part of your search process.

I do recommend perplexity.ai for people who want an easier time checking up on where the AI got some info (it does a search first and provides citations, while packaging the results in a clearer overall explanation than google)

In optimal future Star Trek UI world, giving users control over explanation-style seems good. 

But for near future, my guess is it's not too hard to get a definition that is just pretty all-around good.

This comment caused me to realize: even though generating LaTeX hoverovers involves more technical challenges, I might be able to tell it "if it's a term that gets defined in LaTeX, include an example equation in the hoverover" (or something like that), which might help for some of these.

I'd be pretty into having typography styling settings that auto-detect LM stuff (or, specifically track when users have used any LW-specific LM tools), and flag it with some kind of style difference so it's easy to track at a glance (esp if it could be pretty reliable).

But then, who does carry that responsibility? No-one.

For thie case of this particular feature and ones like it: The LessWrong team. And, in this case, more specifically, me.

I welcome being held accountable for this going wrong in various ways. (I plan to engage more with people who present specific cruxes rather than a generalized "it seems scary", but, this seems very important for a human to be in the loop about, who actually takes responsibility for both locally being good, and longterm consequences)

Reasoning is:

• Currently it takes 40-60 seconds to generate jargon (we've experimented with ways of trimming that down but it's gonna be at least 20 seconds)
• I want authors to actually review the content before it goes live.
• Once authors publish the post, I expect very few of them to go back and edit it more.
• If it happens automagically during draft saving, then by the time you get to "publish post", there's a natural step where you look at the autogenerated jargon, check if it seems reasonable, approve the ones you like and then hit "publish"
• Anything that adds friction to this process I expect to dramatically reduce how often authors bother to engage with it.

To doublecheck/clarify: do you feel strongly (or, weakly) that you don't want autogenerated jargon to exist on your posts for people who click the "opt into non-author-endorsed AI content" for that post? Or simply that you don't personally want to be running into it?

(Oh, hey, you're the one who wrote Please do not use AI to write for you)

Update: 

At the time I posted this, it was at a stage where I had just unlocked the skill, felt it's promise, but it hadn't really paid off yet. I feel like it has paid off in some important ways since then. 

Some of my first round of "Important, Cruxy Predictions" (from 6 months ago, 3 months before I made this post), resolved recently. Many of them were of the form "6 months from now will I have recently used [various rationality techniques I was inventing]", of which "Fluent, Cruxy Predictions" was one. 

I had given relatively low chances of most of them panning out. But, it turns out I've used basically my entire toolkit in the past ~week. 

I'd also made an overall prediction of "a year from now (6 months ago) will the Lightcone team be using various tools." This had essentially been the motivating example behind the "Murphijitsu, and refusing to be satisfied with 'maybe'" section. I noticed I didn't feel optimistic about Lightcone adopting any of the stuff I was generating. Then I made a set of predictions where "in worlds where it was overwhelmingly obvious that these tools were going to end up helping the rest of Lightcone, what other observations would I make along the way?". This was helpful in clarifying further actions I needed to take for it to work (such as "find ways to mention it more often that felt helpful instead of overbearing").

I've more recently started workshopping a "group decisionmaking" UI that helps turn vague strategy disagreements into concrete disagreements. You can see a prototype spreadsheet here.

Yeah, seems good for us to build that today.

I've reverted the part that automatically generates jargon for drafts until we've figured out a better overall solution.

Mmm, that does seem reasonable.

(In addition to this not-seeming-true-across-the-board... also, literally nobody has ever made this claim to me. The entire reason I'm hypothesizing it is because this post suggested it, and it made sense given my model of how my/friends' cognition seems to work. So, IMO the slightly-aggro comment here is just basically wrong? Unless you've specifically seen people claim this?)

Sure, that's a thing that can happen. I'm moderately confident the other thing is a relatively common thing to happen as well. 

This actually was a new way of thinking about it or at least articulating it, for me. Thanks for the link!

I want to register in advance, I have qualms I’d be interested in talking about. (I think they are at least one level more interesting than the obvious ones, and my relationship with them is probably at least one level more interesting than the obvious relational stance)

It does seem like this is pretty different from losing face, and having one word for both of them isn't obviously the best way to carve up concept space.

I think this post introduced an important new angle for this, which is not about face, but instead about "I was actually just trying to solve a different problem than the one you're giving advice about, and it is disruptive to my problem-solving process for you to jam your frame into it. This is bad a) because it's annoying and time-wasting, and b) because there is something delicate about my thought process, and your frame is sort of violating (albeit maybe in a minor, non-traumatizing way)

“Advice can be violating” is the concept-handle I think I will take away.

I like this as a thing to think about, but, Shifgrethor is just a way less easy-to-say or evocative word than "Moloch", "Child of Omelas" or "Dark Forest", alas, so I don't think this particular one will really make it into my repertoir.

I have similar tastes, but, some additional gears:

• I think all day, these days. Even if I'm trying to have interesting, purposeful conversations with people who also want that, it is useful to have sorts of things to talk about that let some parts of my brain relax (while using other parts of my brain I don't use as much)
• on the margin, you can do an intense intellectual conversation, but still make it funnier, or with more opportunity for people to contribute.

I’m pretty interested in this as an exercise of ‘okay yep a bunch of those problems seem real. Can we make conceptual or mechanism-design progress on them in like an afternoon of thought?’

The thing I would bet is "your 'build a lifeboat for some people-like-you to move to somewhere other than EA' plan will work at least a bit, and, one of the important mechanisms for it working will be those effortful posts you wrote."

A thing I wanted to check: were you grokking the general premise that calculus and much of physics haven't been invented yet, and the metaphor here is more about an early stage physicist who has gotten a sense of how "I feel confused here, and I might need to invent [something that will turn out to be calculus]", but, it's at an early enough stage that crisp physics to easily explain it doesn't exist yet?

(If you did get that part, I'm interested in hearing a little bit more about what felt annoying, and if you didn't get that, I'm interested in what sort of things might have helped make the pre-physics/calculus part more clear)

I definitely wouldn't bet money that EA will have evolved into something you can live with (Neither EA nor the threads of rationality that he affeted evolved into things Ben Hoffman could live with)

But, I do think there is something important about the fact that, despite that, it is inaccurate to say "the critiques dropped like a stone through water" (or, what I interpret that poetry to mean, which is something like "basically nobody listened at all". I don't think I misunderstood that part but if I did then I do retract my claim)

fwiw, I think it'd be helpful if this post had the transcript posted as part of the main post body.

I think I actually agree with Lincoln here and think he was saying a different thing than your comment here seems to be oriented around.

I don't think Lincoln's comment had much to do with assuming there was a shadow EA cabal that was aligned with your values. He said "your words are having an impact."

Words having impacts just does actually take time. I updated from stuff Ben Hoffman said, but it did take 3-4 years or something for the update to fully happen (for me in particular), and when I did ~finish updating the amount I was going to update, it wasn't exactly the way Ben Hoffman wanted. In the first 3 years, it's not like I can show Ben Hoffman "I am ready for your approval", or even that I've concretely updated any particular way, because it was a slow messy process and it wasn't like I knew for sure how close to his camp I was going to land.

But, it wouldn't have been true to say "his critiques dropped like a stone through water". (Habryka has said they also affected him, and this seems generally to have actually reveberated a lot). 

I don't know whether or not your critiques have landed, but I think it is too soon to judge.

The setup for the workshop is:

Day 1 deals with constrained Toy Exercises
Day 2 deals with thinking about the big, openended problems of your life (applying skills from Day 1)
Day 3 deals with thinking about your object-level day-to-day work. (applying skills from Day 1 and 2)

The general goal with Feedbackloop-first Rationality is to fractally generate feedback loops that keep you in touch with reality in as many ways as possible (while paying a reasonable overhead price, factored into the total of "spend ~10% of your time on meta")

Some details from The Cognitive Bootcamp Agreement 

I don’t have perfect feedback-loops to tell you if this workshop is working for you. So, there are four 7 different feedback-loop types, with different tradeoffs:

1. Predictions
   1. Guess whether a given strategy will pay off in a concrete way, then see if you were right.
2. Toy Exercises.
   1. They only vaguely resemble your real problems, but you’ll know for sure whether you got the right answer in two hours.
3. Big picture planning
   1. You’ll generate at least one new plan.
      1. You won’t really know if it’s good, but a) you’ll have intuitions about whether it's more or less promising than your previous plan, which are at least some information.
      2. you’ll make predictions about whether it’ll seem worth having thought about in a year.
      3. throughout the planning process, you'll look for minor opportunities to make a prediction about whether how you'll feel about your planning process, and try to compare how it feels compared to previous planmaking you've done
4. Object-level work, in 1-hour blocks
   1. Spend a few timeblocks doing object level work on your second likeliest plan. Each hour, you’ll make conscious choices about how to spend your time and attention. And then, reflect on whether that seemed useful.
      (in addition to crosstraining your skills on the practical object-level, this will help make your second-likeliest plan feel more real)
   2. Nearterm concrete predictions. Again, you can make concrete predictions about how an hour of object-level work will go, and whether pursuing a new strategy will seem to pay off in an intuitive sense.
5. Workshop retention
   1. Make predictions about whether you'll be using various skills from the workshop, 6 months from now.
   2. Iterate on your strategies for retaining things (immediately) to see if you can improve your prediction about how much you'll retain.
   3. 6 months from now, see if you're still using workshop skills, or things clearly descended from them, and see if that feels useful
6. Post Workshop Predictions
   1. Once you return to your day job, start making predictions about whether a given new strategy will pay off in a particular instance, and develop a sense of when they do and don't actually help.
7. [New at the next workshop, not at the one George was at]
   "How many metastrategies are people generating, which go on to help other people at the workshop."
   1. Next time I'm trying a big whiteboard of meta-strategies, where every time someone generates a new strategy which they strongly believe helped them solve a problem, they write it on the whiteboard and put their initials. If other people use that strategy and helps them, the other people also put their initials on it. The person who's strategies go on to help the most people gets a prize.

My own experiences, after having experimented in a sporadic fashion for 6 years and dedicated Purposeful Practice for ~6 months:

First: I basically never feel stuck on impossible-looking problems. (This isn't actually that much evidence because it's very easy to be deluded about your approach being good, but I list it first because it's the one you listed)

As of a couple weeks ago, a bunch of the skills feel like they have clicked together and finally demonstrated the promise of "more than the some of their parts."

Multiple times per day, I successfully ask myself "Is what I'm doing steering me towards the most important part of The Problem? And, ideally, setting myself up to carve the hypothesis space by 50% as fast as possible?" and it is pretty clear:

1. ...that yes there is something else I could be doing that was more important
2. ...that I wouldn't have done it by default without the training
3. ...that various skills from the workshop were pretty important components of how I then go about redirecting my attention to the most important parts of the problem.

The most important general skills that come up a lot are asking:

• "What are my goals?" (generate at least 3 goals)
• "What is hard about this, and how can I deal with that?"
• "Can I come up with a second or third plan?"
• "What are my cruxes for whether to work on this particular approach?"
• "Do those cruxes carve hypothesis space 50%? If not, can I shift my approach so they come closer to 50%, or will take less time to resolve an experiment?"

Things that I don't yet know for sure if they'll pay off but I do successfully do most days now:

• Asking "How could I have thought that faster?"
  • (subjectively, I feel like good strategies come to me fairly automatically without much effort, in pretty much the way Tuning your Cognitive Strategies predicted when I started it 6 years ago, although it is hard to verify that from outside)
• Observing where most of the time went in a given Lightcone team activity, and asking "what would be necessary to cut this from hours/days, down to ~5 minutes of thought and an automated LLM query?"
• Observing places where other Lightcone employees feel cognitively stuck, and often coming with prompts for them that get them unstuck (they self-report as the prompts locally helping them come unstuck, we'll see over time whether that seems to pay off in a major way)
• (Notably, since getting into the groove of this, I've also gotten headaches from "overthinking", and one of my current projects is to learn to more effectively process things in the background and come back to hard things when I've had more time to consolidate stuff. Also generally taking more rest in the middle of the day now that I have a clearer sense of my limits)

I am generally thinking of myself as having the goal of doubling Lightcone's productivity in 12 months (via a combination of these techniques + LLM automation), in a way that should be pretty obvious to the outside world. I don't actually know that I'll succeed at that, but holding that as my intention feels very clarifying and useful. I would be interested in operationalizing bets about it.

(People at Lightcone vary in how bought into that goal. I am currently mostly thinking of it as a thing I'm personally aiming for, and getting people bought into it by demonstrating immediate value is one of the subgoals. 

But, notably, six months ago I made a prediction: "6 months from now, in the past week, will I have suggested to a Lightcone employee that they make multiple plans and pick the best one?", and I only gave it 10% because most of my brilliant-seeming ideas don't actually pan out. But, when the prediction resolved last week, it resolved multiple times in the previous week)

Some clarification of fine points:

Ray thinks this is like, 40-60% of the skill of making better plans (generating them in the first place)

Here's a slightly more nuanced take:

• I think, when I get/invent the "metacognition textbook from the future", there will turn out to be ~5 meta strategies that are simple english prompts that work for most people, that accomplish like 50-85% of the value of "learn the whole-ass skill of metastrategic brainstorming", for most people, who are solving problems that are only moderately hard.
• I predict^[1] if you are solving Really Quite Difficult problems (such as navigating a singularity that might be coming in 3-10 years. Or, probably founding most companies and doing most openended research), I think the full skill of "successfully generate a new metastrategy you haven't thought of before, on the fly, in less-than-a-minute" will be pretty important, and probably 40-60% of the value. Although a large reason for that is that what you'll actually need is [some other particular specific thing], and metastrategic brainstorming is the swiss-army-knife tool that is likely to help you find that thing.

1. ^{^}

   This is me making some kind of bet, I don't have empirics to back this up. If you disagree I'd be happy to operationalize a more specific bet.