LessWrong 2.0 Reader

Yeah, getting specific unpause requirements seems high value for convincing people who would not otherwise want a pause, but I can't imagine actually getting it in time in any reasonable way, instead it would need to look like technical specification. "Once we have developed x, y, and z, then it is safe to unpause" kind of thing. Just we need to figure out what the x, y, and z requirements are. Then we can estimate how long it will take to develop x, y, and z, and this will get more refined and accurate as more progress is made, but since the requirements are likely to involve unknown unknowns in theory building, it seems likely that any estimate would be more of a wild guess, and it seems like it would be better to be honest about that rather than saying "yeah, sure, ten years" and then after ten years if the progress hasn't been made saying "whoops, looks like it's going to take a little longer!" As for odds of survival, my personal estimates feel more like 1% chance of some kind of "alignment by default / human in the loop with prosaic scaling" scheme working, as opposed to maybe more like 50% if we took the time to try to get a "aligned before you turn it on" scheme set up, so that would be improving our odds by about 5000%. Though I think you were thinking of adding rather than scaling odds with your 25%, so 49%, but I don't think that's a good habit for thinking about probability. Also I feel hopelessly uncalibrated for this kind of question... I doubt I would trust anyone's estimates, it's part of what makes the situation so spooky. How do you think public acceptance would be of a "pause until we meet target x and you are allowed to help us reach target x as much as you want" as opposed to "pause for some set period of time"?

Thanks a lot for the post! It's really useful to have so many charities and a bit of context in the same place when thinking about my own donations. I found it hard to navigate a post with so many charities, so I put this into a spreadsheet that lets me sort and filter the categories - hopefully this is useful to others too! https://docs.google.com/spreadsheets/d/1WN3uaQYJefV4STPvhXautFy_cllqRENFHJ0Voll5RWA/edit?gid=0#gid=0

(Relevant, although "involving its GPT-4 AI model" is a considerably weaker statement than 'initialized from a GPT-4 checkpoint'.)

I've been enjoying the Sold a Story podcast, which explains how many schools stopped teaching kids to read over the last few decades, replacing phonics with an unscientific theory that taught kids to pretend to read (cargo cult vibes). It features a lot of teachers and education scholars who come face-to-face with evidence that they've been failing kids, and respond in many different ways — from pro-phonics advocacy and outright apology to complete refusal to engage. I especially liked one teacher musing on how disconcerting it was to realize her colleagues were "refuse to engage" types.

The relatable topic and straightforward reporting make the podcast very accessible. It's a good way to share a story with people outside the LessWrong bubble that may get them angry in a way that supports rationalist virtues.

I agree that which terms people use vs taboo is a judgment call, I don't mean to imply that others should clearly see these things the same as me.

"Trauma" meaning psychological as opposed to physical damage goes back to the late 19th century.

I agree that there's a widespread tendency to exaggerate the unpleasantness/harm done by mere words. (But I suggest there's an opposite temptation too, to say that obviously no one can be substantially harmed by mere words, that physical harm is different in kind from mere psychological upset, etc., and that this is also wrong.)

I agree that much of the trans community seems to have embraced what looks to me like a severely hyperbolic view of how much threat trans people are under. (But, usual caveats: it's very common for the situation of a minority group to look and feel much worse from the inside than from the outside, and generally this isn't only a matter of people on the inside being oversensitive, it's also a matter of people on the outside not appreciating how much unpleasantness those on the inside face. So my guess is that that view is less hyperbolic than it looks to me.)

I agree that the term "deadname" is probably popular partly because "using my deadname" has more of an obviously-hostile-move sound than "using my old name" or similar. But if we avoid every term with any spin attached, we'll have to stop calling people liberals (as if no one else cared about freedom) or conservatives (as if their opponents were against preserving valuable things) or Catholics (the word means "universal") or pro-life or pro-choice or, or, or, or. For my part, I avoid some spinny terms but not others, on the basis of gut feeling about how much actual wrongness is baked into them and how easy it is to find other language, which (I don't know how coherently) cashes out as being broadly OK with "liberal" and "conservative", preferring to avoid "pro-life" and "pro-choice" or at least making some snarky remarks about the terms before using them, avoiding the broadest uses of the term "transphobia", etc. And for me "deadname" seems obviously basically OK even though, yes, the term was probably chosen partly for connotations one might take issue with. Your mileage may vary.

Finding the technical solution against trolls isn't that difficult; you basically need invite-only clubs. The things that the members write could be public or private; the important part is that in order to become a member, you need to get some kind of approval first. This can be implemented in various ways: a member needs to send you an invitation link by an e-mail, a moderator needs to approve your account before you can post. A weaker version of this is the way Less Wrong uses: anyone can join, but the new accounts are fragile and can be downvoted out of existence by the existing members, if necessary. (Works well against individual accounts created infrequently. Wouldn't work against hundred people joining at the same time and mass-upvoting each other. But I assume that the moderators have a red button that could simply disable creating new accounts for a while until the chaos is sorted out.)

But when you look at the offline analogy, these things are usually called "old boy networks", and some people think they should be disrupted. Whether you agree with that or not, probably depends on your value judgment about the network versus the people who are trying to get inside. Do you support the rights of new people to join the groups they want to join, or the rights of the existing members to keep out the people they want to keep out? One person's "trolls" are other person's "diverse voices that deserve to be heard".

This is indeed probably a large portion of the solution, and I agree with this sort of solution becoming more necessary in the age of AI.

However, there are also incentives to become more universal than just an old boy's club, so this can't be all of a solution.

I think my key disagreement I have with free speech absolutists is that I think the outcome they are imagining for online spaces without moderation of what people say is essentially a fabricated option, and what actually happens is non-trolls and non-Nazis leave those spaces or go dark, and the outcome is that the trolls and Nazis talk to each other only, not a flowering of science and peace, and the reason why this doesn't happen in the real world is because disruption is way, way more difficult IRL than it is online, but AGI and ASI will lower the cost of disruption by a lot, so free-speech norms become much more negative than now.

I also disagree with moderation being a tradeoff between catching trolls and catching criminals, and with well-funded moderation teams, you can do both quite well.

Maybe there is a more general lesson for the society, unrelated to tech. If you allow people to organize bottom-up, you can get a lot of good things, but you will also get groups dedicated to doing bad things. Western countries seem to optimize for the bottom-up organizations: companies, non-profits, charities, churches, etc. Soviet Union used to optimize for top-down control: everything was controlled by the state, any personal initiative was viewed as suspicious and potentially disruptive. As a result, Soviet Union collapsed economically, but the West got its anti-vaxers and flat-Eathers and everything. During the Cold War, USA was good at pushing the Soviet economical buttons. These days, Russia is good at pushing the Western free speech buttons.

This is why alignment becomes far more important than it is now, because of the fact that it's too easy for a misaligned leader without checks or balances to ruin things, and I'm of the opinion that democracies tolerably work in a pretty narrow range of conditions, but I see the AI future as more dictatorial/plutocratic, due to the onlineification of the real world by AI.

Yes, I think that framing would help. I doubt it would shift public opinion that much, probably not even close to more than 50% in the current epistemic environment. The issue is that we really don't know how hard alignment is. If we could say for sure that pausing for ten years would improve our odds of survival by, say, 25%, then I think a lot of people of the relevant ages (like probably me) would actually accept the framing of a heroic sacrifice.

LessWrong readers concerned with existential risks should certainly care, as frameworks of this nature send dangerously mixed signals about acceptable behavior in high-stakes conflicts. Similar views have grown increasingly influential in global policy contexts. Leaving states uncertain about how their actions will be interpreted or responded to encourages brinkmanship and destabilizes diplomacy and deterrence strategies. Existing crisis management protocols are already fragile, and the last thing you want to do is to add more uncertainty and lack of consistency to the mix, especially when nuclear-armed states are concerned.

Your post is vastly more complete. Nice work!