LessWrong 2.0 Reader

To doublecheck/clarify: do you feel strongly (or, weakly) that you don't want autogenerated jargon to exist on your posts for people who click the "opt into non-author-endorsed AI content" for that post? Or simply that you don't personally want to be running into it?

(Oh, hey, you're the one who wrote Please do not use AI to write for you [LW · GW])

Your central argument seems to be a metaphor: We caused the Holocene extinction of animals, so godlike AI will kill us.

The problem with metaphorical arguments is that they can be reversed. As humans have become more intelligent, we've started to value animals, created zoos, natural reserves and now even work on the resurrection of extinct animals like mammoths. See more examples of such reversal by Gwern https://gwern.net/modus

Presenting weak arguments is evidence that there are no strong arguments, and this is obvious to outside readers.

The main problem is that we can't predict what superintelligent AI will do, and thus we can't 100 percent prove that it will necessarily kill us. But we shouldn't have to.

Instead, we should show that superintelligence will disempower us and that it may want to kill us for some reasons.

I do think there's something to that idea - physical injury and pain is a very universal and visible experience, whereas mental illness is difficult to parse for those who've never experienced it. I also think there's some sense in which 'treatment' and 'cure' are treated differently for mental and physical illness.

A doctor wouldn't just prescribe painkillers for a broken arm and call it a day because your symptoms have been dealt with; they'd want to actually fix the problem. Depression, on the other hand, doctors seem perfectly fine with merely mitigating the symptoms. Perhaps because that's all they're confident they can do?

BTW, memantine is weak (but legal) analog of ketamine and helped me to cure my depression. 

Update: 

At the time I posted this, it was at a stage where I had just unlocked the skill, felt it's promise, but it hadn't really paid off yet. I feel like it has paid off in some important ways since then. 

Some of my first round of "Important, Cruxy Predictions" (from 6 months ago, 3 months before I made this post), resolved recently. Many of them were of the form "6 months from now will I have recently used [various rationality techniques I was inventing]", of which "Fluent, Cruxy Predictions" was one. 

I had given relatively low chances of most of them panning out. But, it turns out I've used basically my entire toolkit in the past ~week. 

I'd also made an overall prediction of "a year from now (6 months ago) will the Lightcone team be using various tools." This had essentially been the motivating example behind the "Murphijitsu, and refusing to be satisfied with 'maybe'" section. I noticed I didn't feel optimistic about Lightcone adopting any of the stuff I was generating. Then I made a set of predictions where "in worlds where it was overwhelmingly obvious that these tools were going to end up helping the rest of Lightcone, what other observations would I make along the way?". This was helpful in clarifying further actions I needed to take for it to work (such as "find ways to mention it more often that felt helpful instead of overbearing").

I've more recently started workshopping a "group decisionmaking" UI that helps turn vague strategy disagreements into concrete disagreements. You can see a prototype spreadsheet here.

I didn't, but I'm not surprised your sister had that experience. It's a loud, repetitive noise going off next to your ear. My clinic offered me earplugs, which I didn't need, but perhaps your sister could have used?

I like the section where you list out specific things you think people should do. (One objection I sometimes hear is something like "I know that [evals/RSPs/if-then plans/misc] are not sufficient, but I just don't really know what else there is to do. It feels like you either have to commit to something tangible that doesn't solve the whole problem or you just get lost in a depressed doom spiral.")

I think your section on suggestions could be stronger by presenting more ambitious/impactful stories of comms/advocacy. I think there's something tricky about a document that has the vibe "this is the most important issue in the world and pretty much everyone else is approaching it the wrong way" and then pivots to "and the right way to approach it is to post on Twitter and talk to your friends." 

My guess is that you prioritized listing things that were relatively low friction and accessible. (And tbc I do think that the world would be in better shape if more people were sharing their views and contributing to the broad discourse.)

But I think when you're talking to high-context AIS people who are willing to devote their entire career to work on AI Safety, they'll be interested in more ambitious/sexy ways of contributing. 

Put differently: Should I really quit my job at [fancy company or high-status technical safety group] to Tweet about my takes, talk to my family/friends, and maybe make some website? Or are there other paths I could pursue?

As I wrote here [LW(p) · GW(p)], I think we have some of those ambitious/sexy/high-impact role models that could be used to make this pitch stronger, more ambitious, and more inspiring. EG:

One possible critique is that their suggestions are not particularly ambitious. This is likely because they're writing for a broader audience (people who haven't been deeply engaged in AI safety).

For people who have been deeply engaged in AI safety, I think the natural steelman here is "focus on helping the public/government better understand the AI risk situation." 

There are at least some impactful and high-status examples of this (e.g., Hinton, Bengio, Hendrycks). I think in the last few years, for instance, most people would agree that Hinton/Bengio/Hendrycks have had far more impact in their communications/outreach/policy work than their technical research work.

And it's not just the famous people– I can think of ~10 junior or mid-career people who left technical research in the last year to help policymakers better understand AI progress and AI risk, and I think their work is likely far more impactful than if they had stayed in technical research. (And I'm even excluding people who are working on evals/if-then plans: like, I'm focusing on people who see their primary purpose as helping the public or policymakers develop "situational awareness", develop stronger models of AI progress and AI risk, understand the conceptual arguments for misalignment risk, etc.)

I'd also be curious to hear what your thoughts are on people joining government organizations (like the US AI Safety Institute, UK AI Safety Institute, Horizon Fellowship, etc.) Most of your suggestions seem to involve contributing from outside government, and I'd be curious to hear more about your suggestions for people who are either working in government or open to working in government.

I think (not sure!) the damage from people/orgs/states going "wow, AI is powerful, I will try to build some" is larger than the upside of people/orgs/states going "wow, AI is powerful, I should be scared of it". It only takes one strong enough one of the former to kill everyone, and the latter is gonna have a very hard time stopping all of them.

By not informing the public that AI is indeed powerful, awareness of that fact is disproportionately allocated to people who will choose to think hard about it on their own, and thus that knowledge is more likely to be in reasonabler hands (for example they'd also be more likely to think "hmm maybe I shouldn't build unaligned powerful AI").

The same goes for cyborg tools, as well as general insights about AI: we should want them to be differentially accessible to alignment people than the general public.

In fact, my biggest criticism of OpenAI is not that they built GPTs, but that they productized it, made it widely available, and created a giant public frenzy about LLMs. I think we'd have more time to solve alignment if they kept it internally and the public wasn't thinking about AI nearly as much.

Oh I missed the quotations; you're right

Yeah, seems good for us to build that today.