Engaging First Introductions to AI Risk

I'd also add the question of "When can we expect GAI?" Some people I've talked to about this issue don't think it's possible to get GAI this century.

I'd caution about conflating FOOM with AGI risks, or even with hungry AGI. Hard takeoff is the most obviously risky situation from an AGI safety perspective because it would be impossible to react to an unfriendly AGI without possible world-ending consequences, and is very compelling in story-telling, but a soft takeoff situation is not particularly likely to be safe just because someone /could/ turn off its supplies if no one would be able to tell that such a concern is necessary. An AI that's only several dozen times smarter than humans probably won't crack the protein folding problem, but it's probably smart enough to navigate corporate red tape or manipulate a team of researchers.

g. Why is AGI 'hungry', i.e., desirous of unlimited resources? Why do we expect FAI and UFAI to have galaxy- or universe-wide effects?

Comparative and Absolute Advantage in AI demonstrates the simple lies-to-children version rather easily. The strong case would be the Riemann Hypothesis Catastrophe, but I've not seen any particularly good writeups of that even as it seems to be local jargon.

h. What's a reasonable (as non-squick goes) fun interstellar growth scenario? Why is the future so important?

Just Another Day In Utopia is a little overly gamified and surreal, but generally pleasant. While intended as subtle horror, Iceman's Friendship Is Optimal is about the right mix of 'almost right' that it almost seems like the ponies are added just to explain why someone might resist the Singularity -- it's only on deeper analysis that the value-of-your-values metric falls apart. The full text is too long for this sort of work, but excerpts from the work could be informative enough

j. What would a meta-ethical discourse between Clippy and a human look like?

Mu? Or moo, depending on severity.

k. Why is it hard to give AGI known or predictable goals?

Lost Purposes is a little shallow and not terribly well-researched, but it's a compelling example of subgoal stomp and value drift that's likely to be particularly applicable in an educational environment. And, of course, Magical Categories, which you've already linked, is rather illustrative on its own.

l. What makes value fragile and complex? What are some plausible horror stories if we get things slightly wrong?

The first few paragraphs of Interpersonal Entanglement are very illuminating.

d. What's the most plausible scenario for how an intelligence explosion will initially go? Why?

I don't think anybody knows enough to answer this question with any certainty.

Why should we expect hard takeoff?

It's a bit on the long side, but Why an Intelligence Explosion is Probable might work for this.

Oooh, I'll check out OFI.

What are the main things you find embarrassingly inadequate, and/or newbie-turning-away, about So You Want to Save the World (LW, off-site update)?

Since this rationalist mix tape isn't directed at specialists, my goals for its 'Open problems in FAI' portion are:

1. Impress. Make it clear that FAI is a Real Thing that flesh-and-blood mathematicians are working on. Lots of scary-looking equations probably aren't useful for this, but formal jargon certainly can be, since it makes us look more legit than terms like 'Fun Theory' or 'Friendliness Theory' would initially suggest. Relatedly...

2. Intimidate. Decrease armchair philosophers' confidence that they can shrug off the Five Theses without thinking (and researching) long and hard about them. This is important, because I've set up my Introduction to appeal to people of philosophical, big-picture, let-me-figure-this-out-for-myself mindsets. Impatience, imprecision, and egomania are big failure modes for that demographic, so a lot of good can be done just by showing what patience, precision, and intellectual humility look like in this arena.

3. Ground. Increase people's interest in pragmatic, solutions-oriented responses to the Friendliness problem, by providing a model for how such responses will look. Mitigate the airy and theoretical tendencies that EY's writing can encourage (and that my target audience is already disposed toward), as well as various defeatism/nihilism/relativism attractors.

4. Intrigue. Have a couple of the easier-to-explain open problems tempt the reader into digging deeper.

Given the A,B,C,D goals I outlined above, do you think 1-4 are good objectives for the FAI Open Problems part of the Introduction? And does this goalset change how relevantly useful you think "So You Want to Face the World" is?

OFI looks nice, but is there going to be a Kindle edition?

Amazon doesn't list one, which I suppose could be an artifact of it not being out yet.

To be perfectly honest, mentioning Artificial Intelligence at all might be the wrong way to start a discussion about the risks of superintelligence.

I think that's giving up too much ground. Talking about AI risk while trying to avoid mentioning anything synthetic or artificial or robotic is like talking about asteroid risk while trying to avoid mentioning outer space.

The vast majority of us (myself included) have only very basic ideas of how even modern computers work much less the scientific / mathematical background to really engage with AI theory

But is that necessary for understanding and accepting any of the Five Theses?

not to mention that we're so inundated with inaccurate ideas about AI from fiction that it would probably be easier just to dodge the misconceptions entirely.

Don't a lot of those misconceptions help us? People are primed to be scared that AI is a problem. We then only have to mold that emotion to be less anthropomorphic and reactionary; we don't have to invent emotion out of whole cloth.

An additional concern is that "serious people" who might otherwise be capable of understanding the issue won't want to be associated with a seemingly fantastical and/or nerdy discussion

It's a deliberate feature of my mix that it's (I hope) optimized for philosophical, narrative, abstractive thinkers -- the sort who usually prefer Eliezer's fleshy narratives over Luke's skeletal arguments. Both groups are important, but I prioritized making one for the Eliezer crowd because: (a) I think I have a better grasp on how to appeal to them; (b) they're the sort of crowd that isn't always drawn to, or in conversation with, programmer culture; and (c) Luke's non-academic writings on this topic are already fairly well consolidated and organized. Eliezer's are all over the place, so starting to gather them here gets returns faster.

Short of just starting with That Alien Message

I think That Alien Message is one of the more background-demanding equationless articles Yudkowsky's written. It's directed at combating some very specific and sophisticated mistakes about AI, and taking away the moral requires, I think, enough of a background with the AI project to have some quite specific and complicated (false) expectations already in mind.

I'm not sure even someone who's read all 20 of the posts I listed would be ready yet for That Alien Message, unless by chance they spontaneously asserted a highly specific relevant doubt (e.g., 'I don't think anything could get much smarter than a human, therefore I'm not very concerned about AGI').

I would suggest something along the same lines of replacing AI with a biological superintelligence.

I think the main problem with this is that we think of biological intelligences as moral patients. We think they have rights, are sentient, etc. That adds a lot more problems and complications than we started with. Also, I'm not sure Friendliness is technologically possible for a biological AGI of the sort we're likely to make (for the same reason it may be technologically impossible for a whole-brain emulation).

Throwing in applause lights like having the "wise fools" who unleash the smartpocalypse being a or the AI-equivalent being a would probably make it more palatable to targeted audiences, but might be too far on the dark side for your tastes.

I'm less worried about whether it's dark-side than about whether it's ineffective. Exploiting 'evil robot apocalypse' memes dovetails with the general message we're trying to convey. Exploiting 'GMOs and big companies are intrinsically evil' actively contradicts a lot of the message we want to convey. E.g., it capitalizes on 'don't tamper with Nature'. One of the most important take-aways from LessWrong is that if we don't take control over our own destiny -- if we don't play God, in a safe and responsible way -- then the future will be valueless. That's in direct tension with trying to squick people out with a gross bioengineered intelligence that will be seen as Bad because it's a human intervention, and not because it's a mathematically unrigorous or meta-ethically unsophisticated intervention.

Which isn't to say that we shouldn't appeal to that audience. If they're especially likely to misunderstand the problem, that might make it all the more valuable to alter their world-view. But it will only be useful to tap into their 'don't play God' mentality if we, in the same cutting strike, demonstrate the foolishness of that stance.

I think a lot of smart high schoolers could read the sequence I provided above. If they aren't exceptional enough to do so, they're probably better off starting with CFAR-style stuff rather than MIRI-style stuff anyway, since I'd expect reading comprehension skills to correlate with argument evaluation skills.

What can you do to help? In order of priority, I think the top choices for non-specialists are:

1. Get more money funneled into MIRI and FHI.
2. Memetically propagate informed worry about AI risk. Find ways to make the idea stick.
3. Improve your own and others' domain-general rationality.
4. Try to acquire math knowledge and skills that might be in demand by FAI researchers over the next few decades.

If this sequence mix does its job, 2 is simple enough — tell people to begin by sharing the list. (Or other targeted articles and lists, depending on target audience.)

3 is a relatively easy sell, and the primary way I expect to contribute.

4 is quite difficult and risky at this stage.

1 is hard to optimize at the meta level because good signaling is hard. Our default method for beginning to combat high-level biases — write a Sequence focused on this specific issue to bring it to people's attentions — is unusually tricky to pull off here. Something off-site and independent — a petition, signed by lots of smart people, telling everyone that AI risk is important? an impassioned personal letter by a third party, briefly and shibbolethlessly laying out the Effective Altruism case for MIRI? — might be more effective.

I'll have to think about how best to do this. Part I is a bit of a grab-bag, trying to rapidly reset our assumptions about intelligence and agenthood before Part II begins supplying the Official Arguments and Definitions. (Replacing the current frame with one that makes more sense is also an option.)

This is some of why I asked about inferential gaps; how would y'all walk through or supplement some of the transitions to a reader? Knowing what you'd do differently, or additionally, will also tell me things about what you see as the function of each component post.