Roadmap: Plan of Action to Prevent Human Extinction Risks

I'm not sure "Improved translation/global English fluency" is an unalloyed good... could lead to a harmful global monoculture (putting all our eggs in one basket culturally). Feel free to reduce my award count by one :)

Also helping Westerners chill out could leave them less prepared to deal with belligerent Middle Easterners.

Embryo selection is a third alternative, the progress of which is more susceptible to policy decisions than gene therapy, and the effects of which are more immediate than selective breeding. I recommend Shulman & Bostrom (2014) for further information.

If we have 200-300 years before well proved catastrophe, this technic may work.

If you're talking about significant population changes in IQ, then I agree, it would take a while to make that happen with only reproduction incentives. However, I was thinking more along the lines of just having a few thousands or tens of thousands of >145 IQ people more than we would otherwise, and that could be achieved in as little as one or two generations (< 50 years) if the program were successful enough.

Now for a slightly crazier idea. (Again, I'm just thinking out loud.) You take the children and send them to be unschooled by middle-class foster families, both to save money, and to make sure they are not getting the intellectual stimulation they need from their environment alone, which they might if you sent them to upper-class private schools, for example. But, you make sure they have Internet access, and you gradually introduce them to appropriately challenging MOOCs on math and philosophy specially made for them, designed to teach them a) the ethics of why they should want to save the world (think some of Nate's posts) and b) the skills they would need to do it (e.g., they should be up to speed on what MIRI recommends for aspiring AI researchers before they graduate high school).

The point of separating them from other smart people is that smart people tend to be mostly interested in money, power, status, etc., and that could spread to them if they are immersed in it. If their focus growing up is simply to find intellectual stimulation, then they would be essentially blank slates and when they're introduced to problems that are very challenging and stimulating, have other smart people working on them, and are really, really* important, they might be more likely to take them seriously.

*Please see my clarification below.

Yup, that works. Thanks.

In plans: 1. Is not "voluntary or forced devolution" the same as "ludism" and "relinquishment of dangerous science" which is already in the plan?

I was thinking more along the lines of restricting the chance for divergence in the human species. I guess I am not really sure what is it that you are trying to preserve. What do you take to be humanness? Technological advances may allow us to alter ourselves so substantially that we become post-human or no longer human. This could be for example from cybernetics or genetic engineering. "ludism" and "relinquishment of dangerous science" is a way to restrict what technologies we use, but note that we are still capable of using and creating these technologies. Devolution, perhaps there is a better word for it, would be something like the dumbing down of all or most humans so that they are no longer capable of using or creating the technologies that could make them less purely human.

I think that "some movement or event that will cause a paradigmatic change so that humanity becomes more existentially-risk aware" is basically the same idea as "smaller catastrophe could help unite humanity (pandemic, small asteroid, local nuclear war)", but your wording is excellent.

Yes you are right. I guess I was more implying man-made catastrophes which are created in order to cause a paradigmatic change rather than natural ones.

I still don't know how we could fix all the world system problems which are listed in your link without having control of most of the world which returns us to plan A1.

I'm not sure either. I would think you could do it by changing the way that politics works so that the policies implemented actually have empirical backing based on what we know about systems. Perhaps, this is just AI and improved computational modelling. This idea of needing control of the world seems extremely dangerous to me. Although, I suppose a top-down approach could solve the problems. I think that you should also think about what a good bottom-up approach would be. How do we make local communities and societies more resilient, economical and capable of facing potential X-risks.

In survive the catastrophe I would add two extra boxes:

• Limit the impact of catastrophe by implementing measures to slow the growth and areas impacted by a catastrophe. For example, with pandemics you could: improve the capacity for rapid production of vaccines in response to emerging threats or create or grow stockpiles of important medical countermeasure

• Increase time available for preparation by improving monitoring and early detection technologies. For example, with pandemics you could: supporting general research on the magnitude of biosecurity risks and opportunities to reduce them and improving and connect disease surveillance systems so that novel threats can be detected and responded to more quickly

I could send money to a charity of your choice.

Send it to one of the charities here.

Done

Legally enforced veganism? (But grain spoils. It is also often stored in buildings designed specifically for that purpose, and once they get infected...) All in all, I was just trying to think of a hard-to-contain, hard-to-notice, hard-to-prevent x-risk; those already discussed seem more... straightforward, perhaps. I am sure there are other examples of systemic failure harder to fight with international treaties than nuclear war.

Seems like the definition of "severe" is an issue here. Maybe I should have used "incredibly severe"?

Yes, reinsurance markets deal in large insured risks, but they do not target the incredibly large humanitarian risks that are more informative to us. See reinsurance deals here for reference: http://www.artemis.bm/deal_directory/

I don't think they (or your proposed prediction markets) would be helpful in estimating the probabilities of extinction events.

Care to explain your reasoning? For example, if the market indicated that the chance of a pandemic killing 50% of the population is 1,000x greater than the likelihood of a nuclear war of any kind, wouldn't a forecaster find this at least a little useful?

n4. Xrisk is not always risky to the developer: the Atomic bomb creators did not feel the effects of their weapons personally. in this way an x-risk can be catastrophic but not personally of consequence. I was suggesting something to overcome this predicament. Where one goverment might commission a scientist to design a bioweapon to be released on another country and offer to defend/immunise the creators.

Its a commons-binding contract that discourages the individual. it only takes one. which is the challenge of humans taking x-risky actions, it is not always of direct consequence to the individual (or it may not feel that way to the individual)

n5. radiation from space that we are not currently accustomed to or protected from would be an x-risk to the biological population.

n7. as a way to increase the number of colonies nearby, creating another moon that is close to earth might be easier and cheaper and more viable than mars. Although I might be completely wrong and mars might be easier. really depends on how much inter-colony travel there will likely be.

n1. I meant - buy up supply of nuclear material, push the price above viable for a long time, therefore discouraging the development of surrounding technology.

I didn't really know what x-risks you were talking about; which is why a map of x-risks would have helped me.

(I suggest that maybe you want to offer to take free suggestions before you pay people - at least that might save you some dollars)

I was really saying - save your money till after people shoot off some low-hanging fruit ideas.

I would argue that the current barrier of "it costs lots of money to do bio-hacking right" is a terrible one to hide behind because of how easy it is to overcome it; or do biohacking less-right and less-safely. i.e. without safe containment areas.

Perhaps funding things like clean-rooms with negative pressure and leaving the rest up to whoever is using the lab-space.

Okay. Maybe bolden up the outlines or change the colours so they appear more distinct, or make some lines into arrows?

Simulation is an X-Risk that we stagnate our universal drive to growth and live in a simulation for the rest of our lives and extinguish ourselves from existence.

Bio-Hack is an X-Risk because if done wrong you would encourage all these small bio-tech interests and end up with someone doing it unsafely.

The failure of mini biohack groups could probably be classified as controlled regression->small catastrophy. Similar to the small nuclear catastrophies of current history and their ability to discourage any future risk taking behaviour in the area.

The advantage of common bio-hack groups is less reliance on the existing big businesses to save us with vaccines etc.

Indeed the suggestion of "Invite the full population to contribute to solving the problem" might be a better description.

New suggestion: "lower the barriers of entry into the field of assistance in X-risk". Easy explanation of the X-risks; easier availability of resources to attempt solutions. Assuming your main x-risks are 1. biotech; 2. nanotech; 3. nuclear, 4. climate change and 5. UFAI)

1. Provide Biotech upskill (education, some kind of OpenBio foundation) and Bioresources for anyone interested in the area (starter kit, smaller cheaper lab-on-a-chip, simple biotech "at-home" experiments like GFP insertion).
2. Teach the risks of molecular manufacturing before teaching people how to do it. (or restructure the education program to make sure it is included)
3. Teach 4th gen nuclear technologies to everyone. Implement small scale nuclear models. (i.e. tiny scale - not sure if it would work) to help people understand the possibility of a teeny nuclear failure but scaled up to large. (if it is possible to make a tiny scale nuclear reactor is beyond my knowledge)
4. empower the public with technology or understanding to reverse pollution. i.e. solar + batteries + electric cars; plant-trees initiatives (or oxygen bio-filters), carbon capture programs, educate and make possible small-scale sustainability (or close to it). Teach people 3D printing; Maker (fixer) mindset; reuse/upcycle; Reward disposable packaging more than non-disposable.
5. UFAI free education in the area, starter packages in programming. (I have no idea other than it seems to be being worked on by smart people)

New suggestion: Teach x-risk from 5 years old upwards. So that the next generation of humans understand that when they play with these kinds of powerful forces - they risk a whole lot more than they realise. (hopefully before having an x-risky accident to explicitly warn people about things)

New idea - I don't think you covered: lock down all risk areas beneath piles of bureaucracy, paperwork, safety requirements and bullshit. No one gets to work on nuclear, no one gets to work on biotech without ridiculous safety standards, no one gets to create pollution without being arrested and charged, no one gets to code learning machines without strict supervision.

Reply in a PM.

Linking to the appropriate section of the explanation text would probably be better than linking to primary sources directly once that exists (which in turn would link out to primary sources).

Compressing to "rationality" is reasonable, though most readers would not understand at a glance. If you're trying to keep it very streamlined just having a this as a lot of pointers makes sense, though perhaps alongside rationality it'd be good to have a pointer that's more clearly directed at "make wanting to fix the future a thing which is widely accepted", rather than rationality's normal meanings as being effective. I'd also think it more appropriate for the A3 stream than A2, for what I have in mind at least.

I'd think creating world saviors from scratch would not be a viable option with some AI timelines, but getting good at picking up promising people in/leaving uni who have the right ethical streak and putting them in a network full of the memes of EA/X-risk reduction could plausibly give a turnaround from "person who is smart and will probably get some good job in some mildly evil corporation" to "person dedicated to trying to fix major problems/person in an earning to give career to fund interventions/person working towards top jobs to gain leverage to fix things from the inside" on the order of months, with an acceptable rate of success (even a few % changing life trajectory would be more than enough to pay back the investment of running that network in terms of x-risk reduction).

Perhaps classifying things in terms of what should be the focus right now verses things that need more steps before they become viable projects would be more useful than attempting to give dates in general? Vague dates are better, but thinking more I'm not sure if even giving wide ranges really solves the problem, our ability to forecast several very important things is highly limited. I'm not sure about a good set of labels for this though, but perhaps something like:

• Immediate (aka: things which we could/are just working on right now)
• Near future (single digit years? things which need some foundations, but are within sight)
• Mid-term (unsure when we'll get there, may vary significantly from topic to topic, can get a rough idea of what will likely need doing but we can't get into the details until previous layers of tech/organization are ready)
• Distant (getting much harder to forecast, major goals and projects which need large unpredictable tech advances and/or significant social changes before they're accessible)
• Outcomes (ways things could end up, when one or more of the previous projects goes through). Again, I'm not sure about these words, but using things which point more to the number of steps and difficulty of forecasting seems like a thing to explore.

And thank you. I tend to take downvotes as very strong negative reinforcement, it helps that you find my post somewhat useful.

One person downvoted it, which means it could be anything from "I don't like spelling corrections" to "I disagree about not giving dates".

In general, if only one person downvotes, it is best not to ask. I don't see anything worth downvoting in your post myself, although I wouldn't upvote it, because it reads to me more like an attempt at compressing many applause lights into one comment without paying attention to any one than an attempt at genuine suggestions for improvement. (It's a little -too- Less Wrongian.)

The lesson of the potato famine was that crops should be more, rather than less, diverse. The potatoes that were in cultivation didn't have enough genetic variation which is why the the disease had such a huge impact. But if it's true of crops... then it's also true of people. People should be more genetically diverse. This way a new pathogen can't kill all of us. Although I have no idea how you'd practically ensure greater human diversity!!?? History might refer to you as the opposite of Hitler.

Regarding the danger of AI... if greater diversity is better for crops and humans... then it's also better for robots as well. We'll give more resources to the most beneficial robots. Evil robots won't have a leg to stand on.

And war can be eliminated by tax choice.