I'm way more scared about the electrode-produced smiley faces for eternity and the rest. That's way, way worse than dying.

FWIW, it seems kinda weird to me that such an AI would keep you alive... if you had a "smile-maximiser" AI, wouldn't it be indifferent to humans being braindead, as long as it's able to keep them smiling?

I'd like to have Paul Christiano's view that the "s-risk-risk" is 1/100 and that AGI is 30 years off

I think Paul's view is along the lines of "1% chance of some non-insignificant amount of suffering being intentionally created", not a 1% chance of this type of scenario.^[1]

Could AGI arrive tomorrow in its present state?

I guess. But we'd need to come up with some AI model tomorrow, and this model suddenly becomes agentive and rapidly grows in power, and this model is designed with a utility function that values keeping humans alive but does not value humans flourishing... and even then, there'd likely be better ways to e.g. maximise the number of smiles in the universe, by using artificially created minds.

Eliezer has written a bit about this, but I think he considers it a mostly solved problem. 

What can I do as a 30 year old from Portugal with no STEM knowledge? Start learning math and work on alignment from home?

Probably get treatment for the anxiety and try to stop thinking about scenarios that are very unlikely, albeit salient in your mind. (I know, speaking from experience, that it's hard to do so!)

1. ^{^}

    I did, coincidentally, cold e-mail Paul a while ago to try to get his model on this type of stuff & got the following response:

   "I think these scenarios are plausible but not particularly likely. I don't think that cryonics makes a huge difference to your personal probabilities, but I could imagine it increasing them a tiny bit. If you cared about suffering-maximizing outcomes a thousand times as much as extinction, then I think it would be plausible for considerations along these lines to tip the balance against cryonics (and if you cared a million times more I would expect them to dominate). I think these risks are larger if you are less scope sensitive since the main protection is the small expected fraction of resources controlled by actors who are inclined to make such threats."

   TBH it's difficult to infer a particular probability estimate for one's individual probability without cryonics or voluntary uploading here; it's not completely clear just how bad a scenario would have to be (for a typical biological human) in order to fall within the class of scenarios described as 'plausible but not particularly likely'.

I'm way more scared about the electrode-produced smiley faces for eternity and the rest.

Due to Complexity of Value, it's unlikely for a UFAI to optimize against human wellbeing for its own sake; making an AI that does that seems only a little easier than making a perfectly anti-aligned AI, which is basically exactly as hard as making an FAI.

Due to Orthogonality, it seems unlikely for a UFAI to "coincidentally" create suffering by optimizing for something "nearby". E.g. smiley faces, as a "near-miss" of alignment, basically has nothing to with human value (other than being neutral, ignoring the wasted potential). So if there's suffering, there probably has to be an instrumental goal that coincidentally involves conscious beings.

It's a little plausible that the AI has an instrumental goal of acausally bargaining with FAIs in other possible worlds by threatening to torture conscious beings; though, to the extent that negotiations succeed, there's not actually suffering created (because that's not good for anyone), there's just positive value ceded from humanity to the AI (e.g. in the form of us, in worlds where we get an FAI, devoting some fraction of the universe to paperclips or whatever). And to the extent negotiations don't succeed and the AI has to carry out the threat, that seems like it's because the AI is being dumb, which seems to contradict the AI being superintelligent. Though, some bargaining solutions do include burning value, so this isn't open-and-shut.

It's plausible that the AI has an instrumental goal of analyzing the circumstances of its creation, e.g. to bargain with other nearby possible worlds; doing seems to involve humans, which involves consciousness, and probably therefore involves some suffering. I don't see how this leads to very much suffering that's much more extreme than suffering that exists in our world. We could imagine stuff like, the AI "pulls human minds apart, in silico, to see what makes them tick", and that doing so induces lots of suffering. This is somewhat worrying to me. There's an instrumental incentive to *mostly* stick to studying non-extremely-suffering minds, just because most actual human minds aren't extremely suffering most of the time, and humans who do stuff that has historical relevance is I'd guess somewhat biased towards less extreme suffering. But there could be lots of reasons to study extreme stuff, in the spirit of science, so this is still worrying.

What can I do as a 30 year old from Portugal with no STEM knowledge? Start learning math and work on alignment from home?

Doubtful, though it's worth a try, right? If nothing else you'll understand more. Another type of route to decreasing X/S-risk is to repair the social fabric that's supposed to be keeping people from, and offering genuinely more appealing alternatives to, developing technology in a suicidally blind way. Why are people suicidally blind? What contexts bring out suicidal blindness, what supports those contexts? In other cases of suicidal blindness, what seems to hurt or help? What are the ways you're being suicidally blind, and what can you learn from that? What's it like to be suicidally blind? When do people collectively act wisely, and what's it like to be those people, what are the preconditions for that?