LessWrong 2.0 Reader

I have a lot of uncertainty about the difficulty of robotics, and the difficulty of e.g. designing superviruses or other ways to kill a lot of people. I do agree that in most worlds robotics will be solved to a human level before AI will be capable of killing everyone, but I am generally really averse to unnecessarily constraining my hypothesis space when thinking about this kind of stuff.

>90% seems quite doable with a well-engineered virus (especially one with a long infectious incubation period). I think 99%+ is much harder and probably out of reach until after robotics is thoroughly solved, but like, my current guess is a motivated team of humans could design a virus that kills 90% - 95% of humanity.

Everyone I talk with disagrees with most of these. So maybe we just hang around different groups.

I have identified as polyamorous for over a decade. I started and ran a local community for a number of years. I’m not an authority, but I have seen patterns.

If you want to be polyamorous, then do so. I can’t imagine monogamy at this point. Being polyamorous has brought me a lot of happiness and satisfaction.

It can also be stressful, especially when starting out. You are right that it is more work than monogamy. Don’t choose polyamory because of a cost-benefit analysis. Your relationships won’t look like what you expect, your analysis is waisted time and effort.

Choose polyamory if it is right for you. I can make book suggestions, but they’d likely be out of date at this point. Honestly though, you’d likely find them with some Google searches.

Really though? Find your local poly community. Go to a meetup. Talk to people. Listen to their stories. See if they resonate with you.

All of these seem pretty cold tea, as in true but not contrarian.

Any updates?

Yes, I expect AI labs will run extensive safety tests in the future on their systems before deployment. Mostly this is because I think people will care a lot more about safety as the systems get more powerful, especially as they become more economically significant and the government starts regulating the technology. I think regulatory forces will likely be quite strong at the moment AIs are becoming slightly smarter than humans. Intuitively I anticipate the 5 FTE-year threshold to be well-exceeded before such a model release.

Putting aside the question of whether AIs would depend on humans for physical support for now, I also doubt that these initial slightly-smarter-than-human AIs could actually pull off an attack that kills >90% of humans. Can you sketch a plausible story here for how that could happen, under the assumption that we don't have general-purpose robots at the same time?

The infrastructure necessary to run a datacenter or two is not that complicated. See these Gwern comments [LW(p) · GW(p)] for some similar takes: 

In the world without us, electrical infrastructure would last quite a while, especially with no humans and their needs or wants to address. Most obviously, RTGs and solar panels will last indefinitely with no intervention, and nuclear power plants and hydroelectric plants can run for weeks or months autonomously. (If you believe otherwise, please provide sources for why you are sure about "soon after" - in fact, so sure about your power grid claims that you think this claim alone guarantees the AI failure story must be "pretty different" - and be more specific about how soon is "soon".)

And think a little bit harder about options available to superintelligent civilizations of AIs*, instead of assuming they do the maximally dumb thing of crashing the grid and immediately dying... (I assure you any such AIs implementing that strategy will have spent a lot longer thinking about how to do it well than you have for your comment.)

Add in the capability to take over the Internet of Things and the shambolic state of embedded computers which mean that the billions of AI instances & robots/drones can run the grid to a considerable degree and also do a more controlled shutdown than the maximally self-sabotaging approach of 'simply let it all crash without lifting a finger to do anything', and the ability to stockpile energy in advance or build one's own facilities due to the economic value of AGI (how would that look much different than, say, Amazon's new multi-billion-dollar datacenter hooked up directly to a gigawatt nuclear power plant...? why would an AGI in that datacenter care about the rest of the American grid, never mind world power?), and the 'mutually assured destruction' thesis is on very shaky grounds.

And every day that passes right now, the more we succeed in various kinds of decentralization or decarbonization initiatives and the more we automate pre-AGI, the less true the thesis gets. The AGIs only need one working place to bootstrap from, and it's a big world, and there's a lot of solar panels and other stuff out there and more and more every day... (And also, of course, there are many scenarios where it is not 'kill all humans immediately', but they end in the same place.)

Would such a strategy be the AGIs' first best choice? Almost certainly not, any more than chemotherapy is your ideal option for dealing with cancer (as opposed to "don't get cancer in the first place"). But the option is definitely there. 

If there is an AI that is making decent software-progress, even if it doesn't have the ability to maintain all infrastructure, it would probably be able to develop new technologies and better robots controls over the course of a few months or years without needing to have any humans around. 

Do you expect AI labs would actually run extensive experimental tests in this world? I would be surprised if they did, even if such a window does arise.

(To roughly operationalize: I would be surprised to hear a major lab spent more than 5 FTE-years conducting such tests, or that the tests decreased the p(doom) of the average reasonably-calibrated external observer by more than 10%).

I'm not saying AIs won't have a large impact on the world when they first start to slightly exceed human intelligence (indeed, I expect AIs-in-general will be automating lots of labor at this point in time). I'm just saying these first slightly-smarter-than-human AIs won't pose a catastrophic risk to humanity in a serious sense (at least in an x-risk sense, if not a more ordinary catastrophic sense too, including for reasons of rational self-restraint).

Maybe some future slightly-smarter-than-human AIs can convince a human to create a virus, or something, but even if that's the case, I don't think it would make a lot of sense for a rational AI to do that given that (1) the virus likely won't kill 100% of humans, (2) the AIs will depend on humans to maintain the physical infrastructure supporting the AIs, and (3) if they're caught, they're vulnerable to shutdown since they would lose in any physical competition.

My sense is that people who are skeptical of my claim here will generally point to a few theses that I think are quite weak, such as:

1. Maybe humans can be easily manipulated on a large scale by slightly-smarter-than-human AIs
2. Maybe it'll be mere weeks or months between the first slightly-smarter-than-human AI and a radically superintelligent AI, making this whole discussion moot
3. Maybe slightly smarter-than-human AIs will be able to quickly invent destructive nanotech despite not being radically superintelligent

That said, I agree there could be some bugs in the future that cause localized disasters if these AIs are tasked with automating large-scale projects, and they end up going off the rails for some reason. I was imagining a lower bar for "safe" than "can't do any large-scale damage at all to human well-being".