jeffreycaruso's Shortform

I downvoted this post. I claim it's for the public good, maybe you find this strange, but let me explain my reasoning.

You've come on Less Wrong, a website that probably has more discussion of this than any other website on the internet. If you want to find arguments, they aren't hard to find. It's a bit like walking into a library and saying that you can't find a book to read.

The trouble isn't that you literally can't find any book/arguments, it's that you've got a bunch of unstated requirements that you want satisfied. Now that's perfectly fine, it's good to have standards. At the same time, you've asked the question in a maximally vague way. I don't expect you to be able to list all your requirements. That's probably impossible and when it is. possible, it's often a lot of work. At the same time, I do believe that it's possible to do better than maximally vague.

The problem with maximally vague questions is that they almost guarantee that any attempt to provide an answer will be unsatisfying both for the person answering and the person receiving the answer. Worse, you've framed the question in such a way that some people will likely feel compelled to attempt to answer anyway, lest people who think that there is such a risk come off as unable to respond to critics.

If that's the case, downvoting seems logical. Why support a game where no-one wins?

Sorry if this comes off as harsh, that's not my intent. I'm simply attempting to prompt reflection.

Analogies can be found in many places. FDA prevents you from selling certain kinds of food? Sounds similar to ancient priests declaring food taboos for their followers. Vaccination? That's just modern people performing a ritual to literally protect them from invisible threats. They even believe that a bad thing will happen to them if someone else in their neighborhood refuses to perform the ritual properly.

The difference is that we already have examples of food poisoning or people dying from a disease, but we do not have an example of a super-intelligent AI exterminating the humanity. That is a fair objection, but it is also clear why waiting to get the example first might be a wrong approach, so...

One possible approach is to look at smaller versions. What is a smaller version of "a super-intelligent AI exterminating the humanity"? If it is "a stupid program doing things its authors clearly did not intend", then every software developer has stories to tell.

This is not the full answer, of course, but I think that a reasonable debate should be more like this.

I mean, I agree that there are psycho-sociological similarities between religions and the AI risk movement (and indeed, I sometimes pejoratively refer to the latter as a "robot cult"), but analyzing the properties of the social group that believes that AI is an extinction risk is a separate question from whether AI in fact poses an extinction risk, which one could call Armageddon. (You could spend vast amounts of money trying to persuade people of true things, or false things; the money doesn't care either way.)

Obviously, there's not going to be a "proof" of things that haven't happened yet, but there's lots of informed speculation. Have you read, say, "The Alignment Problem from a Deep Learning Perspective"? (That may not be the best introduction for you, depending on the reasons for your skepticism, but it's the one that happened to come to mind, which is more grounded in real AI research than previous informed speculation that had less empirical data to work from.)