Students asked to defend AGI danger update in favor of AGI riskiness

In the (season) semester of (year), I decided to see how effectively I could communicate the idea of a threat from (noun) to my undergraduate classes. I spent three sessions on this for each of my two classes. My goal was to convince my students that all of us are going to be killed by a(n) (noun). My strategy was to induce the students to come up with the ideas themselves. I gave out a survey before and after. An analysis of the survey responses indicates that the students underwent a statistically significant shift in their reported attitudes. After the three sessions, students reported believing that (noun) would have a larger impact1 and also a worse impact2 than they originally reported believing.

This is less than surprising. I can't think of any threat already existing in the minds of some undergraduates that a competent believing professor requiring attendance couldn't, on average, increase. Control groups are needed.

This seems like an incredibly biased presentation, with the author never realizing the depth of the bias. Then again, he writes "My goal was to convince my students that all of us are going to be killed by an artificial intelligence," not to probe the validity of the point, so his bottom line was already written.

He says "I presented a neutral summary" after judiciously guiding the students through one-sided claims and refutations about the AGI (can never play -- plays ), not any of the claims that have not (yet) been refuted, then spicing it up with the Terminator quotes.

He says "At all points in the discussion I did my best to appear neutral and to not reveal my views." right after scaring them with a bomb in a trashcan.

He assigned no homework and gave no time outside the class for the students to come up with counter-arguments.

He writes:

In my classes, my primary goal was to teach students how to construct and assess arguments... Arguments can be assessed. If an argument has flaws, you can find those flaws. If you find flaws in an argument, the argument is refuted. If you cannot find flaws, you can take time to think about it more. If you still cannot find flaws, you should consider the possibility that the argument has no flaws. And if there are no flaws in an argument, then the conclusion of that argument has to be true, no matter what that conclusion might be.

The idea that an argument can sometimes be tested experimentally seems utterly foreign to him (even when it is in his favor, like the "AI can never be better at chess" one). Must be something about the philosophers in general, I suppose.

He primed his students in advance:

Up to this point, I had not presented any AI material to anyone in any of the classes. I had only remarked a couple of times that the AI arguments were “awesome” or “epic” or some such.

He did not attempt to provide a balanced context by inviting (or at least quoting) an expert in the area who does not share his views.

So his conclusion, that it is possible to convince a person who never thought about a topic before of the dangers of an AGI, was a foregone one. He could probably have convinced them that AGI is the second coming of Christ, if he bothered (it is a Catholic college, so the leap is not that large).

The paper gives what it describes as the “AGI Apocalypse Argument” - which ends with the following steps:

_12. For almost any goals that the AGI would have, if those goals are pursued in a way that would yield an overwhelmingly large impact on the world, then this would result in a catastrophe for humans.

_13. Therefore, if an AGI with almost any goals is invented, then there will be a catastrophe for humans.

_14. If humans will invent an AGI soon and if an AGI with almost any goals is invented, then there will be a catastrophe for humans, then there will be an AGI catastrophe soon.

_15. Therefore, there will be an AGI catastrophe soon.

It is hard to tell whether anyone took this seriously - but it seems that an isomorphic argument 'proves' that computer programs will crash - since "almost any" computer program crashes. The “AGI Apocalypse Argument” as stated thus appears to be rather silly.

If the stated aim was: "to convince my students that all of us are going to be killed by an artificial intelligence" - why start with such a flawed argument?

Does anyone else see a problem with the data table on page 22 of that PDF file?

The paper mentions this criteria on page 22:

Cells in the “change” column where I did not receive both a “before” answers and an “after” answer have been left blank.

Yet the data points at 15 8:30am, 16 8:30am, and 17 8:30am on page 22 all appear to have blank Before, present After, and a Change which is identical to the After column.

This also appears to affect the change column statistics on page 24, (8:30 a.m) and on page 26 (Both Classes Combined) For page 28, the people who only have one survey are dropped entirely, and I no longer see this problem.

Since he uses the statistics on page 28 for his conclusion, this may not change the conclusion, but I did want to point it out.

Hi everyone. Thanks for taking an interest. I'm especially interested in (a) errors committed in the study, (b) what sorts of follow-up studies would be the most useful, (c) how the written presentation of the study could be clarified.

On errors, Michaelos already found one - I forgot to delete some numbers from one of the tables. That error has been fixed and Michaelos has been credited. Can anyone see any other errors?

On follow-up studies, lessdazed has suggested some. I don't know if we need to see what happens when nothing is presented on AGI; I think our "before" surveys are sufficient here. But trying to teach some alternative threat is an interesting idea. I'm interested in other ideas as well.

On clarity of presentation, it will be worth clarifying a few things. For instance, the point of the study was to test a method of persuasion, not to see what students would do with an unbiased presentation of evidence. I'll try to make that more obvious in the next version of the document. It would be good to know what other things might be misunderstood.

This post seems to serve no purpose except to promote the dark arts.

I found this an extremely surprising result. Geoff Anders claims immediate effects from essentially only two interventions:

But you see, a plan can’t be very good if it can be thwarted by some mild fluctuations in the weather. Let’s say there’s a thunderstorm and the power goes out. Well, then the AGI will turn off. And if it turns off, it won’t be able to accomplish its goal of becoming the best possible chess player. You see, if we humans executed your plans, we would all die of starvation. We would study the rules of chess, we’d calculate chess moves and then we’d die...

and

But what if humans don’t want to install a backup generator? ... Alright, you have made some progress. You’ve solved the power source problem. But in doing so you replaced it with another problem: the human compliance problem.

There were more interventions between these and the surveys of average belief, but these interventions caused at least a few students to generate the idea that AGIs are much more creative and powerful than in Terminator 2. The effect on the tail seems to me more important and surprising than the effect on the mean.

There were a lot of interventions before these two, including whatever idiosyncrasies Anders's philosophy course had, but the outcome before these two interventions seemed pretty standard. The first AI day seemed pretty standard. The chess exercise is probably not common and the two quotes above require its context, but the initial reaction to the chess exercise did not surprise me.

I don't know of any studies backing this up, but I'd be expect that if a position doesn't go against deeply held beliefs then having to argue for it will make you update in that direction.