Comment by mtreder on What I would like the SIAI to publish · 2010-11-05T15:32:35.778Z · LW · GW

Yes, but after the AGI finds out what a paperclip is, it will then, if it is an AGI, start questioning why it was designed with the goal of building paperclips in the first place. And that's where the friendly AI fallacy falls apart.