Is the argument that AI is an xrisk valid? 2021-07-19T13:20:57.379Z

Comment by MACannon on Is the argument that AI is an xrisk valid? · 2021-07-19T14:46:53.318Z · LW · GW

Informally, it's the kind of intelligence (usually understood as something like " the capacity to achieve goals in a wide variety of environments") which is capable of doing that which is instrumental to achieving the goal. Given a goal, it is the capacity to achieve that goal, to do what is instrumental to achieving that goal. 

Bostrom, in Superintelligence (2014), speaks of it as  "means-end reasoning". 

So, strictly speaking, it does not involve reasoning about the ends or goals in service of which the intelligence/optimisation is being pressed.

Example: a chess-playing system will have some pre-defined goal and optimise instrumentally toward that end, but will not evaluate the goal itself.