The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation

I could probably write a lot more about this somewhere else, but I'm wondering if anyone else felt that this paper seemed to be kind of shallow. This comment is probably too brief to really do this feeling justice, but I'll probably decompose this into two things I found disappointing:

1. "Intelligence" is defined in such a way that leaves a lot to be desired. It doesn't really define it in a way that makes it qualitatively different than technology in general ("tasks thought to require intelligence" is probably much less useful than "narrowing the set of possible futures into one that match an agent's preference ordering."). For this reason, the paper imagines a lot of scenarios that amount to basically one party being able to do one narrow task much better than another party. This is not specific enough to really narrow us down to any approaches that deal with AI more generally.
2. As a consequence of the choice of the authors to leave their framework sort of fuzzy, their suggestions for how to respond to this problem also take on this fuzziness. For example their first suggestion is that policy leaders should consult with AI researchers. This reads a little bit like an applause light, and it doesn't seem to offer many suggestions about how to make this more likely, or about how to make sure that policy leaders are well informed enough to make sure they are considering the right people to be advised by and take their advice seriously.

Overall I'm happy that these kinds of things can be discussed by a large group of various organizations. But I think any public efforts to work towards mitigating AI risk need to be very careful that they aren't losing something extremely important by trying to appeal to too many uncertainties and disagreements at once.

Meta: It feels weird to give people karma for linkposts to things they didn't write. StackExchange has a "community wiki" flag for posts that means they award no reputation (and that everyone can edit them); should we have something like that?