A Scholarly AI Risk Wiki

I vote for spending the resources in one or more of the following ways instead:

1. Write down any previously unpublished ideas in SIAI people's heads, as concisely and completely as possible, as blog posts or papers.
2. Incrementally improve the LW Wiki. Add entries for any of the topics on your list that are missing, and link to existing blog posts and papers.
3. Make a push for "AI Risk" (still don't like the phrase, but that's a different issue) to become a proper academic discipline (i.e., one that's studied by many academics outside FHI and SIAI). I'm not sure how this is usually done, but I think hosting an academic conference and calling for papers would help accomplish this.

(I've noticed a tendency in Luke's LW writings (specifically the Metaethics and AI Risk Strategy sequences) to want to engage in scholarship and systematically write down the basics in preparation for "getting to the good stuff" and then petering out before actually getting to the good stuff, and don't want to see this repeated by SIAI as a whole, on a larger scale.)

Consider the risk that after the initial burst of effort the wiki ends up sitting empty; The internet is littered with abandoned wikis, actively maintained ones being rare exceptions.

After the wiki is initially published, what would motivate one to spend an hour curating and revising wiki content instead of doing anything else?

This idea sounds promising, but I find it hard to say anything about "should this be funded" without knowing what the alternative uses for the money are. Almost any use of money can be made to sound attractive with some effort, but the crucial question in budgeting is not "would this be useful" but "would this be the most useful thing".

I recognize that it might take an excessively long time to write up all the ideas that are being thrown around, but even brief descriptions of the top three competing alternatives would help.

Actually, it occurs to me that my previous comment makes the case that SI should have a single logical document, kept up to date, maintaining its current case. It doesn't argue that it should be a wiki. One alternative would be to keep a book in revision control - there are doubtless others, but let me discuss this one.

Pros:

• A book has a prescribed reading order; it may be easier to take in the content if you can begin at the beginning and work forwards. This is a huge advantage - I'd upload it to the Kindles of all my friends who have given me permission.
• The book would be written in LaTeX, so it would be easier to convert parts of it to academic papers. MediaWiki format is the most awful unparseable dog's breakfast; it seems a shame to use it to create content of lasting value.
• Real revision control is utterly wonderful (eg hg, git) - what MediaWiki provides absolutely pales in comparison.
• Real revision control makes it easier for outsiders to contribute without special permissions - they just send you patches, or invite you to pull from them.

Cons:

• Mediawiki is easier to use
• People are used to wikis
• Wikis more naturally invite contribution
• Wikis don't need you to install or compile anything
• Much of the content is more wiki-like than book-like - it's not a core part of what SI are discussing but an aside about the work of others, and in a book it would probably go in an appendix.

I like this proposal a lot. What are the alternatives?

SI could greatly reduce public engagement, directing funds on research

As I see it, SI has two main ways to spend money: on research, and on public engagement. Obviously it has to spend money on running itself, but it's best to see that as money indirectly spent on its activities. It could direct nearly all its funding to research.

Pros: SI's research can directly bring about its central goal. We don't have all the time in the world.

Cons: Public engagement, in various ways, is what makes the research possible: it brings in the funding and it makes it easier to recruit people. In a self-sustaining non-profit, money spent on public engagement now should mean money to spend on research later. Also, public engagement directly serves the aims of SI by making more people aware of the risk.

SI could stick to presenting its existing case, leaving the gaps unfilled

Pros: That would be cheaper, and allow more to be spent on research.

Cons: given that SI's case is not intuitively appealing, making it strong seems the best way to win the right people over; as Holden Karnofsky's commentary demonstrates, leaving the holes unfilled is harming credibility and making public engagement less effective. Further, the earlier problems in this case are discovered, the more effectively future work can be directed.

SI could stick to the academic paper format, or another un-wiki "write, finish, move on" format

Pros: This presents another big cost saving: you only have to write what's new. Much of the proposed wiki content would come from work SI have already written up; there would be significant costs in adapting that work for the new format, which could be avoided if SI stick to writing new work in new papers. Furthermore, SI pretty much have to write the academic papers anyway; the work involved in writing for one format, then converting to another, can be avoided.

Cons: What you have to read to understand SI's case grows linearly. An argument made sloppily in one paper is strengthened in a later one; but you have to read both papers, notice the sloppy argument, and then reach the later paper to fix it. Or try to read the later paper, and fail to understand why this point matters, until you read the earlier one and see the context. A wiki-like "here is our whole case" format allows the case to be presented as a coherent whole, with problems with previous revisions largely elided, or relegated to specific wiki pages that need only be read by the curious.

Further, in practice the academic paper format does not free you from the need to cover old ground; in my experience finding new ways to say the same old things in the "Introduction" section of such papers introducing the problem you intend to discuss is dull and tiresome work.

I think there's lots of discussion to be had about how to get the most out of the wiki and how to minimize the costs, but as you can see, on the "is it a good idea at all" I'm pretty sold.

This sounds like a great idea to me!

A sister project that might be worth considering is to create a Stack Overflow type question-and-answer website as a companion for the wiki. (1, 2, open source clones.)

Potential benefits:

• Iron out confusing, unclear, or poorly supported points in the wiki as they are brought up as questions.
• This sort of question-and-answer platform could be a good way to hear from a wider variety of perspectives. A natural reaction of a visitor to the wiki might be "where are those who disagree"? If successful, a question-and-answer site could attract dissenters.
• By soliciting input from random Internet users, you might see them correctly find flaws in your arguments or contribute useful insights.
• Save even more time explaining things with your searchable question-and-answer database. (It's likely that the software would encourage users to search for their question before asking it.)
• I suspect this question-and-answer model works so well for reason. Don't be surprised if wiki editors find answering questions enjoyable or addictive.

Potential harms:

• Could hurt scholarly atmosphere.
• Could incentivize users to respond quickly when they should be contemplating carefully instead.

Some other thoughts: wiki talk pages could potentially be eliminated or replaced by questions like "how can we improve the page on so-and-so"? Discussion on AI risk topics could potentially be moved off Less Wrong and onto this site.

Do you have plans to invite any particular people outside of SIAI to contribute?

What is your expectation that the academic community will seriously engage with a scholarly wiki? What is the relative value of a wiki article vs a journal article? How does this compare to relative cost?

Could this help SIAI recruit FAI researchers?

The FHI already has a private wiki, run by me (with some access of some outside-FHIers). It hasn't been a great success. If we do a public wiki, it's absolutely essential that we get people involved who know how to run a wiki, keep people involved, and keep it up to date (or else it's just embarrassing). After the first flush of interest, I'm not confident we have enough people with free time to sustain it.

Would a subsection of an existing major wiki be a better way to go?

If a wiki format is what SIAI researchers want to write in- and I suspect it is- then go with that. Like you say, it's fairly easy to switch content between forms and the bottleneck appears to be getting that content out of minds.

The total cost doesn't seem all that relevant- the relative cost between writing wiki articles and writing journal articles seems to me like the driving factor.

Inspired by Wei_Dai's comment, and assuming that SI should have a single logical document, kept up to date, maintaining its current case:

SI could use the existing LW wiki for this purpose, rather than creating a new one

Pros:

• The LW wiki exists now; there will be no further design, development, or hosting costs.
• The LW wiki already has relevant useful content that could be linked to.
• There is at least some chance of at least some useful volunteer engagement

Cons:

• SI want the final say in what their evolving document says. It's not clear that we on LW want SI to treat "our" wiki that way.
• SI want their evolving document to contain only content that they explicitly approve. The LW wiki doesn't work that way.
• SI want their evolving document to have SI branding, not LW branding.

My conclusion: All of the "pros" are about the immediate advantages of using the LW wiki, while all the "cons" are about the longer term goals for the evolving document. That suggests we should start using the LW wiki now, and create an SI wiki later. Taken together with my last comment this suggests the following course of action: first, start putting all that unpublished work straight into the LW wiki. This is a change of the rules that require things to be elsewhere first, but I'd favour it. SI can fork the content into its own, separately branded wiki when it's sufficiently complete to do the job for which it was created.

The biggest thing I worry about here is eg an edit war where eg someone wants to include something in the "Eliezer Yudkowsky" page that SI think is silly and irrelevant criticism. If SI say "we run this wiki, we're not having that in there, go argue with Wikipedia to put it in there" then people may take that as meaning more than it does.

Inspired by Wei_Dai's comment, and assuming that SI should have a single logical document, kept up to date, maintaining its current case:

SI could try to get their "previously unpublished research (not published even in blog posts or comments)" out there in the form of roughly written and loosely argued blog posts, before worrying about creating this document.

Pros:

• As Wei_Dai observes, it's easy to spend lots of time laying the groundwork and never getting to the actual building stage.
• At least some people not in SI are at least roughly familiar with the groundwork already; those people would be better informed for these blog posts and might be able to usefully contribute to the discussion.
• The posts would provide material later to be made part of the single evolving case document.

Cons:

• People will try to use them to evaluate SI's case anyway, despite the disclaimer at the top.
• If I'm more convinced of SI's case from reading it, great! If I'm less convinced, hold off, we haven't finished writing it up yet. Are we asking people to suspend Conservation of Expected Evidence? (I don't think this is a problem, but not confident enough to leave it out)
• Now we're converting material twice: once from blog post to evolving document, and once from document to papers.

My conclusion: I like this idea a lot less having written it up. However, I still like the idea of getting it out there first and polishing it later. I'd rather say, start work on the evolving case document immediately, but write the new stuff first, and worry about the stuff that's already out there in other forms later. Or pages on old stuff could start life simply linking to existing sources, with very little new text. No matter what disclaimers you put on a blog post, people will treat it as a finished thing to some extent, and major edits during discussion are problematic. Linking to a revision-controlled, evolving document like a wiki doesn't have the same feel at all.

It's necessary to understand that "AI Risk" as a field? StackOverflow and MathOverflow have critical mass to create and respond to questions, respectively, in programming/computer science and math, estabilished fields of study. I supose constant feedback will maintain the wiki, if exist out there sufficient scholars who pretend to donate their time to this. But, there are?

That 1920 h should be 24 months of 80 h/month, not 80 h / week.

One risk about making arguments public is that those who originated the arguments stand to lose more face if they are shown to be incorrect publicly than privately. (I'm sure all four of the individuals you refer to care more about having accurate beliefs than preserving face, but I thought it was worth pointing out.) Not that this problem is not exclusive to the wiki proposal; it also applies to just writing more papers.

I think it makes sense in general to present one's conclusions as tentative even if one holds them with high confidence, just to guard against this sort of thing. Redundancy is good. Why not have two mechanisms to guard against any face-saving instinct?

The price tag of the wiki itself sounds too high: If 1920 hours of SI staff costs USD 48000, that's USD 25/h. If hosting and maintenance is 500 / month(should be much less), over 24 months that would leave USD 18k to design and development, and at SI staff rates that would be 720 hours of work, which sounds waaay too much for setting up a relatively simple(?) wiki site

I do not see the point in an exhaustive list of failure scenarios before the existence of any AI is established.

Yeah, I'm not going to care about reading it, and I really don't think it's possible for anyone to get close to AI without it dawning on them what the thing might be capable of. I mean, why don't we get at least /one/ made before we invest our time and effort into something that, in my belief, wont have been relevant, and in all likelihood, wont get to the people who it needs to get to if they even cared about it.