one of the things that has come up most frequently (including in conversations with Rohin, Buck and various Open Phil people) is that many people wish there was more of a review process in the AI Alignment field. 

Hmm, I think I've complained a bunch about lots of AI alignment work being conceptually confused, or simply stating points rather than arguing for them, or being otherwise epistemically sketchy. But I also don't particularly feel optimistic about a review process either; for that to fix these problems the reviewers would have to be more epistemically competent than the post authors, and that currently doesn't seem likely to happen.

Also, when I actually imagine what the reviews would look like, I mostly think of people talking about the same old cruxes and disagreements that change whether or not the work is worth doing at all, rather than actually talking about the details, which is what I would usually find useful about reviews.

(Tbc, it's possible I did express optimism about a review process in conversation with you; my opinions could have changed a bunch. I would be a bit surprised though.)

I feel that LW is quite bad as the system for performing AI safety research. Most likely worse than the traditional system of academic publishing, in aggregate. A random list of things that I find unhelpful:
 

• Very short “attention timespan”. People read posts within a couple of days from publishing unless curated (but curation is not a solution, because it also either happens or not within a short time window, and is a subjective judgement of a few moderators), perhaps within a few weeks, unless hugely upvoted. And a big “wave of upvotes” is also a somewhat immediate reaction rather than a research appreciation.
  • Contrast: academic papers get read when they have many citations. Citations accumulate over time. Citations are a much more reliable indicator of a paper’s usefulness (exception: controversial takes by high-profile authors that invoke a lot of counter-takes). I suspect a paper’s “peak read” is somewhere within a year of the publishing date, but definitely not the day 1 after publishing.
  • Given LW/AF’s naturally faster turnover of posts and ideas (also necessitated by the ultra-fast AI progress), if it adopted showing on the front page posts that accumulated the most citations (backlinks) on these forums in the last few months, it could be more useful for guiding research.
• No standards and practices of review. No expectation that established researchers review the work of novices, thus disseminating ideas and helping novices to fix their conceptual confusion. Unstructured comments are way too “poor man’s” reviews, they often cherry-pick a small point and the ensuing argument loses the big picture quickly, making it largely unhelpful.
  • Yann LeCun proposed a review system here that seems to be implementable on LW/AF. Importantly, a full review of a post (a “paper”) could give it a “rating”, but a mere comment can’t. People can subscribe to streams of posts, approved (or highly rated) by specific researchers or groups (“Reviewing Entities”).
  • The cumulative rating of the post could also be used to guide the community’s attention. But note that reading a big post and writing a full review of it and giving it a rating or approval is much harder than just clicking the upvote button, and is in itself an intellectual bar. The “crowd” that just clicks upvotes (usually before even reading the post beyond the title) shouldn’t guide the attention of the research community.
  • The current review system on LW emphasises focusing on specific lines and claims, rather than on the big picture, and doesn’t guide the reviewers to ask the right questions about the work.
• No community standards (nor LW native support, e. g. for a dedicated "references" section) for referencing prior work. An academic paper in physics, computer science, biology, etc. with zero or a couple of references to prior work is unfathomable, in LW/AF this is standard practice (including for posts attempting to make research contributions: develop new theories, etc.). Also, these references shouldn’t be solipsistically limited to LW/AF alone.
• The selectivity of AF doesn’t seem to me to work or help (of course, this take is shaped by my personal experience of posts that IMO are obviously relevant for AF and of no lower quality than a big portion of posts on AF that don’t get published there because I’m not a member of the in-group, and, perhaps, not completely onboard with the currently most popular paradigms for thinking about alignment). Seems that AF just reinforces the “echo chamber” effect that is already present on LW (relative to the rest of scientific publishing/thought).

AF as a research medium

I think the following setup would be interesting: all posts with tags like “AI”, “Alignment”, “Agency”, plus an additional tag/label that authors may add, like “Research”, are automatically visible on “AF” (though, it wouldn’t make much sense to call that medium “forum” anymore). On these posts, apart from regular comments, people could also post structured reviews (with many obligatory sections, similar to academic reviews). There is already similar machinery on LW: on questions, there could be “answers”, but there also could be regular comments. 

Finally, the content on the “new AF” is ranked completely differently than on LW, along the lines as I described above: regular LW’s upvotes and downvotes don’t matter at all, however, the ratings given out in reviews do matter (along, perhaps, with the reputation of the reviewers), as well as backlinks/reference counts. This necessarily implies that the content on this “AF” becomes visible on the front page only slowly (busy researchers are unlikely to review work quicker than on about a week timescale, and reviews themselves take time), but also potentially stays there for longer, if the content generates a lot of backlinks.

Also, this "new AF" should provide formal academic labelling, similar to how https://distill.pub/ did.

If someone shares these ideas and has more thoughts on why LW/AF system doesn’t cater for AI safety progress well, I would like to collaborate on a shared post (for LW).

PS. We, the AI Safety community, need to “outcompete” capability research, including on this front, so it’s unacceptable that we toil behind including because our epistemic systems don’t support a more effective accumulation of knowledge and insight.

I've also been thinking about how to boost reviewing in the alignment field. Unsure if AF is the right venue, but it might be. I was more thinking along the lines of academic peer review. Main advantages of reviewing generally I see are:
- Encourages sharper/clearer thinking and writing;
- Makes research more inter-operable between groups;
- Catches some errors;
- Helps filter the most important results.

Obviously peer review is imperfect at all of these. But so is upvoting or not doing review systematically.

I think the main reasons alignment researchers currently don't submit their work to peer reviewed venues are:
- Existing peer reviewed venues are super slow (something like 4 month turnaround is considered good).
- Existing peer reviewed venues have few expert reviewers in alignment, so reviews are low quality and complain about things which are distractions.
- Existing peer reviewed venues often have pretty low-effort reviews.
- Many alignment researchers have not been trained in how to write ML papers that get accepted, so they have bad experiences at ML conferences that turn them off.

One hypothesis I've heard from people is that actually alignment researchers are great at sending out their work for feedback from actual peers, and the AF is good for getting feedback as well, so there's no problem that needs fixing. This seems unlikely. Critical feedback from people who aren't already thinking on your wavelength is uncomfortable to get and effortful to integrate, so I'd expect natural demand to be lower than optimal. Giving careful feedback is also effortful so I'd expect it to be undersupplied.

I've been considering a high-effort 'journal' for alignment research. It would be properly funded and would pay for high-effort reviews, aiming for something like a 1 week desk-reject and a 2 week initial review time. By focusing on AGI safety/Alignment you could maintain a pool of actually relevant expert reviewers. You'd probably want to keep some of the practice of academic review process (e.g., structured fields for feedback from reviewers), ranking or sorting papers for significance and quality; but not others (e.g., allow markdown or google doc submissions).

In my dream version of this, you'd use prediction markets about the ultimate impact of the paper, and then uprate the reviews from profitable impact forecasters.

Would be good to talk with people who are interested in this or variants. I'm pretty uncertain about the right format, but I think we can probably build something better than what we have now and the potential for value is large. I'm especially worried about the alignment community forming cliques that individually feel good about their work and don't engage with concerns from other researchers and people feeling so much urgency that they make sloppy logical mistakes that end up being extremely costly.

Hmm. Perhaps it could be a good idea. However, what do y'all think about the various takes that much of the problem with science boils down to the peer review process? eg:

• https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review (dec 2022, general)
• https://www.vox.com/2015/12/7/9865086/peer-review-science-problems (2015, general)
• https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/peerreview-practices-of-psychological-journals-the-fate-of-published-articles-submitted-again/AFE650EB49A6B17992493DE5E49E4431 (2010, psych replication crisis)
• https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0166387 (2016, biomedical)
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/ (2006, general)

and I'm sure even more to-the-point sources could be found, though these are no slouch themselves. of course, I was quite literally searching for articles that claim peer review is not working, so there's an obvious bias to the articles I found! but that's because I'd recently seen the dec 2022 one. The full metaphor results have a wider variety of nuanced takes, I just selected some that seemed like they made the point I wanted to make, so there's an additional layer of selection there. nonetheless, it seems like they're making an interesting point.

many people wish there was more of a review process in the AI Alignment field. 

Could you provide more details about these opinions? I imagine the people you talked to have different ideas about what a review process might look like and why it would be useful.

Is it just me or are alignment-related post titles getting longer and longer?