How to (not) do a literature review

post by Katarina Slama · 2020-05-30T00:11:12.097Z · LW · GW · 7 comments

This is a link post for https://katarinaslama.github.io/2020/05/17/OpenAI-blog5/

Contents

  OpenAI Scholars: Fifth Steps - The Dreaded Literature Review
  How to do a literature review
    1. Start with your goal.
    2. Decide what constitutes success: How will you know that you're done?
    3. Define your questions.
    4. Answer your questions.
    5. Stop.
  How to avoid doing literature reviews
    1. Mind whom you follow on Google Scholar.
    2. Mind whom you follow on Twitter.
    3. Organize the papers that come flying your way.
    4. Make it fun.
None
7 comments

This is cross-posted from my personal blog, where I share thoughts on my work and learning process in the OpenAI Scholars Program. I thank Ruby Bloom for suggesting that I share the post here as well.

OpenAI Scholars: Fifth Steps - The Dreaded Literature Review

The OpenAI Scholars , and I among them, recently completed a project proposal for the second half of our program. Having recently finished a PhD, writing a proposal and doing the requisite literature review, should be second nature. But literature reviews were always my least favorite part of research.

Fortunately, I'm in good company. In a previous life as a young aspiring neuroscientist, I once attended a talk by David Hubel . When asked for tips about how to "keep up with the literature", I recall that Professor Hubel responded: "You know, at some point in your career, you have to decide if you want to be a consumer - or a producer." He elaborated that he would only really look at papers that his advisor or his long-term collaborator Torsten Wiesel pointed him to.

There's good reason to want to avoid literature reviews. To begin with, the problem formulation is intractable: "Know everything." If you've spent any amount of time around academics, it will soon become apparent that this is exactly what they expect from you. "Oh you haven't read that paper?" The assumption is that you have read every paper there is.

Of course, this is an impossible task. Last time I visited Google Scholar and searched on a few project-relevant terms, I encountered 105,000 papers. It takes me a full day to read and satisfactorily understand an academic paper. So reading 105 kilo-papers would take me 288 years. And I had hoped to also do some coding work during the Scholars program.

Now suppose you want to cook a ghormeh sabzi . To do so, you buy yourself a can of ghormeh sabzi mix , and follow the instructions on the can. There are about five steps. However, as ubiquitous an activity as the literature review is (in academic circles anyway), I am yet to encounter an honest and pragmatic recipe for how they are to be cooked. I really wish there was one.

I for one am a great fan of Thich Nhat Hanh's how-to books. Some of my favorite titles among Nhat Hanh's books are: "How to sit", "How to relax", and - perhaps the best one - "How to see".

I wish there had been a series of books like that available for me as a beginning grad student, covering seemingly trivial but often intractable activities like "How to read a paper", "How to attend a talk" and, indeed, "How to do a literature review".

(**Updated: Apparently there exist plenty of expositions on this topic. See the comment by AllAmericanBreakfast below. I hope that some of the ideas in this post might nonetheless add something to whatever else you might have read.)

So, this is a first draft toward such a pragmatic guide to literature reviews: How to do them and how to avoid them. Again, this is absolutely in draft form, so please do send your constructive feedback my way.

How to do a literature review

1. Start with your goal.

Do not start your literature review with the goal of "doing a literature review". Notice that such a goal does not have an end point when you can look at your work in satisfaction and notice that you are done. (Unless you have 288 years to spare). Instead, start with identifying your goal: Why are you doing this literature review? In my case, I had completed a spreadsheet with 50 papers, including their main points, type of neural recording, type of network architecture, a usefulness rating etc., by the time I realized that I had not identified my goal. I had to take a deliberate step back to recall that one goal of my literature review was to write a project proposal. Hence, my literature review should be tailored to that goal. Once I realized that, I decided to approach my work from the opposite direction: I drafted a project proposal "blind" to the literature. In other words, I wrote out a draft in bullet points with only the information I had in my mind. Once I had done that, it became quite obvious where I had knowledge gaps. Those knowledge gaps created the questions that I needed to search for in my literature review (see point 3 below.)

2. Decide what constitutes success: How will you know that you're done?

This is really important, and it's often the hardest part with an infinity-project like a literature review. In my case, the done-point was when I had completed the project proposal. But how do I know that my proposal is good enough? In practice, for me, it was when the deadline hit. If you have come up with better metrics and markers for when to consider a project completed, please do let me know.

3. Define your questions.

When doing a literature review, it's critical to know what information you're searching for. Trying to absorb all the information contained in a dense document is rarely a good plan. In my case, my questions fell out of my first, naive project proposal draft. I also talked my project through with a friend , who is a more experienced researcher. He offered the following thought points for guidance to a literature review for a new project:

1. The goal for your literature review should be to identify the problem that your project will solve. Your project might either aim to solve a new problem, or improve on an existing method.
2. Use the literature revew to identify *gaps* in the literature: What do we not yet know?
3. What are the shortcomings of existing methods to solve the problem? Why are current approaches not yet useful in practice? What is the bottleneck?

4. Answer your questions.

I think this is self-explanatory, but please let me know if not! One thing to keep in mind here is to not get lost in literature-tangents while finding answers to your questions. Provided your goal is to finish, that is. If you have the time to go off on philosophical tangents, feel free and enjoy!

5. Stop.

Just stop. Seriously. When you're done, you should stop. Go take a walk outside.

How to avoid doing literature reviews

The first section described the second-best way of doing a literature review. The best way to do a literature review is to not do a literature review. The art of avoiding doing a literature review is very similar to the art of avoiding deep cleaning your house: Keep it tidy on a daily basis. (Just like with deep cleaning your house, you probably still need to do a proper deep dive into the literature from time to time, just not as frequently as you would if you didn't have a routine of daily up-keep). Here are some ideas for "keeping up with the literature":

1. Mind whom you follow on Google Scholar.

Once you've gotten your feet a little bit wet in your field of interest, you will have a clue of who often writes interesting papers. If you follow them on Google Scholar, you will be notified as they publish more interesting stuff.

2. Mind whom you follow on Twitter.

This is especially true of deep learning, but it's becoming increasingly common in every scientific field: Those same people that you follow on Google Scholar - they will also be tweeting when they publish new cool stuff. Also when their friends publish new cool stuff. They might even highlight the most interesting aspect of their work through a figure, or a digestible blog post. Notice that whom you do *not* follow on Twitter is at least as important as whom you do follow. Optimize your twitter feed, so that you don't clutter away an excellent opportunity to find out about the most exciting and inspiring new developments in your field.

3. Organize the papers that come flying your way.

To harness your daily literature upkeep for avoiding future literature reviews, be sure to organize the work you encounter. Mendeley is a useful, and mostly free, tool for doing this, but there are many of them. (Those of us who are old and remember typing out references letter by letter, will especially appreciate the convenience of a citation manager. It will never be a good use of your time to have to think about what part of a citation should be italicized.) Once the time comes to write your next proposal or paper, you have a treasure trove of results that you've already thought about, readily available at your fingertips.

4. Make it fun.

This is the most important point. It's much easier to absorb information that you care about. One way that I hacked this in my current literature review is that I reached out to a number of authors that I had encountered in my literature search for an informal chat. This was one of the best experiences of scientific exchange I've ever had! I talked to scientists in Louisiana, Greece, and Australia, all of whom had several years more experience with my topic of interest than I did. Once you've heard a scientist tell you about their own paper - what part of the project was hard, what was interesting, what parts are only there because of reviewer request etc. - the content of their paper pops out at you in a much more colorful and living way. It's no longer such a chore to look at a paper when, instead of some dry black-and-white characters on a page, it's a friend's documentation of their work and thought process. That said, it's definitely a good idea to take a look at a paper *before* you reach out to its author. But having a scheduled call is also a good incentive to engage more deeply with your reading process.

There. I hope that's somewhat helpful. Go forth, and enjoy your reading!

7 comments

Comments sorted by top scores.

comment by DirectedEvolution (AllAmericanBreakfast) · 2020-05-31T15:32:13.586Z · LW(p) · GW(p)
I wish there had been a series of books like that available for me as a beginning grad student, covering seemingly trivial but often intractable activities like "How to read a paper", "How to attend a talk" and, indeed, "How to do a literature review".

There are a huge number of papers on Google Scholar on how to do a literature review and how to read a scientific paper. On Google, there's plenty of info on how to attend a scientific conference. Unfortunately, probably most teachers/profs in research programs don't think to point students in these directions.

Replies from: Katarina Slama
comment by Katarina Slama · 2020-06-09T03:09:47.900Z · LW(p) · GW(p)

Thanks!

At least the top hits that you point to on Google Scholar seem like they're going for something slightly different than what I'm doing here though: They seem to talk about how to write a formal literature review paper or section of a dissertation. I'm trying to describe a process for getting familiar with an entirely new field, even though you might not be publishing a paper as a result of your literature review. I'm also trying to write something that's a little bit more friendly and pragmatic than what usually gets published in academic journals.

Some of the work you link to in "how to read a scientific paper" are more similar in style to what I'm going for, and I would probably have benefitted from looking at those earlier in my graduate school career.

I added a note in the post to acknowledge this feedback. Thank you for reading and commenting!

Replies from: AllAmericanBreakfast
comment by DirectedEvolution (AllAmericanBreakfast) · 2020-06-09T20:18:58.322Z · LW(p) · GW(p)

I think that's a good niche to fill! It would be great to sort of have a review of these guides to compare them.

comment by Krabbyos · 2020-05-31T13:58:06.082Z · LW(p) · GW(p)

Are you sure Mendeley is open-source? Cause I can't find any source code for anything but the SDK for the API which is not the same as Mendeley the reference management software.

Also, Mendeley doesn't seem to have an open-source license.

In fact, looking at the terms of use for Mendeley the reference management software, section 2.3 specifically disallows a number of rights that open-source licenses protect. Notably, the right to copy and right to modify.


Anyway, to anyone looking for an actual open-source reference manager, I recommend Zotero.

[Zotero source code]

[Zotero license] (license is AGPLv3 which is actually an open-source license)

Replies from: Katarina Slama
comment by Katarina Slama · 2020-06-09T02:49:39.035Z · LW(p) · GW(p)

Thanks, Krabbyos! You're right. I should have fact-checked that rather than repeating second-hand verbal information. I will update the post.

comment by adamShimi · 2020-05-30T10:16:24.783Z · LW(p) · GW(p)

Thanks for the good post!
I especially liked the recommended questions:

1. The goal for your literature review should be to identify the problem that your project will solve. Your project might either aim to solve a new problem, or improve on an existing method.
2. Use the literature review to identify *gaps* in the literature: What do we not yet know?
3. What are the shortcomings of existing methods to solve the problem? Why are current approaches not yet useful in practice? What is the bottleneck?

I personally like to write literature reviews or related works sections for my papers, and I approach it that way. I am basically trying to place what I did or what I want to do in contrast with what has already been done, so that reader not familiar with the specifics can gauge the value of the work. There's an element of filtering the papers for the only ones that are relevant, while trying to maintain some objectivity and not piss people without intending to (important!)

Another advice I find useful is to ask for feedback on the lit review to collaborators and other researchers you know in the field. That's basically what happens during a good peer-review, where the reviewer will usually recommend you papers if they think you are missing important references.

comment by AnthonyC · 2020-06-03T20:31:18.784Z · LW(p) · GW(p)

Something like this would have been really helpful to me when I was in grad school. The constant, belittling "How could you miss that paper?" (I may not have gotten along well with my advisor) gets old quick when they all look the same to a first year PhD student looking into a topic for the first time. (Note: I eventually quit and didn't get my PhD, so I never got to any thesis-level literature reviews).