How to Do Research. v1

post by Pablo Repetto (pablo-repetto-1) · 2022-09-08T01:08:35.495Z · LW · GW · 4 comments

This is a link post for https://pabloernesto.github.io/2022/09/07/research-v1.html

Contents

  Introduction
  What do I mean by research?
  Cross-cutting concerns
    Reading, fast and slow
    Research as an anytime algorithm
    Motivation
    Bibliothecography
    Note-taking and annotation
      Annotation
      Note-taking
    What is a source?
  The process itself
    Assembling a bibliography
      Definition
      Discovery
      Tracking
    Using your bibliography
      Digesting a book
      Creating a skeleton for your work
      Extracting the discourse
      Writing a coherent argument
    Learning from the community
  Open questions and further research
  References
    Main works
    Ancilliary material
None
4 comments

This post is part of my "research" series.

Introduction

Sometimes the need is desperate. You have children to raise, cancer to treat, something important broke and you're going down with it. These kind of situations rarely afford you the luxury of time. You have to do the best you can, because it's your life, and you have ultimate responsibility over it. I think information is decisive in almost any scenario. Learning to research out of desperate need is like learning to drive on the way to the hospital. The time to practice any skill is years ago. Or now. Whichever is earlier.

I'm working off of two books: Umberto Eco's How to write a thesis and Mortimer Adler's How to read a book. Both have valuable ideas, and between the two there is enough to put together a first approximation to the process.

I'm writing this down so that:

  1. I'll be forced to think through the entire thing,
  2. I can use the resulting guide as a working checklist,
  3. I'll have some place I can keep updating as my thougths on the process change, and
  4. I can share and discuss it.

I am not an expert. This is not a definitive guide. This is work in progress.

What do I mean by research?

In the academic world, "doing research" means getting published. For a thesis, this means a thesis defense; for a paper, this means peer review and editorial review. The process takes a long time and imposes pressures and incentives that I can only describe as perverse.

[...] the rigor of a thesis is more important than it's scope [...] it is better to build a serious trading card collection from 1960 to the present than to create a cursory art collection. The thesis shares this same criterion. — Eco. How to write a thesis. p5

On your specific topic, you are humanity’s functionary who speaks in the collective voice. Be humble and prudent before opening your mouth, but once you open it, be dignified and proud. [...] on the topic you have chosen [...] you must be the utmost living authority. — Eco. How to write a thesis. p183-184 (emphasis in the original)

[...] realistically I would not choose a topic unless I already knew: (a) where I could find the sources, (b) whether they were easily accessible, and (c) whether I was capable of fully understanding them. — Eco. How to write a thesis. p47

If slaving away for months or years over something that rides entirely on the approval of others—others who have no stake on it—does not distort the way you think and act, then go back to mining crypto because you've just failed the Turing test. So when I talk about "doing research", I'm not talking about breaking into academia. I think there is some overlap, as academics obviously need research skills, but academic survival is a different and much larger problem.

So what do I mean by "doing research"?

  1. Doing the best possible effort to learn about a thing,
  2. given a limited amount of time,
  3. without cheating by reducing the scope

Because efficiency matters here, I'll focus on indirect (non-experimental) learning. That means reading books and talking to people. Note that when I say "book" I am talking about source material in general, including papers, videos, web pages, etc.

Cross-cutting concerns

In aspect-oriented software development, a cross-cutting concern is some work a program needs to do that is conceptually separate from other aspects of the software, but that in practice touches everything else. The classical example is logging: whenever something happens, a message needs to be written to a log. The do-something code and the logging code are completely independent at some level, so even though they have to run together, we'd like to talk about them separately.

There are quite a few cross-cutting concerns in reseach. I'll talk about them here because it's cleaner to do so, but understand that these are relevant at every step.

Reading, fast and slow

In chapter 2 of How to read a book, Adler proposes a classification of reading skills into four levels:

  1. Elementary reading: the skill of looking at a text and understanding what its words say.
  2. Inspectional reading: the skill of looking at a text and understanding what its parts are. Composed of:
    • Systematic skimming: purposefully looking at key places (title, introduction, table of contents, index, chapter beginnings/ends) to understand what is the book's subject, who are its audience, what kind of structure it has, and most importantly whether you should keep reading it.
    • Superficial reading: reading the book from cover to cover without stopping, to get a sense of it and prepare for a more careful reading.
  3. Analytical reading: the skill of looking at a text and understanding what its author is trying to do, and how.
  4. Syntopic reading: the skill of looking at a collection of texts and undertanding what its authors are saying to each other (sometimes implicitly, sometimes unknowingly).

Each level of reading corresponds roughly to a part of the research process. Performing well in one does not automatically mean you will do so in the others, or that you'll know when to use each one. I'll discuss these in more detail as they become relevant, but at the bare minimum get good at skimming.

Research as an anytime algorithm

In computer science we have the (slightly obscure) notion of an anytime algorithm. This is an algorithm that is trying to find some solution to a problem (say, a configuration of a problem that gives the highest score, or a plan that minimizes a cost) and will run to completion if you let it, but also at any point when you get bored you can stop it and ask it for its current best answer and it will give it to you. — MacIver. Life as an Anytime Algorithm.

Many problems can be approached by successive approximation. Say you need to find the best hepathologist money can buy. You can start by just picking a random one. As you learn more, you could favor the ones that come from the most prestigious medical schools (reasoning that they got prestigious by training good doctors) or those that work for the most prestigious hospitals (reasoning that they got prestigious by having good doctors). At some point you may know enough to evaluate a doctor's performance directly. But at any point, you can just stop and pick a doctor.

Some problems cannot be handled this way, at least not in any meaningful way. For instance: back in 2004, billboards popped up in various places reading:

{ the first 10-digit prime in consecutive digits of e }.com

These turned out to be recruitment ads for Google[1]. What can we tell about the result without solving the problem? Very little: it is a 10-digit number, it shows up in e, it is prime. Using one of these pieces of information to narrow down your search is no easier than using all of them and just solving the problem. If you're faced with a really hard problem of this kind (say, breaking RSA) you are probably boned.

While this is discouraging, it doesn't have to mean you've wasted your time. Research is hard work, and important problems have a tendency to crop up again and again. Research will involve writing lots of things down, and a lot of it could be useful down the line, possibly to someone else.

These are the useful side products I know of:

Motivation

I wish I had this one locked down. This post took me months because I couldn't put in more than 20 hours a week. Showing up is huge. I wasn't pressed for time, I just can't seem to keep myself going. At least I got it done. Going by precedent, I would have bet on me giving up.

Here are the things I've noticed working for me:

Bibliothecography

While these days "to Google" means "to search", Google Search is not actually a library (Google Books notwistanding). The landscape of knowledge is not just the books that were written and the authors that wrote them, but also how and where they were kept.

As you build your bibliography—a catalog of sources—you should build a catalog of sources of sources (a bibliothecography). Where did you find your books? Where else could you search? Many specialized collections are either not accessible online, or don't play ball with Google.

There are also university libraries, newspaper archives, academic journals, private collections, etc. Keep track of these explicitly.

Note-taking and annotation

What's the point of taking notes? Aren't you just repeating what your sources say?

Kind of. When you read a book for research, you and its author may not care about the same things. Taking notes means you can highlight the parts that are key to your argument. If you are reading several books, you can record connections between them. And you get to write down your own ideas.

You're basically building an outline of important details and events right into the pages of your book so you can effectively keyword-search it later just by flipping through. It's command-f in real life. So you can narrow your search for a detail or quote down to a page rather than an entire chapter. — Blue. How to Read Books and Write Essays. 05:01

Command-F in real life? Computers are real life! Why not use it and forget about annotation entirely?

Because annotations are intentional. Full-text search can cause three kinds of errors:

  1. False positives: search for "IT" (information technology) and you'll find the word "it".
  2. False negatives: "color" instead of "colour", "Tokyo" instead of "Edo".
  3. Irrelevant results: mentioning something is not the same as talking about it.

There's a distinction between adding notes to a book, and writing notes about a book. I'm going to call the first "annotation" and the second "note-taking".

Annotation is convenient, but it locks the information into its original context. Say you read Robert Heinlein's The Puppet Masters and make a note on the margin that says "body snatchers". When you read Applegate's Animorphs you may not remember that you've read about body snatchers before, or remember that you did, but not where. Now you have to go back and review every annotation you've ever made, which could have been avoided by making a note for "body snatchers" in the first place. Plus, note-taking is medium-agnostic, annotating a movie can get quite involved.

The more you read, the more stuff you have to remember, the harder it gets to keep everything straight. Relying on your memory simply doesn't scale, while note-taking does.

It doesn't matter if you think you'll remember it, you want to write it down anyway, because this isn't about memorizing, it's about putting all your information in one place so you can deal with it all at once later. — Red. How to Do Research. 5:51

Annotation

Readings index cards are useful for organizing critical literature. I would not use index cards, or at least not the same kind of index cards, to organize primary sources. [Filing references to primary sources] would require a huge effort, because you would have to practically catalog the texts page by page. — Eco, Umberto. How to Write a Thesis. p123,125

Reading a book should be a conversation between you and the author. [...] Marking a book is literally an expression of your differences or your agreements with the author. — Adler and Van Doren. How to Read a Book. ch5, 13.20

I'm squeamish when it comes to annotating books, but it doesn't seem like I have much of a choice. Just as annotation doesn't scale to lots of books, note-taking doesn't scale to very dense books. So how do we annotate?

[Underlined passages] allow you to return to the book even after a long period, and find at a glance what originally interested you. [...] Some people underline everything, which is equivalent to not underlining at all. — Eco, Umberto. How to Write a Thesis. p124

Using color lets you add more marks without sacrificing speed, if used intelligently. Color must mean something, and you must know what that meaning is. If you know you are looking for the red mark, you'll find it quickly. If you know you are looking for a red mark, you'll have to stop and check each one. If you don't know what color mark you are looking for, then what is even the point?

Add a note at the beginning or end of the book explaining the meaning of each color. Alternatively, you could make an index, but this is painstaking work.

You can highlight passages by underlining them, circling them, adding a dash or vertical bar at the margin, etc. If you use more than one type, use that to convey meaning. For example: underline key arguments, circle definitions, use a vertical bar for quotable passages, and use dashes for everything else.

You don't have to be consistent across books—and probably shouldn't be. If you are annotating a novel you'll probably want to jump to important events or memorable quotes. That isn't that helpful when reading Das Kapital. As long as you can tell what the system is on each book, use whatever makes sense.

Hamlet's "To be or not to be" is about suicide, but he doesn't use the word "suicide". This makes highlighting somewhat awkward, so write the keyword you wish you had instead. When parts of an argument are scattered across several paragraphs, number them. Finally, you can use marginalia to add in connections to other pages or books. Use abbreviations where possible to conserve space.

Note-taking

Journaling

[Clippings] can be vital for refinding old things you read where the search terms are hopelessly generic or you can't remember an exact quote or reference; it is one thing to search a keyword like "autism" in a few score thousand clippings, and another thing to search that in the entire Internet! — Branwen, Gwern. Internet Search Tips. s1.4

While organization isn't optional, dumping your thoughts as they come on a notebook or plain text file is a big step up from nothing. Because your notes contain only those things you put there on purpose, Ctrl+F or even manually looking for things can work well enough. And the things you are most likely to look up are those that you've thought about recently, so they should be in the last few pages. Even if you keep a more sofisticated system, keep a journal as a low-friction way of getting stuff out of your head.

Files

Remember that an index card file is an investment that you make during your thesis, but if you intend to keep studying, it will pay off years—and sometimes decades—later. — Eco, Umberto. How to Write a Thesis. p122

I'll cover this in more detail in s4.1.2.3, but the basics are to split your notes by type and keep them sorted.

The main categories you'll want are:

What makes this useful is connections. If you go to the note on "Variance" finding a definition is nice, but finding a list of people and books that talk about variance is great.

What is a source?

The absolute first step to any research project is... WIKIPEDIA. [...] We all learned years ago that you're never supposed to cite Wikipedia, and this is completely true. [...] But what Wikipedia is good at is directing you to ACTUAL sources. Right at the bottom, in that sweet little references section is a goldmine of all kinds of sources. — Red. How to Do Research. 0:31

But, why shouldn't you cite Wikipedia?

Warning: this section is unsourced.

To my mind, there are two main criteria that should be applied when evaluating a source:

  1. Can it be pinned down?
  2. Can it be tracked down?

Citing a source should make it possible to go and look at it. If I cite The Bible, am I talking about the New King James Bible, the New International Version, or Noah Webster's Bible Translation? Getting the wrong version can be a big problem. And even when everybody agrees on what a source says, the context of a claim is critical to evaluating whether it's true.

To understand the possible biases and factors present in the primary sources you want to know who wrote them, and when, and what exactly was happening at the time. Primary sources help you examine the thing, secondary sources let you examine the primary sources. [...] Context for your sources is seriously everything, I can't stress that enough." — Red. How to Do Research. 4:15

Wikipedia actually does really well on both of these metrics. Sure, if you cite Wikipedia is not a reliable source the article could change under you without any warning; but if you cite Wikipedia is not a reliable source, 2022-04-08 17:42 you'll be referring to a specific version of the article, and the problem goes away. You can even check its edit history and see every single change to the article, when it was made, by whom, and what other contributions that person made to Wikipedia. University websites and newspaper sites are usually not this stable. So maybe you should cite Wikipedia, if you're careful about it?

There is a second consideration, which is that questions central to your research deserve more attention and care than questions in its periphery. This means being more or less strict with following claims back to their source. Since you'll have to revise your understanding of what your topic is multiple times, inevitably some sources will in and out of scope as you work.

The fact that Napoleon died on May 5, 1821 is common knowledge, usually acquired through indirect sources, such as history books written on the basis of other history books. If you wish to study the precise date of Napoleon's death, you would need to locate original documentation. But if you wish to address the influence of Napoleon's death on the psychology of European liberal youth, you can trust the date that appears in any history book. — Eco. How to Write a Thesis. p52

The process itself

Assembling a bibliography

[...] although we sometimes go to the library to find a book that we already know exists, we often go to the library to find out if a book exists, or to discover books about which we have no previous knowledge. — Eco. How to write a thesis. p54

While the research material you need will be inside books, the other resources in a library are the map that will let you find it. A mere heap of books is not a library, no matter how high you stack them. In fact, the higher you stack them, the bigger the mess gets.

Inside a library you can find:

  1. Books
  2. Catalogs
  3. A reference section
  4. A librarian

Online, Google generally serves the part of the catalog and librarian, while Wikipedia serves the part of the reference section.

The process of assembling a bibliography can be conceptualized as three steps:

  1. Defining your research topic: deciding what is in scope and what is out of scope.
  2. Discovering relevant sources: learning of the existence of books connected to your topic.
  3. Tracking down sources: geting the books so you can read them.

These steps don't happen in strict sequence. Reading one book will frequently tell you about others, and refine your ideas about what your topic is. I've chosen to discuss the process in this order because as you work, you'll transition from doing more of the earlier kinds of work to more of the latter.

Definition

Begin to read the material as your bibliography grows. It is unrealistic to think that you will compile a complete bibliography before you actually begin to read. — Eco. How to write a thesis. p115

A curious paradox is involved in any project of syntopical reading. Although this level of reading is defined as the reading of two or more books on the same subject, which implies that the identification of the subject matter occurs before the reading begins, it is in a sense true that the identification of the subject matter must follow the reading, not precede it. In the case of love, you might have to read a dozen or a hundred works before you could decide what you were reading about. And when you had done that, you might have to conclude that half of the works you had read were not on the subject at all. — Adler and Van Doren. How to Read a Book. ch20

When pressed for time, being able to skip an entire book with just a brief inspection is no small thing. Writing down a definitive statement on what your subject is may seem like a hopeless endeavor, but do it anyways. You'll work faster and surer, and having your criteria written down will help your understanding evolve.

As you iterate this process, you'll find that you've wasted time with some books, and that you need to re-examine others. Accept the fundamental swingyness of the game, and play your best.

Discovery

A preliminary inspection of the catalogs allows you to prepare a list of books that you can then begin borrowing. However, the list you derive from the catalogs does not say much about each book's contents, and it is sometimes difficult to determine which books you should borrow first. For this reason, in addition to consulting the catalogs in the reference room, you should preliminarily inspect each book. When you find a chapter and its accompanying bibliography that pertain to your topic, you can skim the chapter (you will return to it later), but be sure to copy all of that chapter's bibliography. Together with the chapter that you have skimmed, its bibliography (and if it is annotated, the bibliography's comments) will show which books the author considers fundamental among those he cites, and you can begin by borrowing those. Additionally, if you cross-check the bibliographies with some reference works, you will determine which books are cited most often, and you can begin to establish a first hierarchy of sources for your topic. — Eco. How to write a thesis. p58

There are three sub-skills here:

  1. Finding books that are likely to be relevant.
  2. Skimming them to find out if they are.
  3. Organizing and keeping track.

Finding books

I'll talk about the process as if you were on a library. On the internet things are much the same, except that there is much more to be found, it can be quite a bit harder to find it, and you are missing the (frankly invaluable) help of a librarian. For these reasons, if you live in a major city you may have an easier time doing your research at a physical library instead of online.

In rough order, you'll probably want to:

  1. Check a general reference work.
    General encyclopedias such as Wikipedia are the first thing you should check. If there is an article about your topic, read it and look for related articles. Going over the Wikipedia article for "Research" I quickly stumbled upon "Epistemology", "Scientific method", "Metascience", "Primary source", "Library science", and "Scientific literature", among others. Write down relevant-looking books, keywords, and people.
  2. Scrape course syllabi.
    If your subject is well-studied, there are likely courses about it online, in universities, or elsewhere. Obviously, you can take one. But let's say you don't; check if they have a publicly available syllabus. This is a sort of table of contents for the course, so scrape it for books, keywords, and people.
  3. Use the catalogs.
    Look up everything you've collected thus far. Use online catalogs; you can consult the catalogs of major national libraries (such as the US Library of Congress) from anywhere with an internet connection.
  4. Check a specialized reference work.
    Check reference works written by and for specialists (such as topic encyclopedias, "handbooks", or bibliographical indexes). Keep in mind some reference works are basically tables of figures in book form, and thus unhelpful to you at this stage. Without some reading done you may be hard pressed to query these effectively, or to understand what they have to say. However, that's ok; right now you are just trying to learn what concepts exist, so if something you don't understand keeps popping up, write it down so you know it's there.
  5. Find books through books.
    Skim what you've found (I'll discuss how in the next section) to keep expanding your collection. If a book has a bibliography, go to town on it. If it's not at the end of the book, the references may be on a per-chapter bibliography or even in footnotes. If a book seems useful, try other works by the same author[5].
  6. Ask a librarian.
    I've put this at the end, but don't be deceived: you should ask for help at every single step (except step 2, unless the library is attached to the university whose courses you are scraping). The librarian may know that some topic is under an unusual keyword[6], or that the catalog is split into a new and old version, or what is that one textbook everybody keeps asking for.

I believe that online journals and Google Scholar have wholly supplanted the traditional role of bibliographical indexes, but they are worth a mention because they are a libary's catalog in book form, and that is a really neat idea.

For some disciplines there are famous manuals where the student can find all the necessary bibliographical information. For other disciplines there are periodical indexes that contain updates in each issue, and even journals dedicated solely to a subject's bibliography. For others still, there are journals that include an appendix in every issue that documents the most recent publications in the field. Bibliographical indexes are essential supplements to catalog research, as long as they are updated. — Eco. How to write a thesis. p56

Skimming

As was promised in section 3.1, let's discuss skimming. When you are doing research, you care about a specific thing (your subject's center) and related things that provide context (your subject's periphery). The goal here is to put a book in center, periphery, or out of scope as quickly as we can manage. In an idealized process you'd read the title of every work you discover, skim everything that seems relevant, superficially read everything that is relevant, and then carefully digest (over several passes) only the stuff that contains critical information.

In chapter 4 of How to Read a Book, Adler proposes the following skimming process:

  1. Read the title and preface of the book
  2. Study the book's structure through its table of contents
  3. Study the book's main arguments through its index entries (if present)
  4. Quickly read the publisher’s blurb
  5. Identify key chapters. Often the author summarizes it in one paragraph at the beginnning or end; if this is the case, read it carefully.
  6. Quickly read some random paragraphs
  7. Quickly read the last few pages

I can do the above with any book in about an hour. You don't have to do all the steps for each book, you'll often know enough with the first two steps. And I'd add a zeroth step: look at the book's cover; it contains a wealth of information.

In real life you can't judge a book by its cover, but in fiction that is what the cover is there for. — OSP Red. Trope Talk: Are We The Baddies? 0:30

Adler proposes an initial classification of books into:

Fiction
Expository
    History
        Practical
        Theoretical
    Philosophy
        Practical
        Theoretical
    Science
        Practical
        Theoretical
    Mathematics
        Practical
        Theoretical

This is obviously and opinionated selection, but there is some sense to it:

  1. Fiction is mainly concerned with transmitting experiences. Mathematical fiction (Edwin Abbot's Flatland, for example) needs you to follow the math at some level to keep you invested, so the author is incentivized to build your intution (something that textbooks authors often aren't).
  2. History, Philosophy, Science, and Mathematics is... a choice. While this particular set is kind of arbitrary, explicitly stating the subject of a book will help you decide if you need to read it, how deeply, and when. Earlier on you'll look at more general works before diving into more specialized material.
  3. Equally, depending on whether your research is theoretical or practical you'll get to summarily ignore a lot of books, or even whole fields. This post, for instance, is practical. Whether research can be done, and how we can know things at all, are not in question. Thus, epistemology is at best part of my subject's periphery.

Questions about the validity of something are theoretical, whereas to raise questions about the end of anything, the purpose it serves, is practical. — Adler and Van Doren. How to Read a Book. ch6

Does something exist? What kind of thing is it? What caused it to exist, or under what conditions can it exist, or why does it exist? What purpose does it serve? What are the consequences of its existence? What are its characteristic properties, its typical traits? What are its relations to other things of a similar sort, or of a different sort? How does it behave? These are all theoretical questions. What ends should be sought? What means should be chosen to a given end? What things must one do to gain a certain objective, and in what order? Under these conditions, what is the right thing to do, or the better rather than the worse? Under what conditions would it be better to do this rather than that? These are all practical questions. — Adler and Van Doren. How to Read a Book. ch5

Organizing

You may be able to mentally keep track of the books you've read, but keeping track of every book you've seen mentioned is clearly out of the question. In chapter 3 of How to Write a Thesis, Eco explains how to use index-cards to keep track of your bibliography. Obviously we can do better with computers, but I'll explain Eco's system because the principle is the same, and physical index cards are easier to reason about.

Information is organized in files consisting of an ordered collection of cards. A file can be stored in a filing cabinet, carboard box, binder, or by wrapping it in an elastic band, depending on the number and size of cards.

We use cards because they can be kept sorted (a notebook doesn't really lend itself to random insertion), and we keep them sorted so that they can be found quickly. The files we'll discuss are alphabetical, but you could sort them in some other way. For example, you could create a chronological file for historical events.

At the bare minimum you'll want:

  1. A bibliographical file.
    This lists all references you are aware of, regardless of whether they make it into your final bibliography or even if they are available. Cards on this file contain full and accurate bibliographic information (or as close as you can manage), directions on how to obtain a copy (when one is available), which category it belongs to (see the previous section), and a space to mark priority books to obtain/read. You may also want to leave some space to note how you originally found out about a book.
  2. A readings file.
    This lists all the books you have actually read (not just skimmed) and includes notes on their contents (opinions, summaries, and quotable excerpts). Since you already have an entry on these books in the bibliographical file, you can skip a lot of details.

As recommended extras:

  1. An author file.
  2. A quotes file. Put clever or amusing quotes here so they won't get in the way of more serious work.
  3. An ideas file (one card per concept).

You can add references to other cards with:

The goal is to have one place where you can go to and see all that you have on a thing. For ideas, you want to know where they are discussed, by whom, and what related ideas there are (eg: "modularization (programming)" would link to "Parnas, 1972", "Donald Knuth", and "programming languages", respectively). For authors, what did they write, what is written about them, and who did they debate/collaborate with.

An annotated bibliography, complete with cross references to author and idea catalogs, takes a long time to compile, so they are a valuable partial result (see s3.2).

I don’t give app of the year awards, but I would 100% give it to Obsidian for slowly taking over almost everything I do that has anything to do with text files. — CGP Grey and Myke Hurley. Cortex podcast ep122. 2021. https://www.relay.fm/cortex/122.

I had originally written an explanation of how to use Obsidian and Zotero, but it ended up being a huge diversion, so I cut it. If you'd rather have no system than going through the indignity of installing new software, then use a spreadsheet. It's not an elegant solution, but it can be made serviceable with a bit of patience.

Tracking

These are some main points from Gwern's Internet Search Tips, see the original for a far more thorough discussion.

Searching on the internet is work which can get harder without prior notice. Websites change and die, breaking links. This is out of your control. If you've found something useful, archive a copy and make it more findable, at least to yourself. Ideally you'd host a public version. For websites, go to the Internet Archive's Wayback Machine and request an archive copy to be made.

When using a search engine you can search for:

...among others. The operators used above are Google's, but they'll usually work in other search engines. Check out your engine's version of Advanced Search.

If you aren't finding what you are looking for, don't give up right away. Try altering the query a bit.

Remember that Google Search isn't your only recourse, or even your best.

If you are having trouble tracking down an academic paper, it may be hiding inside a poorly-formatted anthology. Check Gwern's post for details on how to track it down and create a new document with just the relevant pages.

A note on piracy: unless you sip from a truly inexhaustible fountain of money, you'll probably have to resort to pirated material. Even with institutional backing, you may find 100% above-board researcg finantially straining. This in spite of the fact that academic publishers don't pay for researcher or reviewer salaries, and the marginal cost of sending you a copy of a PDF is basically zero. Remember that this is a universe where lootboxes exist, and finding "whales" to sell them to is considered a viable business model.

Using your bibliography

It seems to me the most sensible answer is this: approach two or three of the most general critical texts immediately, just to get an idea of the background against which your author moves. Then approach the original author directly, and always try to understand exactly what he says. Afterward, explore the rest of the critical literature. Finally, return to examine the author in the light of the newly acquired ideas. But this advice is quite abstract. — Eco. How to write a thesis. p104

Now that you have source material, let's discuss reading and writing. Let me remind you that, while this post is sequential, the actual process will be a lot messier. Every step can happen at any time, with only a gradual transition from doing mainly one kind of work to another.

Digesting a book

Pre-reading

Reading for entertainment is easy because it's designed to enthrall you. Reading for academic purposes has no such kindness. [...] Reading is easy for exactly as long as it's interesting or fun, after that: Welcome to the deep end, kids. — Blue. How to Read Books and Write Essays. 01:33

Before even attempting a thorough reading, you should be following the advice on skimming from section 4.1.2. If you have a summary available, use it to grasp the structure of the text before you start reading. Check what it says against what you've learned from skimming the book.

That's really what a summary does best: It primes you on what to expect so that you can read for detail with the broad strokes already in your mind. [...] it's a much better idea to look up the summary before you read the book than after. If you didn't get the book when you first read it, odds are you won't remember it too well, and the summary won't jog memories that you don't have. — Blue. How to Read Books and Write Essays. 02:15

Audiobooks probably aren't great for every kind of book (math audiotextbook?), but if it's available and it works for you, use it.

You may also want to try reading a limited reading of a book:

A focused reading is one in which you ignore all but a specific aspect of it. Blue suggests reading a story for one of "plot, character, themes, symbolism, or style". This can be a lot of fun. Try picking up a story you've enjoyed, choose a random secondary character, and go through the whole thing again pretending they are the protagonist. Star Wars from Lando's perspective?

A superficial reading is one in which you read from cover to cover without stopping. From chapter 4 of Adler's How to Read a Book:

In tackling a difficult book for the first time, read it through without ever stopping to look-up or ponder the things you do not understand right away. [...] You will have a much better chance of understanding it on a second reading, but that requires you to have read the book through at least once.

Most of us were taught to pay attention to the things we did not understand. We were told to go to a dictionary when we met an unfamiliar word. We were told to go to an encyclopedia or some other reference work when we were confronted with allusions or statements we did not comprehend. We were told to consult footnotes, scholarly commentaries, or other secondary sources to get help. But when these things are done prematurely, they only impede our reading, instead of helping it.

If you insist on understanding everything on every page before you go on to the next, you will not get very far. In your effort to master the fine points, you will miss the big points [...]. You will miss the forest for the trees. You will not be reading well on any level.

Analytical reading

This is a summary of ideas from Adler's How to Read a Book, see the original for a more thorough discussion. This is where you'll engage most deeply with a given book, so be prepared to write copious notes.

There are three stages to analytical reading:

  1. Finding the book's structure
    1. What kind of book is it?
    2. What is the unity of the book?
      In a narrative, this is the plot in one paragraph. In an expository book, this is the main thread of argument.
    3. What are the parts of the book?
      And not just what the parts are, but how they fit together. An example could be "the introduction presents the problem and states the solution, chapters 1 and 2 provide a historical summary of the field, chapters 3 to 5 explore alternative solutions, chapter 6 presents the solution in full, chapter 7 shows benchmark metrics, appendix 1 proves a point used in chapter 7, appendix 2 is an example implementation"
    4. What are the author's problems?
      These are the things that the book revolves around, and that it should answer. Crime and Punishment asks, among other things, "why do men do things that later cause them guilt?"
  2. Interpreting the book's meaning
    1. Come to terms with the author.
      Any word the author uses specially, you must understand precisely. You must understand what the author is saying even if he uses the same word to refer to more than one concept, or more than one word for the same concept.
    2. Find the author's propositions.
      These are the individual claims that an author makes.
    3. Locate or construct the book's arguments.
      "Locate or construct" because sometimes arguments are made implicitly. In persuasive writing, this is often used to hide flimsy arguments. "The outsiders are all bad and we should kick them out", for example, is often argued tacitly.
    4. Find the author's solutions.
      Crime and Punishment concludes "men are more vulnerable to emotion than they believe".
  3. Disagreeing with the author
    1. "You must be able to say, with reasonable certainty, 'I understand,' before you can say any one of the following things: 'I agree,' or 'I disagree,' or 'I suspend judgment.'"
    2. "When you disagree, do so reasonably, and not disputatiously or contentiously."
      This means not seeking pointless conflict and sticking to the Four valid criticisms, listed below.
    3. "Respect the difference between knowledge and mere personal opinion, by giving reasons for any critical judgment you make."

Adler considers higher levels of reading to include lower levels, so analytical reading includes inspectional reading, which includes elementary reading. Stage one of analytical reading is performed by inspecting the book.

Adler claims that there are four valid criticisms one can make of an argument:

By "valid" we mean that these criticisms show that the author's arguments are not sound, in the formal sense.

You should distrust the feeling that you've understood a book. Test your understanding by restate the author's ideas in your own words, and looking for particular examples of general propositions ("Is a brick an essential object?"[7]).

Distrust your sources

To understand the possible biases and factors present in the primary sources you want to know who wrote them, and when, and what exactly was happening at the time. Primary sources help you examine the thing, secondary sources let you examine the primary sources. [...] Context for your sources is seriously everything, I can't stress that enough. — Red. How to Do Research. 4:15

[...] I should not quote my author through another quote. In theory, a rigorous scientific work should never quote from any quote, even if the material that I wish to quote is from someone other than the object of my thesis. Nevertheless, there are reasonable exceptions [...]. — Eco. How to write a thesis. p51

If you haven't seen it yet, please check out CGP Grey's Someone Dead Ruined My Life... Again. It is a case study on confronting bad sources with truly heroic integrity. The video is 20 minutes long and very entertaining, so I'll proceed under the assumption that you've watched it.

Summarizing the argument: Grey found a poem quoted in several modern sources containing the name "Tiffany", proving that the name existed when the poem was written. The question of when the poem was written is obviously critical.

Let's see the trail that Grey followed:

In this case, there were two problems. First, the original poem was simply wrong; the name was not Tiffany at all. Second, the context of the poem (when it was written and by whom) was missing. But there are three ways in which a source can mislead you:

  1. The source's claim is untrue.
  2. The text you have does not accurately reproduce a claim, or elides critical context.
  3. You interpreted the claim incorrectly.

You should not forget that one word can represent several terms. One way to remember this is to distinguish between the author’s vocabulary and his terminology. If you make a list in one column of the important words, and in another of their important meanings, you will see the relation between the vocabulary and the terminology. — Adler and Van Doren. How to Read a Book. ch8

There are several ways you can misinterpret a claim:

Creating a skeleton for your work

Types of text

Point one is deciding what you should write. I propose three broad categories of work: exposition, reference, and journal. The difference between these should be obvious from their table of contents.

Both Adler and Eco focus on expository work. However, they both recommend organizing your notes like a reference work (as seen s4.1.2.3). While neither discusses keeping a journal, I believe this helps in two ways:

  1. It gives the researcher a low-overhead, low-commitment space.
    Ideas that initially seem not worth pursuing, or too hard to concile with an existing framework, can end up being very fruitful. Being able to write them down without commiting to them lets you put them safely away while you focus on something else.
  2. It provides an audit trail.
    Using an experiment as evidence for a theory it helped develop is double-counting the experiment[8]. Being able to go back and look at a researcher's process is a big help if you are trying to evaluate their work or learn from them.

Thesis last?

After you have conducted your bibliographical research, one of the first things you can do to begin writing your thesis is to compose the title, the introduction, and the table of contents [...] You may object to this idea, realizing that as you proceed in the work, you will be forced to repeatedly revise this hypothetical table of contents, or perhaps rewrite it altogether. This is certainly true, but you will restructure it more effectively if you have a starting point from which to work. Imagine that you have a week to take a 600-mile car trip. Even if you are on vacation, you will not leave your house and indiscriminately begin driving in a random direction. [...] you may change your itinerary in the middle of the voyage. But you will modify that itinerary, and not no itinerary. — Eco. How to write a thesis. p107-108

Beginning a research project by coming up with your conclusion is famously bad advice.

[...] usually when papers are assigned, the teacher tells you to pick a thesis and support it with evidence from the book. I don't know why they phrase it like that because it's completely backwards. Your thesis arises from your thoughts on the book after you read it. Here's a fun secret, lots of college professors will actively tell you to write your thesis statement last, since your opinions on the text will naturally evolve as you examine it in more depth. — Blue. How to Read Books and Write Essays. 03:41

It's dumb. Don't do that. — Red. How to Do Research. 06:51

However, we are not starting off with our conclusion. We've hit the libraries, we've skimmed lots of books (and realistically, carefully read some of them), we've taken notes, and we've assembled a preliminary bibliography. At this point we should start generating hypotheses and narrowing our focus.

There are three things we need to write:

  1. The title.
    This is the guiding question of our research; our thesis in a sentence. "Does chocolate prevents cancer?" "What is the best high-school in the Bay Area?" "How to do Research?"
  2. The introduction.
    Why did you write this? What are the limits of your topic (center, periphery, out of scope)? What were your results?
  3. The table of contents.
    This is the structure of your argument (if you're writing an expository work). "Context, facts, argument, conclusion, appendix" is a sensible place to start. You can elaborate on this basic motif by creating subsections or you could choose some other structure entirely.

Make yourself a provisional table of contents and it will function as your work plan. Better still if this table of contents is a summary, in which you attempt a short description of every chapter. By proceeding in this way, you will first clarify for yourself what you want to do. Secondly, you will be able to propose an intelligible project to your advisor. Thirdly, you will test the clarity of your ideas. — Eco. How to write a thesis. p108

Provisional is the key word here. You can and will come back to these later on (possibly to toss them out and start over) so don't try to get them perfect right away. You can add a guess on the results (if you've got one) or leave a space that you'll have to fill in later. Let future-you deal with the mess, they'll have better perspective and more information.

The table of contents, in particular, will be invaluable:

A well-written hypothetical table of contents is the numerical grid that allows you to create cross-references, instead of needlessly shuffling through papers and notes to locate a specific topic. This is how I have written the very book you are reading. — Eco. How to write a thesis. p114

As I write this, I have a second document open. It is structured exactly as this post, but each section is composed of bullet points, each of which is either a topic I want to discuss, or some piece of evidence I should bring forth (mostly quotes). About fifteen of these one-liners become a full section, and I often find myself talking about things that weren't in my notes. Don't skip this step. I can't explain how relieving is to sit down to write and just be able to read what I'm supposed to talk about.

Extracting the discourse

In chapter 20 of How to Read a Book, Adler explains what he calls "syntopic reading". This is the process of extracting the views of different authors on a single topic and mapping out their arguments.

  1. Bring the authors to terms.
    Translate the authors' arguments to a single common terminology. Ideally, you wouldn't adopt any single author's vocabulary to prevent extraneous connotations from leaking in.
  2. Create a common questionnarie.
    Construct a series of questions that, if answered, would summarize an author's position on your subject, then extract the answers from their texts. If an author does not answer the question directly, you are allowed to perform some inference work, but you must not make shit up[9].
  3. Find the core disagreements.
    Where do different authors disagree? Why? What are the consequences of their disagreements on their overall beliefs? Is there enough common ground to say they are really talking about the same thing[10]?
  4. Write it all out.
    For Adler, this is contribution enough. Finding the shape of the discussion as it stands, without undue bias or prejudgment is difficult and valuable enough.

Writing a coherent argument

"Writing is so much work, do I have to?" Yes, you do. You don't even need to share your writing to get value out it. Say you're a strong chess player. You may even be a strong blind chess player. But playing chess without a board is so much harder. Reasoning is hard, why would you make things harder for yourself by keeping it all in your head?

Now that you've sit down to write, you'll feel an impulse to try and write well. That impulse is a trap, squash it.

Write everything that comes into your head, but only in the first draft. [...] you can remove the parenthetical sentences and the digressions, or you can put each in a note or an appendix [...] — Eco. How to write a thesis. p151

Critic-you interferes with creative-you. I've seen this described as "Babble and Prune" or "doing a vomit pass". Editing prematurely breaks your flow, and inevitably results on a lot of wasted effort when you come back for a second pass on your work. Write badly, and trust future-you to clean up the mess.

Editing

Try to edit from a remove. Wait at least a couple of hours, a day or more if possible. You can't have perspective without distance. You know what you meant by that sentence, you wrote it. Six-months-from-now-you is a total stranger, he doesn't know anything! When it comes to spotting errors, that's a good thing.

[...] you straight-up can't write a good 5-page or more essay in one go. You can get all your thoughts on paper, sure. But you won't be able to pace them out properly. You'll need to take a break then reread it with a fresh perspective to start seeing all the little rough patches [...] it's all about finding that balance between work and breaks. — Blue. How to Read Books and Write Essays. 10:16

On the same token, use spellchecking. Your brain knows what you meant; so after a while it stops showing you the typos.

Reading your writing aloud brings out other kinds of problem.

Once you have a draft, read through it out loud. [...] You'll probably catch some obvious mistakes that way. [...] if you accidentally stumble over a phrase while reading it, that's not just you. That's probably a clunky sentence that could benefit from reworking. — Blue. How to Read Books and Write Essays. 13:10

Finally, a really common advice is to actively seek out feedback. I think that you should seek feedback early or never. Remember that "this is a bad idea, give it up" is possible input. It may even be solid advice and three months into the project you absolutely won't want to hear it. Ask for feedback before sunk costs lock you in. Otherwise? Save everybody some trouble.

Citations

Keep your sources straight. Sources give your reader a way to check your claims. Also, copying someone else's work without attribution is called plagiarism and it's a real dick move. This is important enough that I spent a whole section just setting up the infrastructure. Now it pays off.

There are two things you need to make: a bibliography and a footnote trail. The bibliography is where you put all of your sources together for easy browsing; it is a shopping list for anybody who wants to check your work. Footnotes are where you connect individual ideas or facts with the corresponding source; this way your reader can check one page instead of your entire bibliography.

Ideally, the bibliography would be organized and commentated:

[...] the goal is to organize your bibliography so that it allows readers to identify and distinguish between primary and secondary sources, rigorous critical studies and less reliable secondary sources, etc. — Eco. How to write a thesis. p209-210

In practice, this standard is not met very often. I've tried to meet it myself, and a welcome side-effect is that when I talk about a book, I now know whether I have to do my absolute best when interpreting it, or I can afford to treat it a bit more loosely.

The following table is from chapter 3 of Eco's How to write a thesis. It tells you what information is necessary when you are citing a text. If you are writing for an academic publication, then there's a citation style you have to stick to, so you can ignore it.

Items marked with asterisks are considered essential. Styling (italics, quotation marks, parenthesis, punctuation) and item order should be preserved. Field names ("edition", "publisher") should not be included in the final citation.

Remember that when you cite a work you are aiming to make it as easy as possible for a reader to get a copy of that source. This is why publisher and place matter; getting in touch with the publisher may be the most straighforward way of obtaining a copy. Full author name, full title, and date of publication (of the current edition) are usually enough for recent titles.

For internet-native documents links are not good enough. Websites change and die, links break. If you are citing an internet-native source, make a private copy and request (or host) a public archive copy (as discussed in s4.1.3). Link to both the original URL and the public archive.

Learning from the community

There's always a community. Even dead disciplines have historians and new fields don't pop out of nowhere. You may have to be creative, however, and look for people handling your subject incidentally. For example, if you are interested in statistics, you'll find common ground with people in marketing.

Bibliographic research will point you to other researchers. Get in touch. Conversely, publish your work so other researchers can find you.

When you get in touch, respect people's time. Read beforehand, and learn the discipline's jargon. Besides showing that you've done your homework, it helps you understand other people, and gives you a tool to evaluate their expertise.

If you share an idea, don't oversell it. If it's a bad idea, convincing someone that it's good makes both of you dumber; you want bad ideas to be unpopular and fizzle out. Also, unsolicited advice is usually unwelcome; don't badger.

I'll repeat my earlier advice on requesting feedback: do it early or never. If you wait until you're committed, then you won't change your mind no matter what feedback you receive. It's a waste of everybody's time.

Open questions and further research

References

Main works

Ancilliary material

  1. ^

    NPR.org. Google Entices Job-Searchers with Math Puzzle. 2004. Archived copy at archive.org.
    Kazmierczak, Marcus. Google Billboard Problems. 2004, blog post. Archived copy at archive.org. 

  2. ^

    My girlfriend is currently studying in a different city. We were talking about her coming to visit, but we weren't seeing eye to eye. Eventually we hashed out all of the details of her visit but were still pissed at each other. So we stopped to talk about why we wanted to be together, and reminisce about our first dates, etc. Within ten minutes we were both in tears, and everything was forgiven.

  3. ^

    "[...] your brain kind of tricks you into feeling like you've been doing a thing all day. [...] when I was in university there were days I almost certainly felt like "Oh, God, I've studying all day" but what really happened is that I was probably in the library all day which is a very different thing from studying all day." — CGP Grey and Myke Hurley. Cortex podcast ep45. 2017. https://www.relay.fm/cortex/45. 2:40 

  4. ^

    "The most important thing [for the process of writing] is simply budgeting the time for it. When you get your due date, block in a few hours here and there to sit down and just write. Scheduling is important for three main reasons. First off, you'll be less stressed simply knowing that you have a plan. Two, you're more likely to actually pace yourself and not procrastinate when you follow that plan. And three, [...] multitasking murders your productivity." Blue. How to Read Books and Write Essays. 09:48 

  5. ^

    "Also note that the author catalog is always more reliable than the subject catalog because the act of compiling it does not depend on the librarian’s interpretation, as is the case with the subject catalog." Eco. How to write a thesis. p55

  6. ^

    "[...] querying the subject catalog requires some skill. Clearly we cannot find the entry 'Fall of the Roman Empire' under the letter 'F,' unless we are dealing with a library with a very sophisticated indexing system. We will have to look under 'Roman Empire,' and then under 'Rome,' and then under '(Roman) History.' And if we have retained some preliminary knowledge from primary school, we will have the foresight to consult 'Romulus Augustulus' or 'Augustulus (Romulus),' 'Orestes,' 'Odoacer,' 'Barbarians,' and 'Roman-barbarian (regna).'" — Eco. How to write a thesis. p55

  7. ^

    Feynman, Richard P, Ralph Leighton, Edward Hutchings, and Albert R Hibbs. "Surely You’re Joking, Mr. Feynman!": Adventures of a Curious Character, 1985. "A map of the cat"

  8. ^

    Wentworth. How To Write Quickly While Maintaining Epistemic Rigor. 2021

  9. ^

    "The difference between a coherent argument and a crappy headcanon is evidence. Relativism at its worst is everybody throwing ideas around carelessly, but at its best, it's a rigorous examination of every available idea to determine which ones have ground to stand on. What you end up with isn't the correct answer but one of many valid explanations." Blue. How to Read Books and Write Essays. 08:07

  10. ^

    "[...] how can anybody write a thesis [titled 'The Symbol in Contemporary Thought']? One would have to analyze all of the meanings of 'symbol' in all of contemporary culture, list their similarities and differences, determine whether there is an underlying fundamental unitary concept in each author and each theory, and whether the differences nevertheless make the theories in question incompatible." Eco, Umberto. How to Write a Thesis. p11

4 comments

Comments sorted by top scores.

comment by Elias Schmied (EliasSchmied) · 2022-09-08T14:47:12.909Z · LW(p) · GW(p)

Thanks for this post, this looks very useful :) (it comes at a great time for me since I'm starting to work on my first self-directed research project right now).

comment by Mo Putera (Mo Nastri) · 2022-09-08T01:59:18.314Z · LW(p) · GW(p)

This post is great, I suspect I will be referencing it from time to time.

I don't know if you meant to include the footnotes as well, since they aren't present in this post. For instance, I tried clicking on

After a week, you'll likely remember why you started, but it may be hard to bring yourself to really care[2] [LW · GW]

and it just doesn't lead anywhere, although I did find it on your blog.

Replies from: pablo-repetto-1
↑ comment by Pablo Repetto (pablo-repetto-1) · 2022-09-08T12:49:35.490Z · LW(p) · GW(p)

I'm glad you like it!

Fixed the footnotes. They were there at the end, but unlinked. Some mixup when switching between LW's Markdown and Docs-style editor, most likely.

comment by Zian · 2022-09-09T04:38:40.107Z · LW(p) · GW(p)

"Learning to research out of desperate need is like learning to drive on the way to the hospital."

Fully agreed as to the greater point but the flip side is that in real life, if the need is desperate, then entire chunks of this article can be skipped.

For example, if aliens will murder you if you don't tell them the correct name of the " Tiffany" in the CGP Grey poem, then you can blurt out "Theofania" and collapse into a heap in bed. No need to write anything

Similarly, if you are bleeding profusely from a stab wound, then it is probably sufficient to find the nearest competent emergency room and skip the step of quizzing a librarian.

Finally, so as to avoid creating too many strawmen, if you have an acute disease, then finding a few very reliable summary articles and convincing a doctor to help you implement the steps may have excellent odds of curing the disease. For example, there are many protocols for diseases.