Film Study for Research

post by jsteinhardt · 2021-09-14T18:53:25.831Z · LW · GW · 7 comments

Contents

  Film Study
  Act-Reflect-Ask
  Summary
None
7 comments

Crossposted from my blog, Bounded Regret.

Research ability, like most tasks, is a trainable skill. However, while PhD students and other researchers spend a lot of time doing research, we often don't spend enough time training our research abilities in order to improve. For many researchers, aside from taking classes and reading papers, most of our training is implicit, through doing research and interacting with mentors (usually a single mentor--our PhD advisor or research manager). By analogy, we are like basketball players who somehow made it to the NBA, and are now hoping that simply playing basketball games will be enough to keep improving.

Drawing on this analogy, I want to talk about two habits that are ubiquitous among elite athletes, that have analogs in research that I feel are underutilized. Those who do pursue these habits as PhD students often improve quickly as researchers.

The first habit is film study. Almost every high-level athlete watches films of other players of the same sport, including historical greats, contemporary rivals, and themselves. This allows them to incorporate good ideas from other players' games as well as to catch and eliminate flaws in their own game. Even the very best players benefit from watching film of themselves and others.

The second habit, which I call act-reflect-ask, occurs in the course of a game or scrimmage. I'll describe this from my own experience (although I'm by no means an elite athlete, I've learned this from people who are). After a point ends, I generally think about what happened during the point--Was there anything I wanted to do better? Did anything unexpected happen? Then I'll re-run those parts in my head, simulating what I would have done differently until I feel like I know how to consistently make the right decision. In some cases, I can't figure it out--perhaps I was playing defense, someone beat me, and I can't figure out what they did or can't figure out the counter. In that case I'll ask a teammate about it (or the person who beat me, if it's a friendly scrimmage) and talk it over until I see the right strategy for the future.

Both of these strategies are invaluable for improving. They leverage the fact that as humans, we tend to learn socially: we are very good at adopting strategies from others, so film study and asking are efficient ways to learn. Both strategies also lead to deliberate practice focused on real-world contexts. Below, I'll show that these strategies have analogs in research, and argue that good researchers should adopt both into their own habits.

Film Study

As mentioned above, good athletes watch lots of film of other athletes. This extends to other skills as well--most chess players, including grandmasters, study games by both contemporary and historical greats. They do this to understand how other very strong players play, in order to adopt ideas and, in the case of rivals, to counter those ideas (this part is less relevant to research). Even the very best players do this.

What is the equivalent to this in research? Ideally, we would watch world experts as they work, observing how they think, perform experiments, and so on. Unfortunately, this is difficult--much research work is internal rather than external, and we don't routinely film great researchers in the same way as we do with athletes. The closest obvious analog is working closely with a mentor, as many PhD students do with their PhD advisor. Then, it is often possible to see first-hand how a more experienced researcher approaches a problem. However, this isn't scalable, and most people only get to do this with one person--their advisor. (As an aside, it is very useful for students to develop a good model of their advisor's thinking style--I think this tends to be underrated.)

A more scalable approach would be reading papers, but this doesn't achieve the full goal of film study--you only see the finished product, rather than the thought process, and it tends to only show the part of a writer's thoughts that are widely defensible. What we want is a public record of someone's thoughts, including off-the-cuff thoughts that wouldn't make it into a paper.

In fact, we do have this, in the form of blogs. The right type of blogs can provide a valuable form of "film study". I personally learned a lot about statistics from Andrew Gelman's blog. Often, someone sends him a paper and he just gives his off-the-cuff reactions to it: what he liked and didn't, what was convincing, what parts seem sketchy. I probably learned more from reading his blog than from statistics classes (of which I've taken embarrassingly few, yet somehow managed to get hired by a Statistics department; I'll credit Gelman for this). Scott Aaronson's blog is good in the same way for theoretical computer science. Many posts on the GiveWell and Open Philanthropy blogs are good in this way, too. In all cases, I'd look at the earlier rather than later posts (though not the very earliest); the reason is that once blogs have too large an audience, writers start to feel constrained to write more "professionally" and you get less of the valuable off-the-cuff thinking.

In addition to blogs, debates are another good source of off-the-cuff, in-the-moment thinking, as long as the participants don't overprepare and as long as they are trying to make good arguments rather than score rhetorical points. Actually, the best debates I've seen also take part via blogs, such as the debate over de-worming in global health. Seminars can be good film study, but are primarily film study for giving presentations rather than doing research (and for this, also watch recordings of great talks online). Seminar Q&A can be good film study for research thinking, as long as participants are opinionated and express those opinions in a clear way that exposes their underlying mental model. For programming, you can watch people code on Twitch, or pair program with other students in your research group.

The above are all useful sources of in-the-moment thinking. For research, we also make decisions--such as what directions to pursue--that have consequences on the scale of years. To film study these, I read histories of important scientific developments. Good histories will follow individuals around in detail for an extended period of time, ideally with primary sources. For instance, The Making of the Atomic Bomb covers developments in physics up to and through the Manhattan project, and discusses many of the decisions, discoveries, and dead ends faced by Fermi, Szilard, Oppenheimer, and others. (The dead ends are especially important, so that you can see the whole process and not just what is useful today.) Another great example is The Eighth Day of Creation, which does the same for the development of modern biotechnology. Such histories have helped me gain a better understanding of how science develops on the scale of years or decades, which I would otherwise have to learn the hard way, over my own years and decades of research.

Some other miscellaneous advice: transcripts of talks can sometimes be good in the same way as blogs. Richard Hamming's "You and Your Research" is excellent on this front. For talks, recording yourself and watching the recording may be the fastest route to improvement. Finally, in addition to histories, case studies (often taught in law or business courses) also provide information that would be expensive to gather otherwise.

In summary, film study blogs for off-the-cuff research thinking; watch great presentations and record yourself to learn how to speak; pair program and watch programming streams; and read histories of science for long-term research decisions.

Act-Reflect-Ask

In the act-reflect-ask loop, we reflect on whether something could have gone better after we do it, and ask someone else if we can't figure it out. There are many ways to do this in research:

• When seeing a proof, if you don't see how you would have come up with the proof yourself, discuss with others how to do so (this is usually what people mean when they ask “what's the motivation for that step?”). The same goes whenever you see a cool experiment or idea that you're not sure you would have come up with yourself. First try to think about whether there's a way to modify your thought process to reliably come up with such ideas in the future. If not, discuss with the presenter so that you can learn.
• After you give a talk, pull aside one of the audience members and get feedback on what worked/didn't work in the talk.
• After attending a seminar, discuss what was or wasn't convincing, what was most interesting, etc. Paper reading groups are valuable as they often focus on this. (This isn't quite act-reflect-ask since the seminar was given by someone else; but you can think of it as a way of checking your own thoughts during the seminar against others'.)
• Every week, reflect on what things felt less efficient than they needed to be. Think for yourself how to improve these, then talk to friends, colleagues, or mentors to get additional ideas.

In addition to helping yourself improve, these habits help others as well--asking someone for advice engages their own thinking in a growth-oriented direction, so by helping you they are likely improving themselves, too. This also helps at the level of teams, as it builds chemistry and creates a shared culture of excellence and growth. Indeed, in sports, the best teams do this regularly, and veteran players are proactive in finding ways to help younger players. Some professional players even stay in a league, making millions of dollars a year, solely by being excellent sources of advice and mentorship.

Summary

Find ways to routinely study research decision-making, through blogs, seminars, video streams, and histories. Actively consume these to adopt and build up effective mental heuristics. Whenever you do something, reflect on how it could be better, and ask others for advice. As you learn more yourself, find ways to give back to others. Consistently doing these will help you to become a better researcher over time, and contribute to a culture of excellence among those around you.

7 comments

Comments sorted by top scores.