Lessons I've Learned from Self-Teaching

post by TurnTrout · 2021-01-23T19:00:55.559Z · LW · GW · 74 comments

Contents

  Make Sure You Remember The Content
    Random Anki tips
  Read Several Textbooks Concurrently
  Completing The Whole Textbook Is Usually a Big Waste of Time, Please Don't Do It
  Read Easier Textbooks Instead of Struggling Valiantly
  Be Comfortable with Approximate Models
  Conclusion
None
74 comments

In 2018, I was a bright-eyed grad student who was freaking out about AI alignment. I guess I'm still a bright-eyed grad student freaking out about AI alignment, but that's beside the point. 

I wanted to help, and so I started levelling up [? · GW]. While I'd read Nate Soares's self-teaching posts [LW · GW], there were a few key lessons I'd either failed to internalize or failed to consider at all. I think that implementing these might have doubled the benefit I drew from my studies. 

I can't usefully write a letter to my past self, so let me write a letter to you instead, keeping in mind that good advice for past-me may not be good advice for you.

Make Sure You Remember The Content

TL;DR: use a spaced repetition system like Anki. Put in cards for key concepts and practice using the concepts. Review the cards every day without fail. This is the most important piece of advice.

The first few months of 2018 were a dream: I was learning math, having fun, and remaking myself. I read and reviewed about one textbook a month [? · GW]. I was learning how to math, how to write proofs and read equations fluently and think rigorously. 

I had so much fun that I hurt my wrists typing up my thoughts on impact measures. This turned a lot of my life upside-down. My wrists wouldn't fully heal for two years [LW(p) · GW(p)], and a lot happened during that time. After I hurt my wrists, I became somewhat depressed, posted less frequently, and read fewer books. 

When I looked back in 2019/2020 and asked "when and why did my love for textbooks sputter out?", the obvious answer was "when I hurt my hands and lost my sense of autonomy and became depressed, perchance? And maybe I just became averse to reading that way?"

The obvious answer was wrong, but its obvious-ness stopped me from finding the truth until late last year. It felt right, but my introspection had failed me.

The real answer is: when I started learning math, I gained a lot of implicit knowledge, like how to write proofs and read math (relatively) quickly. However, I'm no Hermione Granger: left unaided, I'm bad at remembering explicit facts / theorem statements / etc. 

I gained implicit knowledge but I didn't remember the actual definitions, unless I actually used them regularly (e.g. as I did for real analysis, which I remained quite fluent in and which I regularly use in my research). Furthermore, I think I coincidentally hit steeply diminishing returns on the implicit knowledge around when I injured myself

So basically I'm reading these math textbooks, doing the problems, getting a bit better at writing proofs but not really durably remembering 95% of the content. Maybe part of my subconscious noticed that I seem to be wasting time, that when I come back four months after reading a third of a graph theory textbook, I barely remember the new content I had "learned." I thought I was doing things right. I was doing dozens of exercises and thinking deeply about why each definition was the way it was, thinking about how I could apply these theorems to better reason about my own life and my own research, etc.

I explicitly noticed this problem in late 2020 and thought, 

is there any way I know of to better retain content? 

... gee, what about that thing I did in college that let me learn how to read 2,136 standard-use Japanese characters in 90 days [LW(p) · GW(p)]? you know, Anki spaced repetition, that thing I never tried for math because once I tried and failed to memorize dozens of lines of MergeSort pseudocode with it? 

hm... 

This was the moment I started feeling extremely silly (the exact thought was "there's no possible way that my hand is big enough for how facepalm this moment is", IIRC), but also extremely excited. I could fix my problem! 

And a problem this was. In early 2020, I had an interview where I was asked to compute . I was stumped, even though this was simple high school calculus (just integrate by parts!). I failed the interview and then went back to learning algebraic topology [? · GW] and functional analysis [LW · GW] and representation theory. You know, nothing difficult like high school calculus. 

I was pretty frustrated with myself.

It's not that I didn't understand it [LW(p) · GW(p)]. I just didn't remember it, especially on the spot. The worst part was that I had brushed up on calculus the previous spring, and I still didn't remember it. Turns out that my brain won't remember material it doesn't use for months on end, even if forgetting that material would be embarrassing.

Enter Anki, an amazing spaced repetition system ($20 for iOS, free for computer). The way I like to explain Anki is: 

Anki is a flashcard application into which you can enter a constant number of cards each day while retaining a constant average daily workload. You can add cards each day, without having to study longer and longer to get through all of the cards.

I currently think that unless you have really good memory or you're not learning content you want to remember months from now, you're making a mistake by not using a spaced repetition system. Read Gwern for more on spaced repetition.

Spaced repetition seems especially useful for students. In college, I ran an experiment: for an upper-level French class, I put things I didn't know how to say into Anki, reviewed daily, and otherwise didn't study at all. I got an A. 

How powerful might a bright 6th grader grow, were they to use Anki every day for their whole life? The best time to plant a tree may have been in sixth grade, and the second-best time may have been in seventh grade, but you should still plant the tree now rather than never. 

I've been using Anki for math for the last 71 days, and I currently have a deck of about 900 cards which I study for ~30 minutes daily. In 2018, I spent about 10 minutes daily reviewing a deck of nearly ten thousand French cards.
If I were to add no more cards, daily reviews drop off quickly. 

(Once completed, the reviews in the next few days will be pushed into some of the future days, so this projection is slightly optimistic, but you get the point.)

I love Anki, and I was foolish to circumscribe it to language-learning. I now use Anki to remember key concepts from academic talks, LessWrong blogposts, and yes - textbooks. Which I now love again, and which I read for ~an hour daily again, because I'm actually retaining the content

Measure theory, ring theory, random stuff about taxonomy and epidemiology, quantum mechanics, basic physics, deep RL papers, they all go into Anki, and Anki cards go into my brain - and stay there! 

What a wonderful time to be alive. 

Random Anki tips

I know quite a bit about how to best use Anki, so if you try this and it doesn't seem to work, please message me instead of banging your head against the wall or giving up!

Memorizing definitions can be useful: when reading a text, it saves you from having to constantly check what the concept is. Make sure to include examples to work through - don't just toss in random definitions you're barely interested in and will never think about again. 
The math here isn't important. The key thing is, I want to remember the "pseudorandom number generator" concept, and so I find an interesting result and make myself prove it. The proof isn't too long, which is key—just a few cloze deletions. Don't try to memorize essays with Anki. 

Am I ever going to actually use this math for my research? Probably not. Doesn't matter. Anki makes it cheap to learn and retain things.
If you want custom commands, use the card type editor.

Read Several Textbooks Concurrently

TL;DR study several topics at once so that your brain has time to cement the concepts you're learning, before the text builds on those concepts further.

AllAmericanBreakfast's recent post is great, so I'll refer you to that to make this point:

Wait, what? You want me to make life easier on myself by, instead of studying calculus...studying calculus, linear algebra, and statistics all at once?

~ AllAmericanBreakfast, The Multi-Tower Study Strategy [LW · GW]

The basic idea is that your brain needs time to really cement a new idea in, and so you should study several topics at once in dependency-heavy areas like mathematics. This advice matches up with both my recent personal experience and with advice I received early on from Qiaochu Yuan [LW · GW], but which I had unfortunately ignored. 

For example, right now I'm reading Nielsen and Chuang's Quantum Computation and Quantum Information, Evan Chen's The Infinite Napkin, and a ridiculously easy physics book, Kuhn and Noschese's A Self-Teaching Guide: Basic Physics (more on this later). Previously, I'd been going back through Wasserman's All of Statistics [LW · GW] and Pearl's Causality after reading Pearl's Book of Why

I always feel like I'm learning something new instead of banging my head against the wall. Sometimes you should just read one book, but if you don't need to cram, I recommend diversifying. 

Completing The Whole Textbook Is Usually a Big Waste of Time, Please Don't Do It

TL;DR extract the most useful / central concepts and remember them forever via Anki. This doesn't require grasping every arcanum, every detail of a textbook. 

When I started reading textbooks, I completed the whole book. I didn't want to miss a crucial concept. But the thing about crucial concepts is that they pop up everywhere. If you missed a crucial concept, you'll know. You're not going to wake up 20 years from now and be like, "OH NO! I forgot to learn about 'force' when I self-studied physics! And I forgot to learn about injectivity in linear algebra!"

As you read more, you'll get a taste for what's probably important, and what's details you can reference later if need be. You can also ask experts if you should study part of a book - this is one key benefit of having an actual teacher. 

But don't complete the whole book and all of its exercises, if you just want to become a polymath. Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on. 

Another reason I used to do this a lot was that I wanted to look good by being able to mention on e.g. my resume that I had read the whole book. It's a lot more impressive to say "I read these 10 textbooks" than "I read large parts of this book, and some of this book, and a little of this one, and a lot of this other book." I've found that learning well requires keeping an eye out for these instincts. My brain is not my friend in this battle.

I now often ask myself "am I doing this, at least in part, in order to look good?". Sometimes I answer 'yes', and sometimes I do it anyways - wanting to look good isn't always bad. But there are sometimes things more important than looking good [LW · GW]. 

Read Easier Textbooks Instead of Struggling Valiantly

TL;DR even though slogging through tough textbooks makes you feel sophisticated and smart, don't. 

Imagine I'm learning how to program, but I've never used a computer before. Learning to program while learning to operate a computer, will probably take longer than learning to operate a computer and then learning to program. Learning time is superadditive in terms of your ignorance / the dependencies you're missing. 

But it's worse than that. Imagine I'm learning quantum mechanics, but I don't know any linear algebra either. I'm now trying to do three things:

  1. Learn linear algebra,
  2. Learn the formal postulates of quantum mechanics, and
  3. Tie all of this into the real world.

Similarly, if I'm trying to learn fluid mechanics without knowing how to manipulate partial differential equations (PDEs), it might look trying to simultaneously

  1. Learn PDEs,
  2. Learn the physical equations, such as Navier-Stokes, and
  3. Tie all of this into the real world to explain what I already know about e.g. water and rivers and blood pressure.

But what if instead I picked up some dumb book that doesn't even have any calculus, and let it give me approximate explanations via e.g. Archimedes' principle:

Archimedes' principle states that the upward buoyant force that is exerted on a body immersed in a fluid, whether fully or partially, is equal to the weight of the fluid that the body displaces. (Wikipedia)

I breeze through this book no problem, and I can see how to tie in these laws to explain my intuitive models: "logs float more easily than rocks because rock is denser than wood, and so the buoyant force from Archimedes' principle is enough to support the weight of a log." So I'm taking care of point #3, "tie this content into the real world." 

Then, suppose I learn about PDEs and become comfortable with them. Now all I need to do to learn a piece of fluid mechanics is to learn the relevant physical equation, and then think about how it implies things like Archimedes' principle. Crucially, via this method, I'm only confused about one thing at a time. Build models in the right order [LW · GW]!

Be Comfortable with Approximate Models

TL;DR allow yourself to learn things in order of comprehensibility: don't try to learn general relativity before Newton's law of gravitation.

In 2019, I wanted to learn a bit of chemistry. I got my hands on a high school chemistry textbook. I got stuck on chapter two because I was being too strict about forming gears-level models [LW · GW].

I was stuck because I was thinking about electron shells. The book acknowledged the Bohr model of the atom was wrong, electrons aren't really discrete particles orbiting the nucleus:

A wrong model.

But I got nerd-sniped into trying to understand how the electron standing wave only has solutions for certain energy levels, which is a result of quantum mechanics (or so I remember reading). I couldn't understand why, and so the knowledge felt "fake." 

It wasn't like I explicitly reasoned "this model is wrong and so I'm not going to keep reading this book", it felt more like... I just wasn't that hungry to learn this, because it wasn't "real." And I gradually stopped returning to that book.

Learn things quickly, note your confusions, and correct them later when the Anki cards show up again. Let yourself learn approximate models of reality.

Conclusion

I knew about the "read easier textbooks [LW · GW]" advice already, but I didn't apply it. Perhaps I just didn't recognize a chance to apply it. The same forces of chaos and entropy and madness which prevented my applying e.g. Luke Muelhauser's advice, may prevent you from applying this post's advice. If you think any of this advice might help you, I recommend setting up a plan now for how and when you'll implement it.

74 comments

Comments sorted by top scores.

comment by Adam Zerner (adamzerner) · 2021-01-25T01:19:32.547Z · LW(p) · GW(p)

use a spaced repetition system like Anki.

Spaced repetition/Anki is one of those things that seems obvious and like it makes perfect sense. However, my experiences have been pushing me further and further away from thinking that it is usually something that is useful.

I spent a year or so self-studying full time. I also spent a few months studying Leetcode stuff for FAANG interviews half-time. In both experiences I spent months using Anki, perhaps 100-200 cards where I would study the cards pretty consistently every day.

I know this might go against the research on spaced repetition, but I've found that the stuff I used to study often just doesn't stick. On the other hand, there are things that I never really did spaced repetition on that have stuck remarkably well. My impression and experiences is that obtaining a deep understanding is a lot more fruitful than doing spaced repetition.

Also, thinking along the lines of a sanity check and the outside view, are there examples of people becoming very successful due to Anki? My impression is that it doesn't happen all too often, and when it does it is because Anki happens to be a great fit for the situation, eg. passing med school exams. 

Completing The Whole Textbook Is Usually a Big Waste of Time, Please Don't Do It

Intuitively, this makes a lot of sense, and is the mindset I've had for a while. I agree that the end worth seeking are those central concepts rather than memorizing certain details. However, I think that playing around with those annoying details might be a good means to the end of grasping central concepts. I spent a lot of time telling myself, "It's only the big concepts that matter, so you can skip this section, you don't have to answer these questions, you don't have to do these practice problems."

Well, it's also true that you could take memorizing details too far (I'm looking at you organic chemistry). I guess it's just a matter of judgement, and that the insight of "wait, it's really only the big concepts that I need to care about" is likely to lead to failure modes where you undervalue the details.

Read Easier Textbooks Instead of Struggling Valiantly

Strongly agree. Seems like a failure mode that is common. I love "X for Dummies" types of books/posts. Reminds me of Explainers Shoot High, Aim Low [LW · GW]:

A few years ago, an eminent scientist once told me how he'd written an explanation of his field aimed at a much lower technical level than usual.  He had thought it would be useful to academics outside the field, or even reporters.  This ended up being one of his most popular papers within his field, cited more often than anything else he'd written.

Be Comfortable with Approximate Models

Reminds me of the blurry-to-sharp approach:

Replies from: eric-siggy-scott, Vaniver, TurnTrout, shorty25h0r7
comment by Eric 'Siggy' Scott (eric-siggy-scott) · 2021-01-25T14:01:57.990Z · LW(p) · GW(p)

"are there examples of people becoming very successful due to Anki?"

It's hard to answer that kind of question, because there are plenty of examples of people becoming successful without SRS, and people who use SRS don't only use SRS.

Personally, I use Anki for professional development (so not to pass exams, but for long-term mastery).  My biggest topics are math, algorithms (ex. machine learning research topics), and programming.

It's got a number of advantages.  Sometimes it helps by keeping technical details (like trig identities or programming syntax) close to the surface, so I can finish a project faster or plow through whiteboarding a problem with a colleague.  Other times it helps by cementing a deep conceptual intuition strongly, so that I can focus on doing more advanced things (instead of rehashing the same slippery basics).  Or by just increasing the breadth of what I know is available, so I don't have to agonize over the right Google terms. Sometimes it helps by making it easier to remember people's names, accelerating my ability to network (I recommend mnemonics here, since names are rote memorization, and rote memorization is difficult with Anki).

But above all, it ensures that I can keep learning, even if I only have 20 minutes a day to spare for deep/slippery topics like quantum computing; or 10 minutes a day to capture a fleeing professional experience that would otherwise be in one ear/out the other.  I'm just 32 years old, but that's old enough that I've forgotten much more than I've learned.  Anki changes that: it acts like compound interest for knowledge.  This spring (after a few years of regular SRS use) I'll reach 40,000 cards, with no sign of slowing down.

It's hard to reproduce that kind of growth with traditional learning methods.

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2021-01-25T17:36:19.861Z · LW(p) · GW(p)

Hm, I seem to have different experiences than you have had.

Sometimes it helps by keeping technical details (like trig identities or programming syntax) close to the surface, so I can finish a project faster or plow through whiteboarding a problem with a colleague.

My experience as a programmer (from myself and from talking to others) is that you pick up syntax pretty quickly. For a new language there's a learning curve. Perhaps SRS would help with that, although that doesn't seem too useful. Eg. I use JavaScript every day and rarely use Python. I could use SRS for Python. Alternatively, I could wait until I need to use Python, take a few days getting familiar with the syntax, and then be fine, which seems better.

Other times it helps by cementing a deep conceptual intuition strongly

Perhaps my memory is failing me, but I can't recall doing SRS over and over again ever cementing a deep conceptual intuition for me. I think looking at the same thing from different angles is often helpful, but with SRS, a given card is largely looking at the concept from one angle.

Here's an example. I'm learning functional programming right now. In functional programming, operators like + and * are just functions. Instead of 2 + 3 you can do (+) 2 3 just like you'd do add 2 3. Similarly, you could call functions using infix notations by using backticks like this 2 `add` 3 and it behaves like an operator. This really cemented for me what an operator is in such a way that I don't think SRS ever could, and it did so much faster.

But above all, it ensures that I can keep learning, even if I only have 20 minutes a day to spare for deep/slippery topics like quantum computing; or 10 minutes a day to capture a fleeing professional experience that would otherwise be in one ear/out the other... It's hard to reproduce that kind of growth with traditional learning methods.

It sounds like your point is moreso that it's hard to be as productive without SRS, but I'd like to note that there are other ways you can use a spare 15 minutes to learn rather than SRS.

Replies from: AllAmericanBreakfast
comment by DirectedEvolution (AllAmericanBreakfast) · 2021-01-25T19:58:46.471Z · LW(p) · GW(p)

There's definitely a tradeoff between breadth and depth/speed. On a real-world project, you can attain great speed and/or great depth on the narrow set of techniques/concepts that are directly relevant to the work you do every day.

It's very expensive to maintain that kind of fluency. I've never done any programming in a language that uses infix vs. prefix operators, yet I know the difference just from a couple minutes total studying the concept at a few different widely-spaced points in time. This concept would not pose a challenge to me were I to pick up a functional programming language, although it would take time to build a habit of using them.

When you overinvest in study-through-practice in a concept you've already mastered, you will forget a lot of the more broad knowledge. Anki gives you a way to retain it, or to get it back when you're ready for it. If you see yourself as having no need for a broad knowledge-base -- if you see your career as being a happy code monkey banging out programs that are within the wheelhouse you've already established, then you're probably good.

But if you have a different vision for your career, it's possible that broad knowledge will be really helpful. And spaced repetition/Anki gives you a set of tools to build and maintain that broad knowledge-base.

I'm really glad you posted this objection to Anki, because I think it's probably common. It's also a fair point: sometimes, we don't care about building a broad knowledge base. We're just trying to become fluent in the narrow set of skills that let us execute a technical project. Clarifying that distinction is very valuable from the perspective of budgeting your study time wisely.

comment by Vaniver · 2021-01-25T19:29:23.320Z · LW(p) · GW(p)

Interestingly, this comment made me more excited about using Anki again (my one great success with it was memorizing student names, which it's well-suited for, and I found it pretty useless for other things), because this comment has a great idea with a citation that I probably won't be able to find again unless I remember some ancillary keywords (searching "blurry to sharp" on Google won't help very much). But if I have it in an Anki deck, not only will be it more likely to be remembered, but also I'll have the citation recorded somewhere easy to search.

Replies from: TurnTrout
comment by TurnTrout · 2021-01-25T20:11:24.495Z · LW(p) · GW(p)

my one great success with it was memorizing student names

+1, I used Anki before my CFAR workshop. had to remind myself not to call people by their names before they even had their nametags on. This was great, because remembering names is generally a source of very mild social anxiety for me. 

Replies from: FinalFormal2
comment by FinalFormal2 · 2022-03-04T14:34:37.780Z · LW(p) · GW(p)

How do you do that? Do you take their picture beforehand or something?

Replies from: TurnTrout
comment by TurnTrout · 2022-03-22T16:51:01.267Z · LW(p) · GW(p)

There was a names/faces doc circulated before the workshop.

comment by TurnTrout · 2021-01-25T14:48:44.896Z · LW(p) · GW(p)

Thanks for all of these thoughts!

My impression and experiences is that obtaining a deep understanding is a lot more fruitful than doing spaced repetition.

I don't see why these should be mutually exclusive, or even significantly trade off against each other. This sounds like saying "drinking water is more important than getting good sleep", which is also true in a literal sense, but which implies some kind of strange tradeoff that I haven't encountered. I aim to get a deep / intuitive understanding, and then use Anki to make myself use that understanding on a regular basis so that I retain it.

However, I think that playing around with those annoying details might be a good means to the end of grasping central concepts. I spent a lot of time telling myself, "It's only the big concepts that matter, so you can skip this section, you don't have to answer these questions, you don't have to do these practice problems."

I agree, and I'm not trying to tell people to not sweat the details. It's more like, you don't have to do every single practice problem, or complete all the bonus sections / applications which don't interest you. Which is a thing I did - e.g. for Linear Algebra Done Right [? · GW], and then I forgot a lot of linear algebra anyways.

Another approach is, you start with the papers you want to understand, and then backchain into the concepts you have to learn in a big tree. Then you're guaranteed not to waste time. I haven't tried this yet, but it sounds very sensible and I mean to try it soon.

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2021-01-25T17:47:16.239Z · LW(p) · GW(p)

I don't see why these should be mutually exclusive, or even significantly trade off against each other... I aim to get a deep / intuitive understanding, and then use Anki to make myself use that understanding on a regular basis so that I retain it.

Because that time you spend using Anki to retain it is time that you could spend seeking deep understandings elsewhere.

It's more like, you don't have to do every single practice problem, or complete all the bonus sections / applications which don't interest you.

Yeah, that makes sense. Sounds like we made opposite mistakes :)

Another approach is, you start with the papers you want to understand, and then backchain into the concepts you have to learn in a big tree. Then you're guaranteed not to waste time. I haven't tried this yet, but it sounds very sensible and I mean to try it soon.

I've tried that. I think it's an approach to have in your toolkit, but for me it's not really my style.

Eg. in learning Haskell I've been trying to skim a book, then work on practice problems, then backtrack when I notice a concept I haven't learned yet that I need to solve the problem. I always seem to get lost/overwhelmed in doing the backtracking. So I pivoted to reading the 1000+ page Haskell Programming from First Principles book, which is exactly what the title implies, and it's going well for me.

Also from a more motivational perspective rather than a pedagogical one, I find that I can get into a flow state more easily with the first principles approach. With the backtracking approach, I often get stuck. With the first principles approach, it may be slow at times, but at least I'm continuing to take small steps forward and not getting stuck.

Replies from: TurnTrout, sinitsa
comment by TurnTrout · 2021-01-25T18:24:37.656Z · LW(p) · GW(p)

Because that time you spend using Anki to retain it is time that you could spend seeking deep understandings elsewhere.

Right, but part of having deep understandings of things is to have the deep understanding, and if I just keep acquiring deep understanding without working on retention, I'll lose all but an echo. If I don't use Anki, I won't have the deep understanding later. I therefore consider Anki to help with the long-term goal of having deep understanding, even if moments spent making Anki cards could be used on object-level study.

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2021-01-25T18:45:55.466Z · LW(p) · GW(p)

I guess the crux is about Anki helping with retention. If you'd lose a lot of your understanding without Anki, I agree it's worthwhile. If you'd only lose a little, it seems like a better use of time to seek deep understandings elsewhere.

comment by sinitsa · 2022-04-09T10:17:03.678Z · LW(p) · GW(p)

Because that time you spend using Anki to retain it is time that you could spend seeking deep understandings elsewhere.

In my personal experience, a deep-understanding seeking study session is not portable (hard to take notes when you don't have a surface) and if it's less than an hour long, it's usually not very good (need time to 'warm up' and fully focus). Anki, on the other hand, is perfect for quick study during a break/short commute/being in a queue. 

Also creating flashcards in itself makes a great interactive reread of things.

comment by manunamz (shorty25h0r7) · 2021-07-17T19:26:01.179Z · LW(p) · GW(p)

+1 for this response. I particularly like the blurry-to-sharp nugget, to which I wanted to add: 

"low-res-breath > high-res-depth"
 

comment by lsusr · 2021-01-23T19:19:15.695Z · LW(p) · GW(p)

There is a lot of material I like in this post.

  • The idea to use Anki for math is something I've considered but never figured out a good way to do because the natural fact unit of advanced math is more complicated than a the definition of a natural language word. I would be happy to hear details more about how you did this.
  • The tips on textbooks are useful. I have never read a textbook cover-to-cover (except athletic training textbooks) even though I have tried many times. It is reassuring to hear this is not important. It is also useful to hear the advice about reading easy textbooks. The value of reading easy textbooks is obvious in retrospect but something I never figured out on my own before reading this post.
Replies from: TurnTrout, Mo Nastri
comment by TurnTrout · 2021-01-23T20:45:25.305Z · LW(p) · GW(p)

EDIT: I now consider these example cards to be too long, although somewhat functional (pun intended). I recommend using cloze deletions instead, with small cards that are a couple of sentences long. The overlapping cloze add-on is also useful for remembering proofs in order, or sequential definitions which are hard to break into several cards. 

The idea to use Anki for math is something I've considered but never figured out a good way to do because the natural fact unit of advanced math is more complicated than a the definition of a natural language word. I would be happy to hear details more about how you did this.

I didn't really have trial and error when I started doing this in November, I just dove in and it worked. I had, after all, experience making well over ten thousand Anki cards. The key is to only make new cards composed of concepts which you already understand. I'll try to unpack how I make math cards, though, by making a new card and introspecting on what my instincts are.

Let's say I'm trying to make a card for "what is a mathematical function?" (this isn't "advanced math", but I find that my card-entry approach remains effective as the material gets harder). I'm first going to show the card I made for this just now.

In writing the flip-side, I notice the following mental events, in order:

  1. I want to motivate why I cared about this concept in the first place. I want to think of an intuitive situation representing a mathematical function.
    1. Different types of cards may not need motivations, but I generally like to motivate definitions. Why on earth did I even put in the effort to click 'new card' to enshrine this concept? What new things does this definition let me compactly discuss?
  2. I want to find a picture illustrating a function - some quantity being "transformed" into some other quantity.
    1. If I'm a computer scientist, I notice that I want to clarify that mathematical functions don't have dynamics - they're just input-output pairs. See my first textbook review ever [? · GW]:

      "Functions  are just static sets of ordered pairs  . They are not dynamic indexing functions, they do not perform efficient lookup, please do not waste an hour of your life trying to figure out how you could do such a thing within the simple machinery afforded by set theory up to that point."
  3. I want to write the formal definition.
  4. I want to check that I've done a good job with the card - the card contains a new insight but explains it using concepts, theorems, and formalisms which I already understand.
    1. In this case, if I didn't understand sets, I wouldn't have used sets. Instead, I'd just say "to each value  in its domain , a  associates to  a single value in its range . For example,  associates the number  to each real number  in its domain.

Once I'm done with the definition card, I now want to add in a few more example problems. I would look up "exercises to understand mathematical functions" and put the best problem / answer pairs in. 

After that, I want to make sure I didn't make any obvious conceptual mistakes. I'm comfortable that I didn't in this situation, but that's because the real me already understands functions; I generally don't already deeply understand concepts for which I make cards. I might do something like search "most common misconceptions which students have about mathematical functions." (I often don't do this step, but it can be a good idea) 

Also, I have a "conflicting cards" deck for cards which are either too complicated for now (I need to learn what a ring is before learning what a ring ideal is) or too similar to another concept I'm presently learning (don't try to learn accuracy vs recall vs specificity vs  score all at once). I'll add those cards to the main deck later, after I've fully learned the current set of concepts.

More advanced math

Here's a card I have that's a little longer and less atomic:

This is only two-thirds of the card! I'm not memorizing this entire thing. Instead, there's a felt sense that corresponds to walking through the definition. I remember that I want to define Lebesgue integration for indicators, and then for simple functions, and then take a supremum in a certain way. Then I think about special cases, and hopefully remember to specify what happens if 

The point to this card is that if I want to be able to walk through the concepts in the right order, with the understanding that I could expand the concepts into full formal detail if needed (since I understand the subconcepts of e.g. the Lebesgue measure, simple functions, measurable sets, etc). I have the prereqs, and I just need to remember how to put them together. 

Then I just add philosophical concepts and clarifications at the end, in case I get confused later / want to just add notes.

(Here's a recent version of my Anki/math/physics deck, so feel free to look through it)

Replies from: Lanrian, lsusr, Gurkenglas, ErrethAkbe
comment by Lukas Finnveden (Lanrian) · 2021-01-23T22:39:52.224Z · LW(p) · GW(p)

I'm really surprised at how big your cards are! When I did anki regularly, I remember getting a big ugh-feeling from cards much smaller than yours, just because there were so many things that I had to consciously recapitulate. It was also fairly common that I missed some little detail and had to choose between starting the whole card over from scratch (which is a big time sink since the card takes so much time at every repeat) or accept that I might never remember that detail.

I'm super curious about your experience of e.g. encountering the function question. Do you try to generate both an example and a formalism, or just the formalism? Do you consciously recite a definition in words, or check some feeling of remembering what the definition is, or mumble something in your mind about how a function is a set of ordered pairs? Is the domain/range-definitions just there as a reminder when you read it, or do you aim to remember them every time? Do you reset or accept if you forget to mention a detail?

Replies from: TurnTrout
comment by TurnTrout · 2021-01-23T23:24:24.744Z · LW(p) · GW(p)

I'm really surprised at how big your cards are! When I did anki regularly, I remember getting a big ugh-feeling from cards much smaller than yours, just because there were so many things that I had to consciously recapitulate. It was also fairly common that I missed some little detail and had to choose between starting the whole card over from scratch (which is a big time sink since the card takes so much time at every repeat) or accept that I might never remember that detail.

First, the Lebesgue integration card is one of my is one of my largest; I selected it to show Isusr how I might handle a non-decomposable question. I recommend that new Anki users be careful not to do this kind of thing: find a way to decompose, whenever possible! 

I generally tolerate missing details that I would have worked out were I actually expanding the mental sketch into formal writing; for example, if I make an arithmetical error but perform the correct kinds of calculations, I count that. If I make a sign error that I wouldn't make if I had been writing it out pencil and paper, I'll usually count it. If I think momentum is a scalar quantity or something crazy, though, I fail that card. 

I want to penalize the kind of error that I'm unlikely to fix with unaided reflection, or instinctive errors (like my gut sense for a concept just being wrong). 

I guess I use Anki to recall the "felt sense" of a proof sketch, and I don't sweat the details. Here's another long card I have:

If you're new to Anki: don't try this at home!

Believe it or not, I can decompose this proof into a sequence of ~five "urges", each of which I can (but generally don't) unpack into actual math if pressed. 

That said, this card was a PITA to learn, even with my level of experience. It took a lot of 'fail's to get right consistently. Again: if one is thinking about using Anki for the first time, do not do this kind of thing! It's hard to pull off at first, knowing if your mistake was worth redoing; if  you make yourself recite the whole card above, you'll be reviewing it forever.

This is more representative of my Anki deck. Note the lack of a million-page backside.

I'm super curious about your experience of e.g. encountering the function question. Do you try to generate both an example and a formalism, or just the formalism? Do you consciously recite a definition in words, or check some feeling of remembering what the definition is, or mumble something in your mind about how a function is a set of ordered pairs? Is the domain/range-definitions just there as a reminder when you read it, or do you aim to remember them every time? Do you reset or accept if you forget to mention a detail?

Imagining that card, I first imagine the motivation before superimposing the formal definition "on top" of it to capture the relevant part of the situation. I remember the phenomenon of apple exchange, and then overlay the function to describe what happens. Then I recite the formal definition. 

I don't make myself remember to label the "domain" or "range", because that's not fair game - it wasn't on the prompt. Those should go on separate cards. (I'd only learn one of those at a time, for similarity reasons)

Replies from: voider1
comment by voider1 · 2021-01-24T12:20:08.148Z · LW(p) · GW(p)

Interesting. I have used multiple different approaches with Anki. Last year I had to learn all pre-university math to enter uni, and I decided to use Anki which worked out pretty well. For a lot of the problems it's about remembering what steps to take in general. So I put in practice problems so I could remember the steps. The numbers don't matter, they could've been anything. 

I also tend to put in definitions, rules, theorems, standard and harder derivatives and primitives.

 

Now I'm in uni and I used Anki for learning linear algebra, I mostly tended towards clozes.

For linear algebra I tended to put in all theorems and facts, tried to keep them short so they're easier to remember and then do complementary exercises handed out by the course. Being an autodidact myself, I must say that if the course is done right it can make learning a lot easier. Sadly many of the courses weren't as good as the linear algebra course.

comment by lsusr · 2021-01-25T06:59:23.847Z · LW(p) · GW(p)

I don't have much to contribute to your comment except to acknowledge that it's useful, quality information. An upvote didn't feel sufficient.

comment by Gurkenglas · 2021-01-24T11:07:01.927Z · LW(p) · GW(p)

(Why not simply define the integral of f as the LM of {(x,r)|x in Omega, r <= f(x)}?)

Replies from: TurnTrout, gjm
comment by TurnTrout · 2021-01-24T16:00:33.526Z · LW(p) · GW(p)

The definition I gave is the standard one, but that's an interesting question. I'm not sure I know enough measure theory to find a hole in this right away; presumably there is one, because this would be a much more succinct definition. 

EDIT: TheMajor pointed me to this SE question; looks like this can work if you can prove that the aforementioned set is measurable in the product sigma-algebra. I like this definition a lot more.

Replies from: TheMajor
comment by TheMajor · 2021-01-24T16:15:48.543Z · LW(p) · GW(p)

Mathoverflow has discussion on it. In short:

  • This area definition is equivalent to the standard definition, although this was (to me) not immediately obvious.
  • Some statements (linearity of integrals, for example) are obvious from the one definition, while others (the Monotone Convergence Theorem) are obvious from the other definition. Unfortunately, proving that the two definitions are equivalent is pretty much the proof for these statements (assuming the other definition).
  • The general approach of "given a claim, test it on indicator functions, then simple functions, then all integrable positive functions, then all integrable functions, then (if desired) integrable complex functions" is called the standard machine of measure theory, so there is educational benefit to seeing it.
comment by gjm · 2021-01-24T17:14:44.105Z · LW(p) · GW(p)

I haven't thought it through, but I suspect the theorems you want to prove about integrals are probably easier to prove with the definition TurnTrout gave than with that one. 

comment by ErrethAkbe · 2021-01-24T20:15:47.123Z · LW(p) · GW(p)

What process do you use to review cards? Do you look at a prompt until you can say exactly what is on the card? Or if not verbatim what tolerance do you have for missing details/small mistakes?

Replies from: TurnTrout
comment by TurnTrout · 2021-01-24T20:22:15.309Z · LW(p) · GW(p)

Not verbatim. I mark the card 'correct' if I conjured the concept in the right way. If I was supposed to do a physics calculation, I don't pass it if I could recite the answer, I pass it if I went through the correct reasoning steps. "Calculate the gravitational force of a .1kg apple on a 70kg person, when they're 1m apart" -> ".1 * 70 * G / (1^2) = ?" is marked correct.

Replies from: ErrethAkbe
comment by ErrethAkbe · 2021-01-24T21:57:03.862Z · LW(p) · GW(p)

How do you find doing problems/exercises from these textbooks when you have prepared using Anki? And are you finding that earlier material seems obvious when reread?

Sorry if this is all coming across as critical and /or doubtful. I've tried to use Anki for theory before and dismally failed; the success you claim is very exciting and I'm trying to understand where I was going wrong.

So far I think I have focused too much on creating cards that can be memorised exactly (formulae and what-not), rather than having general concept cards that are used to develop fluency and familiarity (and later understanding), which sounds like what you are doing.

Replies from: TurnTrout
comment by TurnTrout · 2021-01-26T15:26:26.013Z · LW(p) · GW(p)

I think I do find Anki’d subjects much easier when I go back to them, yes.

I think that’s probably it - focus on concepts. If you like, I’d be happy to take a look at a deck you make / otherwise give feedback!

comment by Mo Putera (Mo Nastri) · 2021-01-25T01:15:45.216Z · LW(p) · GW(p)

On using Anki for math, Michael Nielsen wrote up his own experience trying it in this essay: http://cognitivemedium.com/srs-mathematics 

I was previously skeptical that Anki could be used for anything more complex than "basic facts" (whatever I thought that meant), but Nielsen's essay changed my mind. 

comment by Raemon · 2021-01-24T21:52:43.693Z · LW(p) · GW(p)

Curated.

There's a lot of nuggets to appreciate here, but I'm particularly grateful to a reminder about "Hey, you know you can just go out and learn a bunch of things? It's not mysterious, here are some concrete ideas for how to go about that."

Replies from: Benito
comment by Ben Pace (Benito) · 2021-02-08T19:41:11.439Z · LW(p) · GW(p)

I also really like the concrete info in the comments about how TurnTrout made math anki work for him.

comment by micpie · 2021-01-24T21:03:09.512Z · LW(p) · GW(p)

Very interesting article! A big part of the outlined techniques I also kind of discovered by trial and error.

I recently also started to work more on my math skills and putting in the mathjax was getting very time consuming. To reduce that bottleneck I can recommend this small app: https://mathpix.com

PS: I stumbled once over this collection of proofs without words and they make great Anki cards to build up intuition: https://mathoverflow.net/questions/8846/proofs-without-words

Replies from: TurnTrout
comment by TurnTrout · 2021-01-24T21:20:47.439Z · LW(p) · GW(p)

thank you! This is amazing. I wish I'd known about this sooner.

comment by Alex_Altair · 2023-01-12T01:42:50.681Z · LW(p) · GW(p)

This post is a solid list of advice on self-studying. It matches the Review criterion of posts that continue to affect my behavior. (I don't explicitly think back to the post, but I do occasionally bring up its advice in my mind, like "as yes, it's actually good to read multiple textbooks concurrently and not necessarily finish them".)

I actually disagree with the "most important piece of advice", which is to use spaced repetition software. Multiple times in my life, I have attempted to incorporate an SRS habit into my life, reflecting on why it previously failed, and attempting to account for that failure mechanism. But through it all it, I have not gotten it to work for me. Of course, YMMV, and I am excited for people to continue trying to take advantage of the concept of spaced repetition.

But I solidly endorse the other four major pieces of advice in this post.

comment by adamShimi · 2021-01-24T00:02:10.407Z · LW(p) · GW(p)

This is awesome! That's exactly the kind of post I wanted you to make!

About the lessons themselves:

  • I agree with the value of anki, but I have trouble finding things to ankify. My first impulse what to ankify everything under the sun, which lead to an anki burnout. Now I have the inverse problem of not finding much to put in Anki, mostly because I want to know/understand the concept before ankifying it. Or maybe I just don't read enough maths these days.
  • I want to disagree with the "read several textbook at once", but I think you're right. It's just that I'm trying so hard to focus on things and not jump from one to another all the time that reading multiple textbooks at once triggers all my internal alarms.
    I'll try to find a safe way for me to do that.
  • About not reading the whole textbook, I think I agree with the gist but I disagree with what you actually write. I definitely think that you should read most of a textbook if what you're doing is reading the textbook. On the other hand, you shouldn't try to master every detail in it.
    If you want to apply the pareto principle, then go through papers and write everything that you don't know. Then go search that in textbook. That's the efficient way. But reading a textbook is for getting a general impression of the field and building a map. So the latter chapters are useful. Just don't spend 20 hours on them.
  • The "read easier textbooks" advice looks like a rephrasing of "go just outside of your comfort zone". It makes sense to me.
  • I generally don't have the "approximate models" problem. But I also mostly read maths and computer science, which is additive instead of corrective.

 

And a problem this was. In early 2020, I had an interview where I was asked to compute . I was stumped, even though this was simple high school calculus (just integrate by parts!). I failed the interview and then went back to learning algebraic topology [? · GW] and functional analysis [LW · GW] and representation theory. You know, nothing difficult like high school calculus.

I think this is symptomatic of a problem I have myself, and which I only understood lately. I want to learn the cool shit, and I studied a lot of the fundamentals in my first two years after high-school (where we did 12 hours of maths a week). So I should remember how to do basic calculus! But somehow I forgot. And everytime I study a more advanced book, I feel like I should brush up my analysis and my linear algebra and all that if I want to really understand.

Yet that's wrong, because I mostly read these advanced textbooks for one reason: getting a map of the territory. Then I will know where to look when I need something that looks like that. And for that purpose, getting all the details perfectly right is not important.

On the other hand, there are some parts of maths in which I want skills. There I should actually take the time to learn how to do it, instead of simply knowing that it exists and the big picture description.

Replies from: TurnTrout
comment by TurnTrout · 2021-01-24T22:29:56.378Z · LW(p) · GW(p)

About not reading the whole textbook, I think I agree with the gist but I disagree with what you actually write. I definitely think that you should read most of a textbook if what you're doing is reading the textbook. On the other hand, you shouldn't try to master every detail in it.
If you want to apply the pareto principle, then go through papers and write everything that you don't know. Then go search that in textbook. That's the efficient way. But reading a textbook is for getting a general impression of the field and building a map. So the latter chapters are useful. Just don't spend 20 hours on them.

I agree; I should have written "don't complete textbooks", and have edited that part accordingly. I'm thinking some combination of "don't do all of the exercises" and "don't spend time learning every single concept in the book."

comment by TurnTrout · 2021-12-05T18:01:39.398Z · LW(p) · GW(p)

I've edited the post to incorporate a bit of wisdom I've gained over the last year. 

  • Use cloze deletions for Anki cards. 
    • Cloze deletions hide a small part of the text, and make you remember it given the rest of the content. They are fast to make and fast to review. 
    • The vast, vast majority of my new cards are cloze now, and they weren't when I first wrote this post. I think that was a mistake.
  • In the post, I used my representation theory cards as examples. 
    • What a bad choice. Those cards were too long. I ended up forgetting them (although not really many other cards from that point in time). Out of all the examples I could have chosen, why those?!
    • The problem was that the cards were too long. I made myself remember too much on each pass.
  • I now read using MarginNote on iPad with a nice cloze deletion -> Anki export pipeline.
comment by Raemon · 2021-01-24T21:56:03.507Z · LW(p) · GW(p)

Last year I tried re-learning calculus, and later learning some statistics, and both went kinda okay but I feel like I don't really have the scaffolding to make them actually useful longterm. I have a vague sense that it's good for me to have mathematical literacy but I'm never quite sure what to do with that belief.

This... I guess this comment is me saying "my default action is still to probably not invest much more effort here, but I am interested in anyone pitching me on a concrete strategy they think I would benefit from" (I think this is most useful coming from people who know me reasonably well and have some sense of what my deal is)

comment by Xor · 2023-04-08T05:08:50.919Z · LW(p) · GW(p)

I am excited to see this sort of content here. I am currently finishing up my senior year of high school and making plans for the summer. I have decided to focus much of my free time on learning, and rationality as well as filing out my knowledge base on math, physics, and writing. These will be a valuable set of skills for collage and the rest of my life. This summer I plan to build a course on learning (free stuff on youtube) first because I want to be rigorous in my understanding of learning and teaching ensures that. Second I am looking forward to the experience of making videos and editing as I have never attempted to do this sort of thing. I have started outlining what I want the course to look like and assembling a bunch of resources, books, people, websites, and studies. I found a book called ”Make it Stick” it is scientific but readable and I recommend it as a resource for self teaching and learning it also has a lot of great information on spaced repetition. 

Another great protocol that I recently heard about involves randomly pausing the learning and just letting your mind go blank for ten seconds. I guess this does some incredible things for memory. If you’re a lot smarter than I am you can figure out exactly how much it does for your memory.  :)
https://www.cell.com/current-biology/fulltext/S0960-9822(19)30219-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0960982219302192%3Fshowall%3Dtrue
I believe this is the study the protocol comes from, it has a lot of good information on rest as well as the 10 second breaks. 

comment by TurnTrout · 2023-01-01T00:14:10.917Z · LW(p) · GW(p)

I'm glad I started using Anki. I did Anki 362/365 days this year. 

I averaged 19 minutes of review a day (although I really think review tended to take longer), which nominally means I spent 4.75 clock-days studying Anki. 

Seems very worth it, in my experience. 

comment by rine · 2021-01-24T13:42:18.726Z · LW(p) · GW(p)

Nice article. Like Nielsen, you seem to have discovered Incremental Learning for yourself.

https://supermemo.guru/wiki/Michael_Nielsen_re-discovers_incremental_reading_with_Anki

If you want the whole package in software (i.e. not only the SRS part) give SuperMemo a try :)

https://help.supermemo.org/wiki/Incremental_learning

comment by Vladimir_Nesov · 2021-01-24T06:10:08.441Z · LW(p) · GW(p)

Instead of reading a textbook with SRS and notes, skim more books (not just textbooks), solve problems, read wikis and papers, talk to people, watch talks and lectures, solve more problems from diverse sources. Instead of memorizing definitions, figure out how different possible definitions work out, how motivations of an area select definitions that allow constructing useful things and formulating important facts. The best exercise is to reconstruct main definitions and results on a topic you've mostly forgotten or never learned that well. Math is self-healing, so instead of propping it up with SRS, it's much better to let it heal itself in your mind. (It heals in its own natural order, not according to card schedule.) And then, after you've already learned the material, maybe read a textbook on it cover to cover, again.

(I used to have a mindset similar to what the post expresses, and what it recalls, though not this extreme. The issue is that there is a lot of material available to study on standard topics, especially if you have access to people fluent in them and can reinvent parts of them on your own, which lets this hold even for obscure or advanced topics where not a lot is written up. So the effort applied to memorizing things or recalling old problems can be turned to better understanding and new problems. This post seems to be already leaning in this direction, so I'm just writing down my current guess at a healthy learning practice.)

Replies from: eric-siggy-scott
comment by Eric 'Siggy' Scott (eric-siggy-scott) · 2021-01-24T20:31:38.958Z · LW(p) · GW(p)

Why "instead?"

I find that all of those activities enhance my SRS experience, and my SRS experience (sometimes dramatically) enhances all of those activities!

Particularly since I use SRS to capture concepts, intuitions, and rich relationships, not just isolated facts.  Rote memorization doesn't work very well with SRS anyway--as with everything, it works best when used to understand a topic in its full intuitive glory.

comment by jacquesthibs (jacques-thibodeau) · 2023-01-02T03:27:13.646Z · LW(p) · GW(p)

I wrote a shortform [LW(p) · GW(p)] about my current learning system (and a short follow-up comment on making it useful for solving alignment [LW(p) · GW(p)]), so I figured I'd share it here since I talk a bit about Anki and other similar things in your post.


TLDR of what I've been thinking about lately:

  • There are some great insights in this video called "How Top 0.1% Students Think." And in this video about how to learn hard concepts.
  • Learning is a set of skills. You need to practice each component of the learning process to get better. You can’t watch a video on a new technique and immediately become a pro. It takes time to reap the benefits.
  • Most people suck at mindmaps. Mindmaps can be horrible for learning if you just dump a bunch of text on a page and point arrows to different stuff (some studies show mindmaps are ineffective, but that's because people initially suck at making them). However, if you take the time to learn how to do them well, they will pay huge dividends in the future. I’ll be doing the “Do 100 Things” challenge and developing my skill in building better mindmaps. Getting better at mindmaps involves “chunking” the material and creating memorable connections and drawings.
  • Relational vs Isolated Learning. As you learn something new, try to learn it in relation to the things you already know rather than treating it as isolated from everything (flashcards can perpetuate the problem of learning things in isolated form).
  • Encoding and Retrieval are essential concepts for efficient learning.
  • Deep processing is the foundation of all learning. It is the ability to connect, process, organize and relate information. The opposite of deep processing is rote memorization. If it doesn’t feel like you are engaging ~90% of your brain power when you are learning/reading something, you are likely not encoding the information into your long-term memory effectively.
  • Only use Flashcards as a last resort. Flashcards are something a lot of people use because they feel comfortable going through them. However, if your goal is to be efficient in your learning, you should only use flashcards when it’s something that requires rote learning. Video worth watching on Spaced Repetition.
  • You need to be aiming for higher-order learning. Take advantage of Bloom's Taxonomy.[1]
  • My current approach for learning about alignment: I essentially have a really big Roam Research page called "AI Alignment" where I break down the problem into chunks like "Jargon I don't understand," "Questions to Answer," "Different people's views on alignment," etc. As I fill in those details, I add more and more information in the "Core of the Alignment Problem" section. I have a separate page called "AI Alignment Flow Chart" which I'm using as a structure for backcasting on how we solved alignment and identifying the crucial things we need to solve and things I need to better understand. I also sometimes have a specific page for something like Interpretability when I'm trying to do a deep dive on a topic, but I always try to link it to the other things I've written in my main doc.

Ever since I was little, I have relied on my raw brain power to get to where I am. Unfortunately, I could never bring myself to do what other smart kids were doing. Flashcards, revision? I would either get bored out of my mind or struggle because I didn’t know how to do it well. Mindmaps? It felt OK while I was doing it the few times I tried, but I would never revise it, and, honestly, I sucked at it.

But none of that mattered. I could still do well enough even though my learning system was terrible. However, I didn’t get the top grades, and I felt frustrated.

I read a few books and watched the popular YouTubers on how to learn things best. Spaced Repetition and Active Recall kept coming up. All these intelligent people were using it, and I truly believed it worked. However, whenever I tried it, I either ended up with too many flashcards to have the time to review, or I couldn't build a habit out of it. Flashcards also felt super inefficient when studying physics.

I did use Cal Newport’s stuff for some classes and performed better by studying the same amount of time, but as soon as things got intense (exam season/lots of homework), I would revert to my old (ineffective) study techniques like reading the textbook aimlessly and highlighting stuff. As a result, I would never truly develop the skill (yes, skill!) of studying well. But, just like anything, you can get better at creating mindmaps for proper learning and long-term memory.

I never got a system down, and I feel I’m losing out on gains in my career. How do I learn things efficiently? I don’t want to do the natural thing of putting in more hours to get more done. 1) My productivity will be capped by my inefficient system, 2) I still want to live life, and 3) it probably won’t work anyways.

So, consider this my public accountability statement to take the time to develop the skills necessary to become more efficient in my work. No more aimlessly reading LessWrong posts about AI alignment. There are more efficient ways to learn.

I want to contribute to AI alignment in a bigger way, and something needs to change. There is so much to learn, and I want to catch up as efficiently as possible instead of just winging it and trying whatever approach seems right.

Had I continued working on things I don’t care deeply about, I might have never decided to put in the effort to create a new system (which will probably take a year of practicing my learning skills). Maybe I would have tried for a few weeks and then reverted to my old habits. I could have kept coasting in life and done decently well in work and my personal life. But we need to solve alignment, and building these skills now will allow me to reap major benefits in a few years.

(Note: a nice bonus for developing a solid learning system is that you can pass it on to your children. I’m excited to do that one day, but I’d prefer to start doing this now so that I know that *I* can do it, and I’m not just telling my future kids nonsense.)

So, what have I been doing so far?

I started the iCanStudy course by Dr. Justin Sung (who has a YouTube channel). I’m only about 31% through the course.

My goal will be to create a “How to Create an Efficient Learning System” guide tailored for professionals and includes examples in AI alignment. Please let me know if there are some things you’d like me to explore in that guide.

Before I go, I’ll mention that I’m also interested in eventually taking what I learn from constructing my own learning system and creating something that allows others to do the same, but with much less effort. I hope to make this work for the alignment community in particular (which relates to my accelerating alignment project), but I’d also like to eventually expand to people working on other cause areas in effective altruism.

  1. ^
comment by Eugleo · 2021-04-28T13:46:38.595Z · LW(p) · GW(p)

It so happens that every six or seven months an article like this pops up, and revives my interest in Anki for a short while. So far, it has never stuck with me, but maybe this time…
 

I’m preparing for a bunch of Maths (linear algebra, analysis, automata and grammars, and the like) and Biology (molecular bio, immunology, genetics) exams, all due in June. I haven’t seen/used any of the subjects in question for at least a year, although I have passed each of them with perfect marks back then.

  1. Is it too late to construct the cards now as part of the refresh/relearning process?
  2. If it is, what other learning routine would you recommend? I usually just go through a book and try to connect everything to everything else. I’ve also adopted Zettelkasten notes for some of the subjects, formalising this notion of connecting things.
  3. If it isn’t, how granular should I be wirh the cards?
    1. A card for every definition, every theorem, and the main lemmas? (there’s literally hundreds of those)
    2. Or just a few high level cards like “explain the matrix subspaces” and “what are the eigenvectors about”?
    3. Or should the granularity be chosen relative to the expected granularity of the exam? (They probably won’t ask me to proove every little theorem)
Replies from: TurnTrout
comment by TurnTrout · 2021-04-28T14:36:50.967Z · LW(p) · GW(p)

Definitely not too late to make cards. I've learned a great deal of basic chemistry in the last month or so, just studying for random 30-minute chunks and binging interesting-looking wikipedia articles. A month+ is plenty of time for spaced repetition to work its magic.

For math, I recommend Anki; for non-Latex-intensive subjects like biology, I recommend SuperMemo for fast card creation while you read and review material. Unfortunately, I have yet to write up my thoughts on the latter.

I like to look at the "cheat sheets" for courses and ensure I know how to reason using the key concepts. Add cards for the key concepts you find in course review notes. For definitions and concepts, I like cloze deletions - they're easy to make and easy to learn. For exercises, add small practice problems for the key concepts and the items you know will be on the tests. 

Don't learn every definition and theorem. 

Replies from: Eugleo
comment by Eugleo · 2021-04-28T21:05:04.589Z · LW(p) · GW(p)

A month+ is plenty of time for spaced repetition to work its magic.

Thanks, that sounds encouraging.

For math, I recommend Anki; for non-Latex-intensive subjects like biology, I recommend SuperMemo for fast card creation while you read and review material.

I have to admit, I despise Anki. I love well-designed tools, to the point it's probably detrimental to my life. But alas, it is like it is — and so every time I see Anki (and I've made around 300 cards for it past the last few years), I have an urge to throw the computer out of the window.

But, that's just a detail. I plan to use Mochi, by the way.

I like to look at the "cheat sheets" for courses and ensure I know how to reason using the key concepts. Add cards for the key concepts you find in course review notes.

Ok, so the main advice is: don't make a card for everything, just the important concepts. And those concepts can be found in "cheatsheets" and "course review notes", it seems — unfortunately, I don't have any of those things.

So, I'd be grateful if you could illustrate what do you think constitutes a major, Anki-worthy piece of information. Let's say you're trying to revise (in the context of my current exam prep) a base chapter on vector spaces from Linear algebra done right. You probably want a card with the VS axioms. What else do you view as important enough?

Replies from: TurnTrout
comment by TurnTrout · 2021-04-28T23:56:54.021Z · LW(p) · GW(p)

Ok, so the main advice is: don't make a card for everything, just the important concepts. And those concepts can be found in "cheatsheets" and "course review notes", it seems — unfortunately, I don't have any of those things.

Why not use Google for notes from other schools?

Replies from: Eugleo
comment by Eugleo · 2021-04-29T05:55:16.637Z · LW(p) · GW(p)

If the plan was to understand the given subject, then yes, that would work. And of course, that is the ultimate plan.

However, the more pressing matter are the exams. I would be afraid that the intersection of the two programs won't contain all of the important concepts; that's my experience with "other" textbooks (meaning, other than the few recommend ones), at least.

comment by Quinn (quinn-dougherty) · 2021-01-25T12:23:27.640Z · LW(p) · GW(p)

Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on.

This is hard to instrumentalize regarding difficulty. I find that the hardest exercises are likeliest to be skipped (after struggling with them for an hour or two), but it doesn't follow that I can expect the easier ones (which I happened to have completed) to lie in that key 20%.

Replies from: TurnTrout
comment by TurnTrout · 2021-01-25T14:38:58.140Z · LW(p) · GW(p)

I like to randomly sample X% of the exercises, and read the rest; this lets me know later whether or not I missed something important. Simple rules like "do every fifth exercise" should suffice, with the rule tailored to the number of exercises in the book.

comment by Eric 'Siggy' Scott (eric-siggy-scott) · 2021-01-24T17:54:38.589Z · LW(p) · GW(p)

Fantastic insights!  I especially like how you've articulated the value of using "easy" books to connect concept to the "real world."  I've certainly run into the problem of trying to Ankify a big dense book, and getting bogged down with it.

Incidentally I'm also working through the Book of Why and Causality by Pearl.  Great progression (starting with one, then returning to the other).

I've been using Anki for math and computing for several years now, and one word of warning is that your big, dense cards (i.e. screen-shotted definitions) are the sort of thing that, in my experience, is easy to remember for a month or two (because rote memorization is filling in a lot of the glue), but becomes much more difficult (and trends toward ease hell) once intervals reach the high months/years range.

So I wonder if after a couple more months you might feel inclined to refactor your cards to use many small cards (rather than few big cards) to express definitions and theorems.

Personally, I've converged on a highly visual approach to chunking concepts into their "basic idea," followed later by detail cards to drill specific aspects of a larger whole.  This is obviously a nice approach of diagram-friendly areas like topology:

But also works for equations, algorithms, etc. too.

I've got some more examples here:

 * Advance machine learning concepts

 * Statistics
 

comment by Space L Clottey (TheTrueSquidward) · 2021-01-24T14:00:00.954Z · LW(p) · GW(p)

I think you'd get a lot of value out of using incremental reading, it improves the learning to memorisation workflow tremendously. Currently SuperMemo is the best at IR. (Post I wrote about this: https://gingerjumble.wordpress.com/2020/08/28/the-main-reason-you-should-switch-from-anki-to-supermemo/)

comment by ErrethAkbe · 2021-01-24T11:52:29.387Z · LW(p) · GW(p)

I'm a bit skeptical of what you claim because it is so different from my approach to becoming proficient at pieces mathematics: usually I will work through progressively more-complicated problems in excruciating detail. I don't claim that this is the most efficient method, and it would be nice to find such an approach, just that memorization methods have usually lead to me agreeing with the mathematics, rather than really understanding it.

But maybe you are using Anki differently to how I expect. How exactly do you review cards? Do you look at a prompt until you can say verbatim what is on the card? Or if not verbatim what tolerance do you have for missing details/small mistakes?

Replies from: eric-siggy-scott, TurnTrout
comment by Eric 'Siggy' Scott (eric-siggy-scott) · 2021-01-24T17:58:26.232Z · LW(p) · GW(p)

Anki's value depends a lot on how you use it.

If you use it for rote memorization, without grokking concepts deeply---then what you get is rote memorization.  Think of memorizing capitals without learning anything else about states and nations.

But if you work hard to identify conceptual landmarks and add good questions to tie it all together (more like a high quality PowerPoint presentation), then it can be amazing and facilitate retention of rich intuitions for years.  Think of memorizing major topographic landmarks on a map and a bit of history, so that you have something to relate capitals to in context.

Incidentally, rote-memorization cards are actually harder to review with Anki: the "glue" holding them in memory starts to fade after a month or so, so eventually the drift toward "ease hell" and otherwise become unpleasant to review.

comment by TurnTrout · 2021-01-24T16:08:54.054Z · LW(p) · GW(p)

I went into detail on how I use Anki for math in this thread [LW(p) · GW(p)]. Feel free to comment there with further questions.

comment by johnswentworth · 2021-01-25T00:49:38.331Z · LW(p) · GW(p)

In early 2020, I had an interview where I was asked to compute .

A trick I learned from Jaynes:

... or, if you want to be a smartass about it, write "" in place of "" and tell people you're "differentiating by 1": .

All you actually need to remember is .

comment by Raghu Veer S (raghu-veer-s) · 2021-01-24T11:52:48.562Z · LW(p) · GW(p)

Neat article. Few things I myself have been struggling with and would like some insight into are:

  1. Were you able to address the issue of ankifying the algorithms? If yes, how?
  2. How do you handle the Anki burnout(or for that matter Anki inertia)? I am guessing with multiple subjects at a time, you at least have to review 100-200 odd cards per day(maybe more), and based on the non-atomic nature of your example cards, it seems to me that getting yourself to review that many cards every day is not a sustainable activity?
  3. How do you preserve your sanity between learning new things and practicing new cards? It seems to me that it can be taxing to spend time revising on your own while also learning new things.
  4. How do you handle the inertia that comes with creating new cards or editing cards for so many concepts and subjects?
  5. Does the SM-2 algorithm suffice your needs in terms of frequency, priority, scheduling, etc?
  6. Finally, have you found ways to avoid things like wrist issues, back issues, cognitive exhaustion, etc? Like maybe good computer etiquettes or something?

Sorry for the long list of questions. I am curious to know more because I have been struggling with getting my way into auto-didacting for more than 2 years now due to issues like being unable to differentiate between wanting-to-want and really want, burnout, haphazardness of it all etc. Any pointers on it would be really appreciated.

Replies from: eric-siggy-scott
comment by Eric 'Siggy' Scott (eric-siggy-scott) · 2021-01-24T18:08:41.737Z · LW(p) · GW(p)

Not the OP, but got a few opinions there:

  1. Algorithms can be Ankified with the same strategies that work for other subjects: chunk the basic intuition into a short overview card, use diagrams, and hit the "basic idea" of how it works before adding more details cards for particular tricks & minutia.  As always, most cards should take <15 seconds to review (including time spent reading the card).
  2. Personally, I avoid burnout a few ways: 
    A) pair Anki with activities like taking a daily 1-mile walk or doing dishes (iOS voice commands help with the latter, so you can review hands-free),
    B) don't be shy about reviewing my professional development cards during work hours,
    C) deleting social media from my phone, so Anki is the go-to gap-filler app,
    D) always be learning a few new cards in each deck (since decks with 100% old material are less fun to review),
    E) refactoring: I set a reminder to work on marked cards every few days, so I can break down difficult cards into higher-quality material.  A little maintenance goes a long way.
    F) card design, card design, card design.  Well-designed cards are far less fatiguing to review than hastily made ones.
  3. Not sure I understand.  Anki kind of abolishes the line between reviewing and learning: you do both with the same system.  If you mean motivating yourself to study old material---my trick is to review from large, combined superdecks, so there is no pause between "Oh, I finished my learning decks, time to move on to old decks."  If I give myself that break, I'm more likely to procrastinate, but if I treat it all as one big deck there's motivation to finish it all in one go.
  4. I actually find it motivating.  Learning complex subjects is usually hard, because when you hit something hard there's an urge to procrastinate rather than figure out what's blocking you.  Anki removes these energy barriers for me: I can always learn a little more about quantum mechanics or whatever by adding a card or two.  It makes hard subjects easy, and makes it possible to say "I'll just study a little topology for 10 minutes" and actually make progress (which is usually impossible outside of SRS).
  5. I very rarely try to tweak the algorithm.  90% of the usefulness of SRS is just being able to handle the spacing schedule for you.  Tweaking it doesn't lead to much gain in efficiency (with a few exceptions: adding learning stages to avoid ease hell, and changing the reset interval to something like 20% or 50% instead of 0% both help avoid a lot of unnecessary reviews).
  6. Voice commands for everything, and audio cards for languages.  Hands-free, eyes-free reviews!
comment by MarcelloV (aram-baghdassarian-1) · 2021-01-24T05:17:27.899Z · LW(p) · GW(p)

Perhaps I am overthinking this, but when it comes to applying your knowledge gained through spaced repetition, is there a difference in how effectively you can apply it in these two situations?

  1. You spend time memorizing what a function is. A function comes up in a problem in a very similar format to how it looked on your Anki card. You are able to understand it because the prompt of the function in the problem causes you to recall your understanding from studying and then apply it.
  2. You spend time memorizing what a function is. A problem comes up that can be solved by using functions. However, the problem doesn't explicitly mention functions like in #1; it's up to you to recall your understanding of functions and recognize that this is a time to use them without seeing the prompt that you studied with.

In case this example is too weird or confusing, you can think of the using Anki for memorizing cognitive biases; you'll be able to recall the definition of them, but how much will spaced repetition help you recognize when cognitive biases pop up in your life (so that you can employ whatever counter-measures you see fit)?

Replies from: TurnTrout
comment by TurnTrout · 2021-01-24T16:07:47.379Z · LW(p) · GW(p)

I don't think spaced repetition runs into the failure mode of #2 if you do it competently. Good spaced repetition prompts you to think about the concepts on a spaced, expanding basis, not to recite the magic words on the back of the card. When I study a card, I'm trying to train urge to understand concept in prompt -> concepts on back of card, not words in prompt -> words on back of card.

For example, when I learned French with Anki, I translated entire sentences English-to-French (and vice versa). I learned vocab words by just finding sentences where I already knew how to say everything except the new word. By learning sentences containing the word for e.g. "swamp", I learned how to say "swamp" in normal, fluent conversation, without having to remember the exact sentence I used for Anki (as #2's model might suggest). 

I don't see any real difference in quality between normal study and Anki study, if you make your cards well.

comment by MSRayne · 2021-01-24T01:34:20.502Z · LW(p) · GW(p)

I have multiple times tried to get into an Anki habit and failed to keep it up. I think the main thing that makes me stop is that I try to make nearly every sentence of something that I'm studying into a card, because I have no idea what's worth remembering and what isn't. (As a general rule, throughout life, I suck at prioritizing.) The other thing, though, is that it feels like Work and things which feel like Work are Unpleasant and I procrastinate them. Do you have any advice for getting over that hump?

Replies from: raj-thimmiah, TurnTrout
comment by Raj Thimmiah (raj-thimmiah) · 2021-01-25T03:16:10.418Z · LW(p) · GW(p)

In general, when I first started using SRS I had this issue too. Over time, I got a fair bit better of it with 1 or 2 rules:

-I should be able to make at least 1 connection from the idea to something else (filter out only shallowly understood things)

-I should be able to either connect it to a situation of real life applicability or a goal

I didn't make it only: I need to be able to figure out applicability when making card, because a lot of useful knowledge might not that have it as obvious. But at the least, I should be learning it in a real context. E.g. some of my main goals now are:

-learn UX design

-understand how SRS/IR can be used to create genius

-get better at teaching

-become more resillient to disruption (emotionally)

Those are specific-ish and learning in context of those instead of I can use this sometime in the next decade is a useful filter

comment by TurnTrout · 2021-01-24T20:19:03.884Z · LW(p) · GW(p)

One way to start is to put in things you find yourself looking up. If you're reading math and you have to keep looking up the quadratic formula, put it in. If you're reading the 'Nash equilibrium' Wikipedia page, put in a few examples (and maybe even the formal definition!). 

Then, while you're reading a textbook, you might develop a taste for what things you'll probably look up later. 

The other thing, though, is that it feels like Work and things which feel like Work are Unpleasant and I procrastinate them. Do you have any advice for getting over that hump?

I like to think about Anki as giving me enduring experience points, which is Fun and Gratifying. Once you find it really work for you on something that matters, maybe you'll find it more exciting? I can only guess - I've naturally enjoyed Anki since nearly the beginning.

Replies from: MSRayne
comment by MSRayne · 2021-01-24T22:48:00.368Z · LW(p) · GW(p)

Hmm... I could try actually counting experience points. Like, each flash card reviewed grants a point, and every time I reach, say, a new Fibonacci Number of points, I gain a level and... um... stuff! The idea of leveling up really isn't very motivating by itself, but it would help.

Here's one idea: when reading a textbook or anything else I want to memorize, I might try to come up with just one question and answer pair about each page, and make a card out of that. Summarize the most important info on that page. Anything that's not too info-heavy, that should work. In things which are info-heavy, your method should work quite well.

comment by George Ingebretsen (george-ingebretsen) · 2023-10-17T00:20:06.348Z · LW(p) · GW(p)

Random Anki tips

 

When using Anki for a scattered assortment of topics, is it best to group topics into different decks or not worry about it and just have a single big deck.

If making multiple decks, would Anki still give you a single "do this today" set? 

I've only used it in the context of studying foreign language. 

Replies from: g-w1
comment by g-w1 · 2023-10-20T15:01:29.493Z · LW(p) · GW(p)

I prefer putting everything into one big deck for a few reasons.

  1. It forces you to do all the cards every day. When I had different decks, I would do the "main" deck every day, but then not do the other decks.
  2. I have not read any research on this, but it seems like interspersing the cards could make the cards less likely to anticipate and force you to learn better.
  3. It also just mixes up the monotony and makes it more fun.
comment by Tim Liptrot (rockthecasbah) · 2021-01-24T22:34:04.772Z · LW(p) · GW(p)

Why did you write autodidacting instead of selfteaching? It just put a google search between me and reading the article...

Replies from: TurnTrout
comment by TurnTrout · 2021-01-24T23:48:40.892Z · LW(p) · GW(p)

It was the first word to come to mind. I changed the title, thanks. 

Replies from: rockthecasbah
comment by Tim Liptrot (rockthecasbah) · 2021-01-25T00:09:35.987Z · LW(p) · GW(p)

Thanks! Looks good.

comment by jimv · 2021-01-23T20:16:52.994Z · LW(p) · GW(p)

You mention that planting a seed of Anki usage in 6th grade is a good thing. Do you have any thoughts about how to help a child get started with an Anki habit?

Replies from: TurnTrout, TurnTrout, raj-thimmiah
comment by TurnTrout · 2021-01-23T20:53:12.517Z · LW(p) · GW(p)

I don't, unfortunately. I don't have any special insight into children at this point in my life, and I can barely remember what my mental life was like back then. (Too bad I didn't use Anki to remember what it was like to be in 6th grade!)

There are always instrumental things to appeal to (e.g. "by using this application, you can do your work faster and have more time to play with your friends"), but that might take away from their intrinsic motivation if they view Anki as just a means to more free time.

I'd like to defer to LessWrong users who have actual experience with kids.

comment by Raj Thimmiah (raj-thimmiah) · 2021-01-25T03:13:00.855Z · LW(p) · GW(p)

The creator of spaced repetition has talked about how usage in children of SRS is not a great idea here

 

Roughly idea is that self-direct learn drive based learning is needed for coherent, well connected models.

It's sort of like how I learned trig in high school and I can't apply it to a real world problem to save my life. If I'd learned trig for myself, when I thought it'd be useful, it'd be way more applicable and not be so siloed to 'this is how you use trig to solve math problems on a test'