Posts
Comments
Note, I consider this post to be “Lynette speculates based on one possible model”, rather than “scientific evidence shows”, based on my default skepticism for psych research.
A recent Astral Codex Ten post argued that advice is written by people who struggle because they put tons of time into understanding the issue. People who succeeded effortlessly don’t have explicit models of how they perform (section II). It’s not the first time I’ve seen this argument, e.g. this Putanumonit post arguing that explicit rules help poor performers, who then abandon the rules and just act intuitively once they become good.
This reminded me of a body of psych research I half-remembered from college called Choking under Pressure.
My memory was that if you think about what you’re doing too much after becoming good, then you do worse. The paper I remembered from college was from 1986, so I found “Choking interventions in sports: A systematic review” from 2017.
It turns out that I was remembering the “self-focused” branch of choking research.
"Self-focus approaches have largely been extended from Baumeister’s (1984) automatic execution hypothesis. Baumeister explains that choking occurs because, when anxiety increases, the athlete allocates conscious attention to movement execution. This conscious attention interferes with otherwise automatic nature of movement execution, which results in performance decrements."
(Slightly worrying. I have no particular reason to doubt this body of work, but Baumeister's "willpower as muscle" -- i.e. ego depletion -- work hasn't stood upwell.)
Two studies found that distraction while training negatively impacted performance. I’m not sure if this this was supposed to acclimatize the participants to distractions while performing or reduce their self-focus while training. (I’m taking the paper’s word and not digging beyond the surface on the numbers.) Either way, I feel very little surprise that practicing while distracted was worse. Maybe we just need fatal-car-crash magnitude effects before we notice that focus is good?
Which makes it all the more surprising that seven of eight studies found that athletes performed better under pressure if they simultaneously did a second task (such as counting backwards). (The eighth study found a null result.) According to the theory, the second task helped because it distracted from self-focus on the step-by-step execution.
If this theory holds up, it seems to support paying deliberate attention to explicit rules while learning but *not* paying attention to those rules once you're able to use them intuitively (at least for motor tasks). In other words, almost exactly what Jacob argued in the Putanumonit article.
Conclusions
I was intrigued by this argument because I’ve argued that building models is how one becomes an expert.[1] After considering it, I don’t actually think the posts above offer a counter argument to my claim.
My guess is that experts do have models of skills they developed, even if they have fewer models (because they needed to explicitly learn fewer skills). The NDM method for extracting experts’ models implies that the experts have models that can be coaxed out. Holden’s Learning By Writing post feels like an explicit model.
Another possibility is that experts forget the explicit models after switching to intuition. If they faced the challenges more than five or ten years ago, they may not remember the models that helped them then. Probably uncoincidentally, this aligns neatly with Cal Newport’s advice to seek advice from someone who recently went through the challenges you’re now facing because they will still remember relevant advice.
Additionally, the areas of expertise I care about aren’t like walking, where most people will effortlessly succeed. Expertise demands improving from where you started. Both posts and the choking under pressure literature agree that explicit models help you improve, at least for a while.
“Find the best explicit models you can and practice until you don’t need them” seems like a reasonable takeaway.
[1] Note, there’s an important distinction between building models of your field and building models of skills. It seems like the main argument mostly applies to models of skills. I doubt Scott would disagree that models of fields are valuable, given how much time he’s put into developing his model of psychopharmacology.
Makes sense! Becoming more rational is a continual journey, and there’s no need to feel ashamed that you’re still learning. I expect you’ll find the process faster and smoother if you approach it as though you’re collaborating with other posters, instead of trying to score points :)
Occasionally, I get asked for feedback on someone’s resume. I’m not really a resume-editing coach, but I can ask them what they accomplished in that roll where they’re just listing their duties. Over time, I’ve found I’m completely replaceable with this rock.
You did X for Y company? Great! Why is it impressive? Did it accomplish impressive outcomes? Was it at an impressive scale? Did it involve impressive other people or companies?
You wrote an application using Z software? Great! Why is it impressive? Did the code speed up run time by an impressive amount? Did it save an impressive amount of money? Did it enable an impressive research finding? Does it display an impressive amount of technical expertise?
You published a thing? Great! Why is it impressive? Was it published somewhere impressive? Was it cited an impressive number of times? Did impressive people say good things about it?
Fabricated goals
I’m really good at taking an abstract goal and breaking it down into concrete tasks. Most of the time, this is super useful.
But if I’m not sure what would accomplish the high level goal, sometimes the concrete goals turn out to be wrong. They don’t actually accomplish the high-level more vague goal. If I don’t notice that, I’ll at best be confused. At worse, I’ll accomplish the concrete goals, fail at the high-level goal, and then not notice that all my effort isn’t accomplishing the outcome I actually care about.
I’m calling the misguided concrete goals as “fabricated goals”. Because I’m falsely believing this goal is an option to get me to my high-level goal.
The alternative feels pretty bad though. If I can’t break the vague goal into concrete steps that I know how to do, I need to be constantly feeling my way through uncertainty. In that situation, sometimes it’s good to pick a small, time-bound concrete goal and do it to see if it helps. But I need to be constantly checking in on whether it’s actually helping.
I’ve been practicing a lot this year with improving feedback loops, and it’s come a long way. Sitting with uncertainty for days and checking in on whether I’m making progress on the scale of minutes, though, that’s hard. I’ve heard early stages of research being called “wandering in the desert” and this feels similar.
It’s so much easier to substitute a fabricated goal – I know what I need to do, I can measure my progress. It’s harder to sit with the uncertainty and hopefully, slowly feel my way toward some insight.
I actually have a heart condition that severely limits my ability to exercise. Walking three miles is beyond what I'm capable of on an average day, let alone jogging anywhere.
This is an surprisingly harsh critique of a minor detail. In the future, I would strongly recommend a more polite, truth-seeking inquiry.
Hmm, it would probably work well to write a longer daily FB post, like if I set a goal to publish at least 500 words each day.
Part of the goal is ‘become comfortable submitting things I'm not fully happy with’ and part is 'actually produce words faster'. The second part feels like it needs the length requirement. I've done daily short FB posts before and found it useful, but I noticed that I tended to write mostly short posts that didn't require me to hammer out words.
Hmm, I'm not certain where you're getting that. I interpreted this as the amount of deliberate practice contributed to success in some fields much more than it did in other fields. (Which could be explained by some fields not having developed techniques and training methods that enable good DP, or could be explained by everyone maxing out practice, or by practice not mattering in those fields.) DP still makes a difference among top performers in music and chess, indicating that not all top performers are maxing out deliberate practice in those areas.
I considered that early on during my exploration, but didn't go deep into it after seeing Scott's comment on his post saying:
These comparisons held positions (specialist vs. generalist) constant. Aside from whether someone is a specialist or not, I don't think there's any tendency for older doctors to get harder cases.
Now, after seeing that the other fields also match the same pattern of decline, I'd be somewhat surprised by evidence that taking on harder cases explained the majority of skill plateaus in middle age for doctors.
Note: I was treating the 2009 study as a psudo-replication. It's not a replication, but it's a later study on the same topic that found the same conclusion, which had allayed some of my concerns about old psychology research. However, I since looked deeper into Dan Ariely's work, and the number of accusations of fraud or academic misconduct makes me less confident in the study. https://en.m.wikipedia.org/wiki/Dan_Ariely#Accusations_of_data_fraud_and_academic_misconduct
I agree with the line of reasoning, but I'd probably err on the side of adding a deadline even for designing your office - if you want to make sure the task gets done at some point, setting the deadline a month away seems better than not having one at all.
I agree that adopting high variance strategies makes sense if you think you're going to fail, but I'm not sure the candle task has high variance strategies to adopt? It's a pretty simple task.
I feel like being the code master for Codenames is a good exercise for understanding this concept.
I wasn't thinking of shards as reward prediction errors, but I can see how the language was confusing. What I meant is that when multiple shards are activated, they affect behavior according to how strongly and reliably they were reinforced in the past. Practically, this looks like competing predictions of reward (because past experience is strongly correlated with predictions of future experience), although technically it's not a prediction - the shard is just based on the past experience and will influence behavior similarly even if you rationally know the context has changed. E.g. the cake shard will probably still reinforce eating cake even if you know that you just had mouth-changing surgery that means you don't like cake anymore.
(However, I would expect that shards evolve over time. So in the this example, after enough repetitions reliably failing to reinforce cake eating, the cake shard would eventually stop making you crave cake when you see cake.)
So in my example, cleaner language might be: For example, I more reliably ate cake in the past if someone was currently offering me the slice of cake, compared to someone promising that they will bring a slightly better cake to the office party tomorrow. So when the "someone is currently offering me something" shard and the "someone is promising me something" shard are both activated, the first shard affects my decisions more, because it was rewarded more reliably in the past.
(One test of this theory might be whether people are more likely to take the bigger, later payout if they grew up in extremely reliable environments where they could always count on the adults to follow through on promises. In that case, their "someone is promising me something" shard should have been reinforced similarly to the "someone is currently offering me something" shard. This is basically one explanation given for the classic Marshmallow Experiment - kids waited if they trusted adults to follow through with the promised two marshmallows; kids ate the marshmallow immediately if they didn't trust adults.)
Cool, I'm happy if you're relaxing with a leisure activity you enjoy! The people I spoke with were explicitly not doing this for fun.
Time inconsistency example: You’ve described shards as context-based predictions of getting reward. One way to model the example would be to imagine there is one shard predicting the chance of being rewarded in the situation where someone is offering you something right now, and another shard predicting the chance you will be rewarded if someone is promising they will give you something tomorrow.
For example, I place a substantially better probability on getting to eat cake if someone is currently offering me the slice of cake, compared to someone promising that they will bring a slightly better cake to the office party tomorrow. (In the second case, they might get sick, or forget, or I might not make it to the party.)
I have lots of points of contact with the world, but it feels really effortful to be always mindful and noting down observations (downright overwhelming if I don't narrowing my focus to a single cluster of datapoints I'm trying to understand)
@Logan, how do you make space for practicing naturalism? It sounds like you rely on ways of easing yourself into curiosity, rather than forcing yourself to pay attention.
(Also, just saw the comment rules for the first time while copying these over - hope mindfulness mention doesn't break them too hard)
Speculating here, I'm guessing Logan is pointing at a subcategory of what I would call mindfulness - a data point-centered version of mindfulness. One of my theories of how experts build their deep models is that they start with thousands of data points. I had been lumping frameworks along with individual observations, but maybe it's worth separating those out. If this is the case, frameworks help make connections more quickly, but the individual data points are how you notice discrepancies, uncover novel insights, and check that your frameworks are working in practice.
(Copying over FB reactions from while reading) Hmm, I'm confused about the Observation post. Logan seems to be using direct observation vs map like I would talk about mindfulness vs mindlessness. Except for a few crucial differences: I would expect mindfulness to mean paying attention fully to one thing, which could include lots of analysis/thinking/etc. that Logan would put in the map category. It feels like we're cutting reality up slightly differently.
Trying to sit with the thought of the territory as the thing that exists as it actually is regardless of whatever I expect. This feels easy to believe for textbook physics, somewhat harder for whatever I'm trying to paint (I have to repeatedly remind myself to actually look at the cloud I'm painting), and really hard for psychology. (Like, I recently told someone that, in theory, if their daily planning is calibrated they should have days where they get more done than planned, but in practice this gets complicated because of interactions between their plans and how quickly/efficiently they work.)
Yet, to the best of my knowledge, psychology isn't *not* physics... It's just that we humans aren't yet good enough at physics to understand psychology.
(Copying over FB reactions from while reading) There’s something that feels familiar so far. For myself and when I’m working with clients, I often encourage experiments and journaling about the experience as they go. Part of the reason is uncertainty about the result, but another part is taking the time to check your expectations against your actual experience as it’s happening.
Like, I recently felt really drained. Noticing that feeling, I could immediately say several things that had been draining, but I wouldn’t have said they were hard while I’d been doing them. I was just caught up in making things go smoothly at the time. It was only afterwards noticing that how drained I felt that made me realize the activity was very energy draining. It would have been easy to brush over the activity with “Oh, I had a fun time”, which was true. But it was also true that I had to recover afterwards.
I feel like there’s something valuable here in being able to pay the kind of attention to my experience that led me to that knowledge. But I don’t have a good vocabulary to describe it to my clients.
Coming back after finishing the series, I notice the "scary!" reaction is gone. Based on some of Logan's comments on the FB thread, I think I updated toward 1. worry less about having to explicitly remember every detail -> instead just learn to pay attention and let those observations filter into your consciousness, and 2. it's still fine to use/learn from frames, just make sure you also have direct observations to let you know if your frames/assumptions are off.
I want to understand how to actually get humans to do the right things, and that task feels gargantuan without building on the foundation of simplified handles other people have discovered.
Yet, I value something like naturalism because I don't trust many handles as they are now, especially coming from psychology. "Confusion-spotting" seems pretty important if I'm going to improve on the status quo.
(Copying FB reactions I made as I read) 1. I care a lot about deep mastery. 2. It doesn’t feel immediately obvious that direct observation is the fastest route to deep mastery. Like, say I wanted to understand a new field - bio for example. I would start with some textbooks, not staring at my dog and hoping I’d understand how he worked. I’d get to examples and direct experience, but my initial instinct is to start with pre-existing frameworks. But maybe I’m just misunderstanding what “direct observation” means?
Probably worth noticing that my mind spent the last half of the post trying to skip ahead. Like, there’s a storyteller setting the scene and my brain wants to skip over “Once upon a time…” ….but I’m guessing that skipping over something because it seems familiar is antithetical to the whole point of this series.
I get the problem about misleading frames and not noticing you’re skipping over counter evidence -- I’m always worried/annoyed/frustrated about how little confidence I have in any claim because I can think of nuances. But ah, that’s why I like frames! I want cached answers, damn it.
Initial reaction: “Ah, scary.” Their move from frames to unfiltered, direct observations feels scary. Like I’m going to lose something important. I rely on frames a lot to organize and remember stuff, because memory is hard and I forget so many important data points. I can chunk lots of individual stuff under a frame.
I wrote reactions on FB while reading, coping them here and on the other posts afterwards.
I don't think this post added anything new to the conversation, both because Elizabeth Van Nostrand's epistemic spot check found essentially the same result previously and because, as I said in the post, it's "the blog equivalent of a null finding."
I still think it's slightly valuable - it's useful to occasionally replicate reviews.
(For me personally, writing this post was quite valuable - it was a good opportunity to examine the evidence for myself, try to appropriately incorporate the different types of evidence into my prior, and form my own opinions for when clients ask me related questions.)
Pro: The piece aimed to bring a set of key ideas to a broad audience in an easily understood, actionable way, and I think it does a fair job of that. I would be very excited to see similar example-filled posts actionably communicating important ideas. (The goal here feels related to this post https://distill.pub/2017/research-debt/)
Con: I don't think it adds new ideas to the conversation. Some people commented on the sale-sy style of the intro, and I think it's a fair criticism. The piece prioritizes engagingness and readability over nuance.
It would be nice if microcovid was updated to take omicron (and future variants) into account. An omicron update would be worth >$10 to me personally (though probably <$100), since it saves me the time of estimating the changing risk myself.
I appreciate and regularly use microcovid to estimate the risks of social gatherings so I can decide how cautious to be socially.
Hmm, that framing doesn't feel at odds with mine. Finding what's rewarding can definitely include whatever it is that's reinforcing the current behavior. I emphasized the gut-level experience because I expect those emotions contain the necessary information that's missing from rational explanations for what they "should" do.
But Ericsson's research found that one group of expert violinists averaged 10,000 hours. Another group of "expert" violinists averaged 5,000 hours, and other numbers he cites for expertise range from 500 to 25,000. So really, it's generalizing from "you should have 10,000 hours of practice by the time you're 20 if you want an international career as a violinist" to "you should get 10,000 hours of practice if you want to be an expert in anything"....
So I put that example because one of the things that felt like a breakthrough in cooking ability for me was seeing a post listing a bunch of world cuisines by spices (I think it was a post by Jeff Kaufman, but I can't find it now). Having a sense of which spices usually contribute to the flavor profile I want made me a better cook than my arbitrary "sniff spice and guess whether that would be good" previous method.
So while you're spending your 10k hours on some creative pursuit, maybe it's worth spending one hour brainstorming these "other means".
Arguably a great example of deliberate practice for finding better methods.
That seems likely. I'm not calling Gladwell out - I also haven't read the book, and there's probably a pretty defensible motte there. However, it seems likely that he laid the foundation for the popular internet version by overstating the evidence for it, e.g. this quote from the book: “The idea that excellence at performing a complex task requires a critical minimum level of practice surfaces again and again in studies of expertise. In fact, researchers have settled on what they believe is the magic number for true expertise: ten thousand hours."
And the rule-run-amok-on-the-internet generally assumes necessary and sufficient, e.g. this quote from Ericsson "The popular internet version of the 10 000 h rule suggests that attaining expert performance is all about getting more and more practice and experience in a given domain of activity and then reaching an expert status at 10 000 h."
Interesting. None of the sleep doctors I spoke to recommended data sources. However, they seemed to consider even at-home professional sleep tests with skepticism, so this might say more about the level of accuracy they want than about the potential usefulness of personal devices.
As for age, I tried to focus this post on actionable advice. The non-actionable factors that influence sleep are simply to numerous for me to cover properly, and, unfortunately, however impactful aging is on sleep, reversing aging isn't (yet!) in my repertoire of recommendations.
Sounds like you're describing autonomy, mastery, and meaning - some of the big factors that are supposed to influence job satisfaction. 80,000 Hours has an old but nice summary here https://80000hours.org/articles/job-satisfaction-research. I expect job satisfaction and the resulting motivation make a huge difference on hours you can work productively.
For retired and homemaking folks, I think that's really up to them. I don't have a good model for external evaluation. For a student who wants to do impactful things later, I think the calculations are similar.
Since I can't link to it easily, I'm reposting a FB post by Rob Wiblin on a similar point:
"There's something kinda funny about how we don't place much value on the time of high school and undergraduate students.
To see that, imagine that person X will very likely be able to do highly valuable work for society and earn a high peak income of say $100 an hour by the time they're 35. As a result they'll work a solid 50-60 hours a week.
But today, as a 19 year old undergraduate, X is only able to earn $15 in hospitality. They also feel they quickly hit declining returns on studying and so, like many undergrads, spend a lot of time goofing off and having fun, because it seems like the opportunity cost of their time is really low.
That's fine as a lifestyle choice, but the whole scenario is also... weird.
If their career advancement is purely determined by how quickly they learn what they need to learn, and generally become fully-fledged adults, then the true opportunity cost of each hour should be closer to $100 than $20.
That's because each extra day of training they do now should bring forward the day they reach their peak productivity by about... a day. Their opportunity cost being low is an illusion stemming from it not yet being tangible and measurable.
If we model career progression as literally just a linear series of steps that take you from zero productivity up to a peak plateau productivity, before then going back down due to the effects of ageing, then the opportunity cost at the outset, before you've learned anything at all is... the productivity at the peak.
Of course many things interfere with this simplified analysis:
• Becoming more productive is partly just a matter of growing older in calendar time, as the brain, body and personality mature.
• Lots and lots of career capital is gained through 'goofing off', following random interests, exploring the world, working on yourself & your mental health, and socialising. People who skip these parts of life often face problem later on. So what looks 'unproductive' will often be as good as or better than formal training.
• If you're on an inflexible path (e.g. becoming a radiologist) there may simply be no way to speed up the rate at which you learn or can start working. You have to go through a series of predetermined steps that suit the average participant, using materials you can't access yourself, and which occur in calendar time no matter what you do.
• People also want to have fun — work and productivity are far from everything.
The main lessons I draw from this are:
• The true opportunity costs of talented young people are higher than they initially appear, maybe much higher.
• When young people can't afford the tools they need to learn most effectively, this is no joke. Rather it's a heartbreaking waste of human capital.
This kind of thing includes: a great laptop, peripherals and desk; ability to commute quickly; a quiet house or room to study in; connection with colleagues to form a study group; a great bed and other things that improve sleep; help with mental and physical health when required; etc. Basically all the stuff that's 'profitable' for 40 year-old professionals who earn a lot and so value every hour of their time.
• Having training systems that allow people to choose to work harder and advance faster are good. At least if they don't eat into valuable informal learning.
• It can be a real waste of society's limited human capital to have high school, undergrad and postgrad students waiting tables to pay the bills, just because they have no collateral with which to borrow against their likely future income."
Maximization of neglectedness gives different results from those of maximization of impact.
I don't disagree, but my point is that you can't directly maximize impact without already knowing a lot. Other people will usually do the work that's very straightforward to do, so the highest counterfactually valuable work requires specialized knowledge or insights.
Obviously there are many paths that are low-impact. Since it's hard to know which are valuable before you learn about them, you should make a theory-of-change hypothesis and start testing that best guess. That way you're more likely to get information that causes you to make a better plan if you're on a bad track.
As I understand it, your objection is that "being the best" means traditional career success (probably high prestige and money), and this isn't a good path for maximizing impact. That makes sense, but I'm not talking about prestige or money (unless you're trying to earn to give). When I say "best," I mean being able to make judgement calls and contributions that the other people working on the issue can't. The knowledge and skills that make you irreplaceable increase your chances of making a difference.
Honestly, the main thing was to start treating my life as an experiment. Before that, I was just doing what the doctors told me without checking to see if their recommendations actually produced good results. For me, experimenting mainly meant that I 1. tried tracking a bunch of things on my own and analyzing the results, and 2. was willing to try a lot more things, like caffeine pills and antidepressants, because the information value was high. (I first did my research and, when relevant, checked with a doctor, of course.) I think there was a mindset shift somewhere along the way explicitly rejecting that the status quo was innately good. If I was unsatisfied with something, I could try to change it, and I was effective if it got better. After I started experimenting, I prioritized experiments to deal with the bottleneck of fatigue and it was fairly straightforward.
I used a Lights sheet ( https://www.ultraworking.com/lights ) to track the variables alongside my daily habits, to reduce overhead.
I also shared the PayPal link out of fairness to Dony, who organizes the group.
Yes, that link is the first reply.