Useful Vices for Wicked Problems

post by HoldenKarnofsky · 2022-04-12T19:30:22.054Z · LW · GW · 2 comments

Contents

  Laziness
    Hypothesis rearticulation
    Questions for further investigation
    What standard are we trying to reach? How about the easiest one that would still be worth reaching?
  Impatience
    Checkins and virtual checkins
    Outline-driven research
    Constantly track your pace
    Don't just leave a fix for later; duct tape it now
  Hubris
    Your take is valuable
    Your productivity is fine
  Self-preservation
    It’s a bird, it’s a plane, it’s a schmoe!
  Be virtuous first!
  Footnotes
None
2 comments

I've claimed that the best way to learn is by writing about important topics. (Examples I've worked on include: which charity to donate to, whether life has gotten better over time, whether civilization is declining, whether AI could make this the most important century of all time for humanity.)

But I've also said this can be "hard, taxing, exhausting and a bit of a mental health gauntlet," because:

• When trying to write about these sorts of topics, I often find myself needing to constantly revise my goals, and there's no clear way to know whether I'm making progress. That is: trying to write about a topic that I'm learning about is generally a wicked problem.
• I constantly find myself in situations like "I was trying to write up why I think X, but I realized that X isn't quite right, and now I don't know what to write." and "I either have to write something obvious and useless or look into a million more things to write something interesting." and "I'm a week past my self-imposed deadline, and it feels like I have a week to go, but maybe it's actually 12 weeks - that's what happened last time."
• Overall, this is the kind of work where I can't seem to tell how progress is going, or stay on a schedule.

This post goes through some tips I've collected over the years for dealing with these sorts of challenges - both working on them myself, and working with teammates and seeing what works for them.

A lot of what matters for doing this sort of work is coming at it with open-mindedness, self-criticality, attention to detail, and other virtues. But a running theme of this work is that it can be deadly to approach with too much virtue: holding oneself to self-imposed deadlines, trying for too much rigor on every subtopic, and otherwise trying to do "Do everything right, as planned and on time" can drive a person nuts. So this post is focused on a less obvious aspect of what helps with wicked problems, which is useful vices - antidotes to the kind of thoroughness and conscientiousness that lead to unreachable standards, and make wicked problems impossible.

I've organized my tips under the following vices, borrowing from Larry Wall and extending his framework a bit:

• Laziness. When some key question is hard to resolve, often the best move is to just ... not resolve it, and change the thesis of your writeup instead (and change how rigorous you're trying to make it). For example, switching from "These are the best charities" to "These are the charities that are best by the following imperfect criteria."
• Impatience. One of the most crucial tools for this sort of work is interrupting oneself. I could be reading through study after study on some charitable activity (like building wells), when stepping back to ask "Wait, why does this matter for the larger goal again?" could be what I most need to do.
• Hubris. Whatever I was originally arguing ("Charity X is the best"), I'm probably going to realize at some point that I can't actually defend it. This can be demoralizing, even crisis-inducing. I recommend trying to build an unshakable conviction that one has something useful to say, even when one has completely lost track of what that something might be.
• Self-preservation. When you're falling behind, it can be tempting to make a "heroic" effort at superhuman productivity. When a problem seems impossible, it can be tempting to fix your steely gaze on it and DO IT ANYWAY. I recommend the opposite: instead of rising to the challenge, shrink from it and fight another day (when you'll solve some problem other than the one you thought you were going for).

Overall, it's tempting to try to "boil the ocean" and thoroughly examine every aspect of a topic of interest. But the world is too big, and the amount of information is too much. I think the only way to form a view on an important topic is to do a whole lot of simplifying, approximating and skipping steps - aiming for a step of progress rather than a confident resolution.

Laziness

Hypothesis rearticulation

My previous piece focused on "hypothesis rearticulation": instead of defending what I was originally going to argue, I just change what I'm arguing so it's easier to defend. For example, when asking Has Life Gotten Better?, I could've knocked myself out trying to pin down exactly how quality of life changed in each different part of the world between, say, the year 0 and the year 1000. Instead, I focused on saying that that time period is a "mystery" and focused on arguing for why we shouldn't be confident in any of a few tempting narratives.

My previous piece has another example of this move. It's one of the most important moves for answering big questions.

Questions for further investigation

This is really one of my favorites. Every GiveWell report used to have a big section at the bottom called "Questions for further investigation." We'd be working on some question like "What about the possibility that paying for these services (e.g., bednets) just causes the government to invest less in them?" and I'd be like "Would you rather spend another 100 hours on this question, or write down a few sentences about what our best guess is right now, add it to the Questions for Further Investigation section and move on?"

To be clear, sometimes the answer was the former, and I think we eventually did get to ~all of those questions (over the course of years). But still - it's remarkable how often this simple move can save one's project, and create another fun project for someone else to work on!

What standard are we trying to reach? How about the easiest one that would still be worth reaching?

If you're writing an academic paper, you probably have a sense of what counts as "enough evidence" or "enough argumentation" that you've met the standards for a successful paper.

But here I'm trying to answer some broad question like "Where should I donate?" or "Is civilization declining?" that doesn't fit into an established field - and for such a broad question, I'm going to run into a huge number of sub-questions (each of which could be the subject of many papers of its own). It's tempting to try for some standard like "Every claim I make is supported by a recognizably rigorous, conclusive analysis," but that way madness lies.

I think it's often better to aim for the minimum level of rigor that would still make something "the best available answer to the question." But I'm not absolutist about that either - a frustrating aspect of working with me on problems like this is that I'll frequently say things like "Well, we don't need to thoroughly answer objection X, but we should do it if it's pretty quick to do so - that just seems like a good deal." I think this is a fine way to approach things, but it leads to shifting standards.

Here's a slightly more nuanced way to think about how "rigorous" a piece is, when there's no clear standard to meet. I tend to ask: “How hard would it be for a critic to demonstrate that the writeup's conclusion is significantly off in a particular direction, and/or far less robust than the writer has claimed?”¹

The "how hard" question can be answered via something like:

• "A-hardness": minimum hours needed by literally anyone in the world
• "B-hardness": minimum hours needed by any not-super-hard-to-access person, including someone who’s very informed about the topic in question
• "C-hardness": minimum hours needed by a reasonably smart but not very informed critic, looking on their own for flaws

I seem to recall that with GiveWell, we got a lot more successful (in terms of e.g. donor retention) once we got to the point where we could get through an hour-long Q&A with donors with a “satisfying” response to each question - a response that demonstrated that (a) we had thought about the question more/harder than the interlocutor; (b) we had good reason to think it would take us significant time (say 10-100 hours or more) to get a better answer to the question than we had. At this point, I think the C-hardness was at least 10 hours or so - no small achievement, since lots of not-very-informed people know something about some random angle.

(By now, I'd guess that GiveWell’s A-hardness is over 100 hours. But a C-hardness of 10 hours was the first thing to aim for.)

These standards are very different from something like “Each claim is proven with X amount of confidence.” I think that’s appropriate, when you keep in mind that the goal is “most thoughtful take available on a key action-guiding question.”

Impatience

Many people dream of working on a project that puts them in a flow state:

In positive psychology, a flow state, also known ... as being in the zone, is the mental state in which a person performing some activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity. In essence, flow is characterized by the complete absorption in what one does, and a resulting transformation in one's sense of time.

But if you're working on wicked problems, I recommend that you avoid flow states, nice though they may be. (Thanks to Ajeya Cotra for this point.) Maybe you instead want a Harrison Bergeron state: every time you're getting in a groove, you get jolted out of it, completely lose track of what you were doing, and have to reassemble your thoughts.

That's because one of the most productive things you can do when working on a wicked problem is rethink what you're trying to do. The more you interrupt yourself, and the less attached you are to the plan you had, the more times you'll be able to notice that what you're writing isn't coming out as planned, and you should change course.

Checkins and virtual checkins

I think the ideal way to interrupt yourself is to be working with someone else who's engaged in your topic and has experience with similar sorts of work (at Open Philanthropy, this might mean your manager), and constantly ping them to say things like:

• I've started to argue for point X, but I don't think my arguments are that great.
• I'm thinking I should deeply investigate point Y - sound right?
• I'm feeling dread about this next section and I don't really have any idea why. Any thoughts? (A lot of people are hesitant to do this one, but I think it often is exactly the right move!)

I think this is helpful for a few reasons:

• You may have gotten subconsciously attached to the vision you had in your head for what you were going to write, and it's good to get a reaction from someone else who has less of that attachment.
• It's generally just hard to make yourself look at your work with "fresh eyes" as your goal is constantly changing, so bringing in another person is good.
• It's easy to get caught up in a "virtue" narrative when doing this work - "I'm thorough and rigorous and productive, I'm going to answer this question thoroughly and rigorously and do it on time." It's tempting (as I'll get to) to try to overcome hard situations with "heroic effort." But another person is more likely to ask questions like "Well, how long does it usually take you to do this sort of thing?" rather than "Can you make an incredible heroic effort here?" and “What do we think we can do and by when and is it worth it?” rather than “What would failure to do the thing you thought you could do say about you as a person?”

With early GiveWell, I got a huge amount of value from Elie, who consistently wanted to do things far less thoroughly than I wanted to. I probably ended up doing things 3x as thoroughly as he wanted and 1/3 as thoroughly (and so 3x faster!) as I originally wanted - a nice compromise.

These kinds of checkins can be very vulnerable (especially when the topic is something like "I can't accomplish what we both said I would"), and it can be hard to have the kind of relationship that makes them comfortable. It's best if the manager or peer being checked in with starts from a place of being nonjudgmental, remembering the wicked nature of the problem and not being attached to the original goals.

I also recommend imagining an outsider interrupting you to comment on your work - I think this can get you some of those same benefits.

Outline-driven research

I recommend always working off of a complete outline of what you are going to argue and how, which has ideally been reviewed by someone else (or your simulation of someone else) who said "Sure, if you can defend each subpoint in the way you say you can, I'll find this valuable."

That is:

• As soon as possible after you start learning about a topic, write an outline saying: "I think I can show that A seems true using the best available evidence of type X; B seems true using the best available evidence of type Y; therefore, conclusion C is true (slash the best available guess)." Don't spend lots of time in "undirected learning" mode.
• As soon as your attempt to flesh out this outline is failing, prioritize going back to the outline, adjusting it, getting feedback and being ready to go with a new argument. It's easy to say something like "I'm not actually confident in this point, I should investigate it" (as I did here), but I think it's better to interrupt yourself at that point; go back to the outline; redo it with the new plan; and ask whether the whole new plan looks good.
• Outlines don't need to be correct, they just need to be guesses, and they should be constantly changing. They're end-to-end plans for gathering and presenting evidence, not finished products.

Constantly track your pace

I think it's good to consistently revisit your estimate of how quickly you're on pace to finish the project. Not how quickly you want to finish it or originally said you would finish it - how quickly it will be finished if you do all of the remaining sections at about the pace you've done the current ones.

I think a common issue is that someone looks very thoroughly into the first 2-3 subquestions that come up, without noticing that applying this thoroughness to all subquestions would put the project on pace to take years (or maybe decades?) Consistently interrupting yourself to re-estimate time remaining can be a good prompt to re-scope the project.

Don't just leave a fix for later; duct tape it now

This tip comes from Ajeya. When you reach some difficult part of the argument that you haven't thought about enough yet, it's tempting to write "[to do]" and figure you'll come back to it. But this is dangerous:

• It creates an assignment of unknown difficulty for your future self, putting them in the position of feeling obligated to fill in something they may not remember very well.
• It makes it harder to estimate how much time is remaining in the project.
• It poses the risk that you'll come back to fill it in, only to realize that you can't argue the subpoint as well as you thought - meaning you need to change a bunch of other stuff you wrote that relies on it.

Instead, write down the shortest, simplest version of the point you can - focusing on what you currently believe rather than doing a fresh investigation. When you read the piece over again later, if you're not noticing the need for more, then you don't need to do more.

Hubris

Your take is valuable

A common experience with this kind of work is the "too-weak wrong turn": you realize just how much uncertainty there is in the question you're looking into, and how little you really know about it, and how easy it would be for someone to read your end product and say things like: "So? I already knew all of this" and "There's nothing really new here" and "This isn't a definitive take, it's a bunch of guesswork on a bunch of different topics that an expert would know infinitely more about" and such.

This can be demoralizing to the point where it's hard to continue writing, especially once you've put in a lot of time and have figured out most of what you want to say, but are realizing that "what you want to say" is covered in uncertainty and caveats.

It can sometimes be tempting to try to salvage the situation by furiously doing research to produce something more thorough and impressive.

When someone (including myself) is in this type of situation, I often find myself saying the following sort of thing to them:

• "If what you've got so far were trivial and worthless, you wouldn't have felt the pull to write this piece in the first place."
• "Don't find support for what you think, just explain why you already think it."

I think it can be useful to just take "My take on this topic is valuable" as an almost axiomatic backdrop (once one's take has been developed a bit). It doesn't mean more research isn't valuable, but it can shift the attitude from "Furiously trying to find enough documentation that my take feels rigorous" to "Doing whatever extra investigation is worth the extra time, and otherwise just finishing up."

Your productivity is fine

Understanding deadlines. One of the hardest things about working on wicked problems is that it's very hard to say how long a project is supposed to take. For example, in the first year of GiveWell:

• We felt that we absolutely had to launch our initial product by Thanksgiving 2007. Our initial product would be our giving recommendations for our initial five causes: saving lives in Africa, global poverty (focus on Africa), US early childhood care, US education, US job opportunities.
• As we got close to the deadline, we were both pulling all nighters and cutting huge amounts of our planned content - things we had intended to write up or investigate were getting moved to questions for further investigation. At some point we gave up on releasing all five causes and hoped we would get one out in time.
• We got “saving lives in Africa” up on December 6, and “global poverty” sometime not too long after that.
• We hoped to get the remaining causes out in January so we could move on to other things. I believe we got them out in May or so.

The "deadline miss" didn't come from not working hard, it came from having no idea how much work was ahead of us.

Working on wicked problems means navigating:

• Not enough deadline. I think if one doesn't establish expectations for what will get done and by when, one will by default do everything in way too much depth and take roughly forever to finish a project - and will miss out on a lot of important pressure to do things like cutting and reframing the work.
• Too much deadline. On the other hand, if one does set a "deadline," it's likely that this is based on a completely inaccurate sense of what's possible. If one then makes it a point of personal pride to hit the deadline - and sees a miss as a personal failing - this is a recipe for a shame spiral.

Early in a project, I suggest treating a deadline mostly as a "deadline to have a better deadline." Something like: "According to my wildly uninformed guess at how long things should take, I should be done by July 1; hopefully by July 1, I will be able to say something more specific, like 'I've gone through 1/3 of my subquestions, and the remaining 2/3 would take until September 1, which is too long, so I'm re-scoping the project.'"

At the point where one can really reliably say how much time should be remaining, I think one is usually done with the hardest part of the project.

For these sorts of "deadline to have a deadline"s, I tend to make them comically aggressive - for example, “I’m gonna start writing this tomorrow and have it done after like 30 hours of work,” while knowing that I’m actually several months from having my first draft (but that going in with the attitude “I’m basically done already, just writing it down” will speed me up a lot by making me articulate some of the key premises). So I'm both setting absurd goals for what I can accomplish, and preparing to completely let myself off the hook if I fail. Hubris.

Understanding procrastination/incubation. For almost anyone (and certainly for myself), working on wicked problems involves a lot of:

• Feeling "stuck."
• Not knowing what to do next - or worse, feeling like one knows what one is supposed to do next, but finding that the next step just feels painful or aversive or "off."
• Having a ton of trouble moving forward, and likely procrastinating, often a huge amount.

(More at my previous piece.)

In fact, early in the process of working on a wicked problem, I think it's often unrealistic to put in more than a few hours of solid work per day - and unhelpful to compare one's productivity to that of people doing better-defined tasks, where the goals are clear and don't change by the hour.

Working on wicked problems can often be a wild emotional rollercoaster, with lots of moments of self-loathing over being unable to focus, or missing a "deadline," or having to heavily weaken the thing one was trying to say.

It's a tough balance, because I think one really does need to pressure oneself to produce. But especially once one has completed a few projects, I think it's feasible to be simultaneously "failing to make progress" and "knowing that one is still broadly on track, because failing to make progress is part of the process." I think it's sometimes productive to have a certain kind of arrogance, an attitude like: "Yes, I cleared the whole day to work on this project and so far what I have done is played 9 hours of video games. But the last 5 times I did something like this, I was in a broadly similar state, and then got momentum and finished on time. I'm doing great!" The balance to strike is feeling enough urgency to move through the whole procrastinate-produce-rethink process, while having a background sense that "this is all expected and fine" that can prevent excessive personal shame and fear from the "procrastination" and "rethink" parts.

(Personally, I often draft a 15-page document by spending 4 hours failing to write the first paragraph, then 1 hour on the first paragraph, then 1 hour failing to continue, then 1 hour on the rest of the first page, then 4 hours for the remaining 14 pages. If someone tries to interrupt me during the first 4 hours, I tell them I'm working, and that's true as far as I'm concerned!)

Self-preservation

As noted above, working on wicked problems often involves long periods of very low output, with self-imposed deadlines creeping up. This sometimes leads people to try to make up for lost time with a "heroic" effort at superhuman productivity, and to try to handle the hardest parts of a project by just working that much harder.

I'm basically totally against this. An analogy I sometimes use:

Q: When Superman shows up to save the day and realizes his rival is loaded with kryptonite, how should he respond? What’s the best, most virtuous thing he can do in that situation?

A: Fly away as fast as he can, optionally shrieking in terror and letting all the onlookers say “Wow, what a coward.” This is a terrible time to be brave and soldier on! There are so many things Superman can do to be helpful - the single worst thing he can do is go where he won’t succeed.²

If the project is taking "too long," it might be because it was impossible to set a "schedule" for in the first place, and trying to finish it off at a superhuman pace could easily just leave you exhausted, demoralized and still not close to done. Additionally, the next task sometimes seems "scary" because it is actually a bad idea and needs to be rethought.

I generally advise people working on wicked problems to aim for "jogging" rather than "sprinting" - a metaphor I like because it emphasizes that this is fully consistent with trying to finish as fast as possible. In particular, I prefer the goal of "Make at least a bit of progress on 95% of days, and in 100% of weeks" to the goal of "Make so much progress today that it makes up for all my wasted past days." (The former goal is not easy! I think aiming for it requires a lot of interrupting oneself to make sure one isn't spiraling or going down an unproductive rabbit hole - rather than a lot of "trying to pedal to the metal," which can run right into those problems.)

It’s a bird, it’s a plane, it’s a schmoe!

This section is particularly targeted at effective altruists who feel compelled to squeeze every ounce of productivity out of themselves that they can, for moral reasons and not just personal pride. I think this attitude is dangerous, because of the way it leads people to set unrealistic and unsustainable expectations for themselves.

My take: "Whenever you catch yourself planning on being a hero, just stop. If we’re going to save the world, we’re going to do it by being schmoes." That is:

• Plan on being about as focused, productive, and virtuous as people doing similar work on other topics.
• Plan on working a normal number of hours each day, plan on often getting distracted and mucking around, plan on taking as much vacation as other high-productivity people (a lot), plan on having as much going on outside of work as other high-productivity people (a lot), etc.
• (This is also a standard to hold oneself to - try not to lose productivity over things, like guilt spirals, that other people doing similar work often don't suffer from.)

If effective altruists are going to have outsized impact on the world, I think it will be mostly thanks to the unusual questions they’re asking and the unusual goals they’re interested in, not unusual dedication/productivity/virtue. I model myself as “Basically like a hedge fund guy but doing more valuable stuff,” not as “A being capable of exceptional output or exceptional sacrifice.”

Be virtuous first!

I don't think you're going to get very far with these "vices" alone. If you aren't balancing them with the virtues of open-mindedness, self-criticality, and doing the hard work to understand things, it's easy to just lazily write down some belief you have, cite a bit of evidence that you haven't looked at carefully or considered the best counter-arguments to, and hit "publish." I think this is what the vast majority of people "investigating" important questions are doing, and if I were writing tips for the average person in the world, I'd have a very different emphasis.

For forming opinions and writing useful pieces about important topics, I think the first hurdle to clear is being determined to examine the strongest parts of both sides of an argument, understand them in detail (and with minimal trust), and write what you're finding with reasoning transparency. (All of this is much easier said than done.) But in my experience, many of the people who are strongest in these "virtues" veer too far in the virtuous direction and end up punishing themselves for missing unrealistic self-imposed deadlines on impossible self-imposed assignments. This piece has tried to give a feel for when and how to pull back, skip steps, and go easy on oneself, to make incremental progress on intimidating questions.

 Comment/discuss [EA · GW]

Footnotes

2 comments

Comments sorted by top scores.