Almost everyone I’ve met would be well-served thinking more about what to focus on

post by Henrik Karlsson (henrik-karlsson) · 2024-01-05T21:01:27.861Z · LW · GW · 8 comments

This is a link post for https://www.henrikkarlsson.xyz/p/multi-armed-bandit

Contents

  Life is a multi-armed bandit
  Explore: doggedly looking for what makes you feel alive
  Exploit: trimming everything but your top 1-3 priorities makes life richer
  The explore/exploit trade-off
None
8 comments

Almost everyone I’ve ever met would be well-served by spending more time thinking about what to focus on. —Sam Altman

In May 2020, we parked two moving trucks in the harbor and carried everything we owned from one to the other. Johanna, Maud, and I were leaving Sweden, and Covid restrictions meant we were forbidden from returning once we boarded the ferry. Hence the second truck, which we had gotten a stranger to ferry from the island to us: the Swedish truck had to stay in Sweden.

The motivation to leave was that we wanted to homeschool Maud, who was 3. In Sweden, this is illegal, so most Swedish homeschoolers end up on one of two islands in the Baltic Sea. On our island, we knew no one. We had no jobs awaiting. We were leaving something, more than going somewhere. The life we had grown piecemeal over 30 years disappeared overnight. We had to figure out what to replace it with. Should I start another software consultancy to support us? Could I write? How would we find a meaningful social context?

Life is a multi-armed bandit

The moldy apartment we rented as we looked for a house has a view of the sea. Every day, deep into winter, I’d walk down to the water and dive from the cliffs. Swimming in the channels between the rocks, I realized I could model our situation using a concept from probability theory.

It was a multi-armed bandit problem. This problem, which, under a different name, had first been studied by the biologist William R. Thompson in 1933, centers on a rather surreal thought experiment. A gambler faces a slot machine (“a one-armed bandit”), except this machine doesn’t have one arm—following some twisted dream logic, it has k arms, arms sticking out in every direction. Some of these arms have a high probability of paying out the jackpot, others are worse. But the gambler does not know which is which.

The problem is pulling the arms in an order that maximizes the expected total gains. ("Gains" could be anything. Early on, the problem was used to design drug trials. There, the jackpot was defined as finding a successful treatment. If you are looking for a partner, talking to people is how you pull the multi-armed bandit and the resonance (or lack thereof) is the payoff.)

The gambler needs to learn new knowledge about the machines and simultaneously use what they have already learned to optimize their decisions. In the literature, these two activities are referred to as exploring and exploiting. You can’t do both things at the same time. When you explore, you are pulling new arms on the bandit trying to figure out their expected payout. When you exploit, you pull the best arm you’ve found. You need to find the right balance. If you spend too little time exploring, you get stuck playing a machine with a low expected payoff. But if you spend too much time exploring, you will earn less than you would if you played the best arm. This is the explore/exploit trade-off.

People tend to gravitate to different sides of the explore/exploit spectrum. If you are high on openness, like I am, exploring comes easy. But it is harder to make a commitment and exploit what you’ve learned about yourself and the world. Other people are more committed, but risk being too conventional in their choices. They miss better avenues for their effort. Most, however, tend to do less than optimal of both—not exploring, not exploiting; but doing things out of blind habit, and half-heartedly.

First, I’ll say a few words about exploration and exploitation in real life. Then I'll return to the question of how to navigate the tradeoff between them.

Explore: doggedly looking for what makes you feel alive

There are two kinds of people. Those who do not understand how complex the world is, and those who know that they do not understand how complex the world is.

To navigate life we create mental models of the world out there, and then we confuse the models for reality. Have you ever noticed this when interacting with someone who has a less accurate model than you: that it’s like they have a VR headset on and are fighting against monsters you know are not there? For instance, there was a Swedish couple who were working at the embassy in New Delhi in the 1990s. They had a housekeeper. On the housekeeper’s birthday, they made her a cake and invited her to eat at their table. She refused. She insisted she had to eat on the floor, or else she’d be reborn as a lower animal. It is tempting to say, “You can take off the headset, there is nothing there. Have some cake.” But we all have headsets on. There is no way of living in direct contact with reality.

The trick is to collide your mental model with the outside world as often as possible. This is what exploring does. You think you know the distribution of payoffs of the slot machines, but you try something new. You discover that you were wrong. You update your model.

Many of the mental models I have are things I’ve picked up from others. On closer inspection, it turns up that they too picked up from someone else, who picked it up from someone else—going back to someone who lived in the 1950s. This is not the 1950s. Have some cake.

For instance, as I wrote in “A blog post is a very long and complex search query to find fascinating people and make them route interesting things to your inbox,” I wasn’t able to get my blog to “work” until I unlearned the patterns of communication I had picked up from mass media. Blogs are not mass media. They are more powerful than that. I was stuck in the 50s.

Some mental models have more leverage than others. I try to focus on these. Realizing that I didn’t understand process, or priorities, or relationships, was highly rewarding. Understanding those areas better pays off every day.

If you can break inaccurate mental models, life becomes easier to navigate. But how do you do that? I know two ways.

Find people who understand things better than you and read what they have to say. Read with the intention of answering your questions. If you can’t find the answers, email them.

Perform experiments. By this I don’t mean do random things. I mean, STATE YOUR ASSUMPTIONS and FIND WAYS TO TEST IF THEY ARE FALSE. Most of the time, the slot machine of an experiment yields nothing. But that’s ok. A few will rearrange the world around you.

But, as I was saying before, there is a tradeoff. Time spent exploring to gather new information means less time acting on it. Besides, exploiting is often more valuable than it seems, since narrowing your focus to “slot machines” you know are promising can have nonlinear returns.

Exploit: trimming everything but your top 1-3 priorities makes life richer

As a rule of thumb, you can only do 1 or 2 things well. Some people are exceptional: they can do 3. I’m not exceptional.

I learned this, as many do, when I had my first child. I had been a bit nervous about becoming a father. Having failed to achieve what I had expected I would, I thought strapping a child to my chest meant setting myself up for permanent failure. It did not. When Maud ate about half my time, I had to force myself to make priorities: I would care for her, I would write, and I would say no to everything else.

Narrowing my life like this, at least doubled how much I could achieve. When I had more time, I had spread myself too thin to get stuff done.

This is something I now notice whenever I read biographies of people who have done exceptional work: they lived narrow lives. They allowed themselves to care about less than others do. To take two quotes at random, here is Jony Ives who designed the iPhone:

One of the things Steve [Jobs] would say [to me] because he was worried I wasn’t focused — he would say, “How many things have you said no to?” I would tell him I said no to this. And I said no to that. But he knew I wasn’t interested in doing those things. There was no sacrifice in saying no [to those things]. What focus means is saying no to something that with every bone in your body you think is a phenomenal idea, you wake up thinking about it, but you say no to it because you are focusing on something else.”

And here is Werner Herzog, the German filmmaker:

Although for many years I lived hand to mouth—sometimes in semi-poverty—I have lived like a rich man ever since I started making films. Throughout my life I have been able to do what I truly love, which is more valuable than any cash you could throw at me. At a time when friends were establishing themselves by getting university degrees, going into business, building careers and buying houses, I was making films, investing everything back into my work.

In the terminology of multi-armed bandits, they found a good arm. Then they exploited it to the exclusion of everything else.

Why do some people achieve so many of the things they want, and others not? Do people have a fixed budget of things they can achieve in a lifetime? It doesn’t seem so. Rather, it seems like our achievement budget is a function of the number of priorities we have. Interestingly, it seems to be a nonlinear function. Meaning that if you go from 4 priorities to 3, you can get, say, 10 percent more done; but if you go from 4 to 1, you get 400 percent more done. (I’m obviously making these numbers up.) If I look at Elon Musk, I have a hard time even grasping that he has the same number of hours in a day as I have. But he has, of course. We all do. It is just that his decisions have compounded because of a sharp focus.

Why would focus compound? Part of it is time. If you care about less, you spend more time doing what you care about most. Also, you are always nonconsciously processing the thing you focus on. So cutting priorities means you work even when it looks like you’re not working. These days, I’ll spend the afternoon playing with the kids, doing the dishes, repairing the houses—being busy in a mind-clearing way. Then, when I sit down to write the next morning, I can type 700 words without thinking. The ideas have been churning in my head, just below the surface of conscious thought, and come fully formed.

When I was younger, I was never this lucky. It is partly because I was less skilled. But it is also partly because I would interrupt the nonconscious processing back then. Unintentionally, I would tell my brain to focus on something else—a conflict in a TV series I was watching, for instance. I would watch an episode before bed, and the cliffhanger would open a loop in my head. That loop would be churning in my head as I slept; I woke to a blank page. I don’t have time for that anymore. I make sure to always have an open loop concerning my writing. And I close every other loop—by wrapping it up as fast as I can, or by writing it down on a list, or, preferably, by not opening the loop at all.

Allocating more time and mental processing power alone doesn’t explain the nonlinearity, though. More time is just a linear increase. I would guess the nonlinearity comes from:

• Focus accelerates the accumulation of skills and accurate world models. In open-ended domains, such as writing, relationships, or business, there is nearly endless room for skill growth. When you spend more time, you get a better model of the situation which allows you to allocate your time better, which accelerates your rate of learning, and so on, in a nonlinear way.
• Focus attracts “resources.” This is obvious in business: if you have a ferocious focus investors will start following you around begging you to take their money. Then you can use the money to pull even faster on the arms of the bandit. This is true in writing, too: the more I write, the more interesting people my blog attracts. They start giving me feedback and advice which helps me write better, which, in a flywheel, attracts more interesting people (and some money). If you are curious and kind to people around you, you attract strong supportive networks. Networks have nonlinear properties.

But for me, as a person for whom narrow focus is against my instincts, the most remarkable thing about it is how rich it feels. My life these days is small and boring. I bicycle across the same fields every day, I notice how the wind turbines turn to face the wind, I rarely travel, and I spend my spare time staring at a Word document. Annie Dillard calls the writer’s life colorless to the point of sensory deprivation. That fits. But, as she also knows, there is another kind of color that can only be discovered three years down a writing hole. It is a subtle, nightly color; your eyes need time to adjust to the dark before you can see them. You wouldn't believe their beauty if I told you.

The explore/exploit trade-off

So, as I was saying, I was swimming by the cliffs. Every day, walking up and down the coast I found new places to dive in. I got interested in high diving—something I had been too much of a coward to do as a kid. I felt a childish excitement for water. The kids on the beach would look at me, whisper, and laugh. That is one of the pleasures of youth. The pleasure of being a dad is knowing I was having more fun than them.

But beyond being a dad, I didn’t know what to do with my life. Swimming and thinking about the things I’ve said in this essay, I made a decision. I would approach my situation algorithmically. I would apply rules to decide when to explore and when to exploit to counteract my natural tendencies (which would push me to do too little of both).

There are several algorithmic solutions to the multi-armed bandit problem, going back to Thompson sampling in the 1930s, all the way up to contemporary algorithms used in machine learning, such as EXP3 and Upper Confidence Bounds. What they all have in common is some version of: 1) prioritize exploration early on and 2) dial up exploitation as the situation becomes more clear. If you are new in a city, it makes sense to meet as many people as possible. If you find someone you love hanging with early, you will have years of happiness. But if you are about to leave, it makes more sense to hang with your best friends. Even if you found someone you liked more, you wouldn’t have any time to hang out. The amount of exploration that is optimal depends on the complexity of the problem and the time horizon.

What I decided to do was this. I would divide the next 30 months into three parts. For the first ten months, I would allow myself to explore freely. After that, I’d switch to exploring 2/3 of the time and using 1/3 to double down on the most interesting opportunity I had found, then I’d do 1/3 exploring and 2/3 exploiting, and so on.

We can think of the amount of time I spent on explorative open-ended search as my “temperature.” When you heat atoms, they bounce around faster; when you heat a life, it becomes more explorative. When you cool it, you narrow down and spend more time exploiting what you know. Here’s what it looks like when you search for the highest peak while gradually reducing the “temperature”:

Notice that the search starts in the area to the right, which does not contain the highest peak. If it had been a focused search early on it would have gotten stuck at a local optimum. (This is one of the reasons why education is a nightmare for many, by the way. Schooling forces you to decide what you will pursue long before you get to try it. It makes your future, more knowledgeable self, the servant of the younger, ignorant self.)

It felt comforting to put strict rules on myself like this, especially in the early part of the search. I felt stressed that I would not find the right thing to spend my time on and support my family. This made me want to go into exploit mode too soon. Knowing that there would be a time for focus later made me more willing to try anything and fail. I learned the piano and played around with coding. I wrote a few chapters of a novel. I got involved with a kindergarten/goat farm and studied the history of the island. I took a job at an art gallery.

After 10 months, I reduced the amount of randomness for the first time. I decided to spend 1/3 of my spare time programming. But I kept exploring 2/3s of my days.

The next time I dialed down the temperature I had started this blog. Since writing it was more interesting than programming, I stopped coding and switched 2/3s of my spare time to the blog. In retrospect, this seems like an obvious choice. Writing had always been my main obsession. But it wasn’t obvious at the time. I had for years been frustrated that no one was interested in what I wrote. The magazines that had published me when I was in my early twenties wanted nothing to do with the stuff I was exploring now. Writing seemed like a dead end.

I didn’t tell anyone that I was writing a blog. Having my friends read it would have made it harder for me to experiment and do things that risked embarrassment or failure. I wanted to put myself in a social context where I was rewarded for exploring my illegible potential.

It worked. In January 2023, I dialed down the temperature to 0. I had landed on a peak I could not have imagined when we arrived on the island: being on track to supporting my family by emailing my thoughts to strangers. Even better, it is a small family writing business since Johanna and I work as a team these days (and we project that the essays will be our main income by the end of 2024).

If I had decided on a path when I swam by the cliffs, before exploring, this possibility would not have occurred to me. Thoughtful emails did not seem like the type of thing that can support a family. Had I known it was, I would have thought I lacked what it takes to pull it off. This is an important thing to keep in mind: you don’t know until you try.

8 comments

Comments sorted by top scores.