So You Want To Make Marginal Progress...

This post seems to assume that research fields have big, hard central problems that are solved with some specific technique or paradigm.

This isn't always true. Many fields have the property that most of the work is on making small components work slightly better in ways that are very interoperable and don't have complex interactions. For instance, consider the case of making AIs more capable in the current paradigm. There are many different subcomponents which are mostly independent and interact mostly multiplicatively:

• Better training data: This is extremely independent: finding some source of better data or better data filtering can be basically arbitrarily combined with other work on constructing better training data. That's not to say this parallelizes perfectly (given that work on filtering or curation might obsolete some prior piece of work), but just to say that marginal work can often just myopically improve performance.
• Better architectures: This breaks down into a large number of mostly independent categories that typically don't interact non-trivially:
  • All of attention, MLPs, and positional embeddings can be worked on independently.
  • A bunch of hyperparameters can be better understood in parallel
  • Better optimizers and regularization (often insights within a given optimizer like AdamW can be mixed into other optimizers)
  • Often larger scale changes (e.g., mamba) can incorporate many or most components from prior architectures.
• Better optimized kernels / code
• Better hardware

Other examples of fields like this include: medicine, mechanical engineering, education, SAT solving, and computer chess.

I agree that paradigm shifts can invalidate large amounts of prior work (and this has occurred at some point in each of the fields I list above), but it isn't obvious whether this will occur in AI safety prior to human obsolescence. In many fields, this doesn't occur very often.

The fourth friend, Becky the Backward Chainer, started from their hotel in LA and...

Well, no. She started at home with a telephone directory. A directory seems intelligent but is actually a giant look-up table. It gave her the hotel phone number. Ring ring.

Heidi the Hotel Receptionist: Hello?

Becky: Hi, we have a reservation for tomorrow evening. I'm back-chaining here, what's the last thing we'll do before arriving?

Heidi: It's traditional to walk in through the doors to reception. You could park on the street, or we have a parking lot that's a dollar a night. That sounds cheap but it's not because we're in the past. Would you like to reserve a spot?

Becky: Yes please, we're in the past so our car's easy to break into. What's the best way to drive to the parking lot, and what's the best way to get from the parking lot to reception?

Heidi: We have signs from the parking lot to reception. Which way are you driving in from?

Becky: Ah, I don't know, Alice is taking care of that, and she's stepped out to get more string.

Heidi: Oh, sure, can't plan a car trip without string. In the future we'll have pet nanotech spiders that can make string for us, road trips will never be the same. Anyway, you'll probably be coming in via Highway 101, or maybe via the I-5, so give us a buzz when you know.

Becky: Sorry, I'm actually calling from an analogy, so we're planning everything in parallel.

Heidi: No worries, I get stuck in thought experiments all the time. Yesterday my friend opened a box and got a million dollars, no joke. Look, get something to take notes and I'll give you directions from the three main ways you could be coming in.

Becky: Ack! Hang on while I...

Gerald the General Helper: Here's a pen, Becky.

Trevor the Clever: Get off the phone! I need to call a gas station!

Susan the Subproblem Solver: Alice, I found some string and.... Hey, where's Alice?

This doesn't seem right. Suppose there are two main candidates for how to get there, I-5 and J-6 (but who knows, maybe we'll be surprised by a K-7) and I don't know which Alice will choose. Suppose I know there's already a Very General Helper and Kinda Decent Generalizer, then I might say "I assign 65% chance that Alice is going to choose the I-5 and will try to contribute having conditioned on that". This seems like a reasonable thing to do. It might be for naught, but I'd guess in many case the EV of something definitely helpful if we go down Route A is better than the EV of finding something that's helpful no matter the choice.

One should definitely track the major route they're betting on and make updates and maybe switch, but seems okay to say your plan is conditioning on some bigger plan. 

Besides reiterating Ryan Greenblat's objection to the assumption of a single bottleneck problem, I would also like to add that there is apriori value in having many weakly generalizable solutions even if only few will have posteriori value. 

Designing only best-worst-case subproblem solutions while waiting for Alice would be like restricting strategies in game to ones agnostic to the opponent's moves, or only founding startups that solve a modal person's problem. That's not to say that generalizability isn't a good quality, but I think the claim in the article goes a little too far.

There's one common reason I sometimes undervalue weakly-generalizeable solutions (it's not in response to any claim in the article, but I hope it is relevant still): it sucks to be the individual researcher whose work turns out as irrelevant. I think that both in a utilitarian sense and as a personal way to cope with life, it's better to adopt an apriori mindset of meaning and value in one's work, but we're not naturally equipped with it.

Reasoning through a new example:

There's no google maps and no internet to help with finding a hotel. You haven't chosen a destination city yet. 

You could work out how to choose hotels and facilitate the group identifying the kind of hotel it wants. They're both robustly useful. 

You could start picking out hotels in cities at random. Somehow my intuition is that doing this when you don't know the city is still marginally useful (you might choose that city. Obviously more useful the smaller the set of possible cities), but nonzero useful. 

OTOH, one of the best ways to build hotel identifying skills is to identify a hotel, even if you don't use it. A few practice runs choosing hotels in random cities probably does help you make a new reservation in a different city. 

My shoulder John says "dry running hotels is a fine thing to do as long as you're doing it as a part of a plan to get good at a generalizable skill". I agree that's ideal, but not everyone has that skill, and one of the ways to get it is to gradient ascend on gradient ascending.  I worry that rhetoric like this, and related stuff I see in EA and rationality encouraging people to do the most important thing, ends up paralyzing people when what they need is to do anything so they can start iterating on it.

1. In research you don't usually know your precise destination. Maybe LA but definitely not the hotel.
2. research, in general, is about mapping all of California, not just the quickest route between two points and all the friends helped with that.
3. You say "Alice tackled the central bottleneck" but you don't say what that was, only her "solution". Alice is only key here with the benefit of hindsight. If the I5 didn't exist or was closed for some reason then one of her friends solutions might have been better.

If you do not choose robustly generalizable subproblems and find robustly generalizable solutions to them, then most likely, your contribution will not be small; it will be completely worthless.

Though it also feels worth noting that while individually unsatisfying, it's not necessarily collectively wrong for a lot of people to work on subproblems that will turn out to be completely worthless. If one of them turns out to unexpectedly solve the main bottleneck after all, or if several of them do end up making small progress that lets someone else solve the main bottleneck, all of the "wasted" work may in fact have bought us the useful bits that you couldn't have identified in advance.

Often when I've had a hypothesis about something that interests me, I've been happy that there has been *so much* scientific research done on various topics, many of them seemingly insignificant. While most of it is of little interest to me, the fact that there's so much of it means that there's often some prior work on topics that do interest me. And vice versa - much of the work that I find uninteresting, will probably be a useful puzzle piece in someone else's research. (Even if a lot of it is also genuinely useless for everyone, but that's the price we pay for getting some of the more useful bits.)

Seems accurate, though I think Thinking This Through A Bit involved the part of backchaining where you look at approximately where on the map the destination is, and that's what some pro-backchain people are trying to point at. In the non-metaphor, the destination is not well specified by people in most categories, and might be like 50 ft in the air so you need a way to go up or something.

And maybe if you are assisting someone else who has well grounded models, you might be able to subproblem solve within their plan and do good, but you're betting your impact on their direction. Much better to have your own compass or at least a gears model of theirs so you can check and orient reliably.

PS: I brought snacks [LW(p) · GW(p)]!

Curated. I found this a clearer explanation of "how to think about bottlenecks, and things that are not-especially-bottlenecks-but-might-be-helpful" than I previously had. 

Previously, I had thought about major bottlenecks, and I had some vague sense of "well, there definitely seems like there should be more ways to be helpful than just tackling central bottlenecks, but a lot of ways to do that misguidedly." But I didn't have any particular models for thinking about it, and I don't think I could have explained it very well.

I think there are better ways of doing forward-chaining and backward-chaining than listed here (but which roughly correspond to "the one who thought about it a bit," but with a bit more technique for getting traction).

I do think the question of "to what degree is your field shaped like 'there's a central bottleneck that is to a first approximation the only thing that matters here'?" is an important question that hasn't really been argued for here. (I can't recall offhand if John has previously written a post exactly doing that in those terms, although the Gears Which Turn the World [? · GW] sequence is at least looking at the same problemspace)

I don't see how to map this onto scientific progress.  It almost seems to be a rule that most fields spend most of their time divided for years between two competing theories or approaches, maybe because scientists always want a competing theory, and because competing theories take a long time to resolve.  Famous examples include

• geocentric vs heliocentric astronomy
• phlogiston vs oxygen
• wave vs particle
• symbolic AI vs neural networks
• probabilistic vs T/F grammar
• prescriptive vs descriptive grammar
• universal vs particular grammar
• transformer vs LSTM

Instead of a central bottleneck, you have central questions, each with more than one possible answer.  Work consists of working out the details of different experiments to see if they support or refute the possible answers.  Sometimes the two possible answers turn out to be the same (wave vs matrix mechanics), sometimes the supposedly hard opposition between them dissolves (behaviorism vs representationalism), sometimes both remain useful (wave vs particle, transformer vs LSTM), sometimes one is really right and the other is just wrong (phlogiston vs oxygen).

And the whole thing has a fractal structure; each central question produces subsidiary questions to answer when working with one hypothesized answer to the central question.

It's more like trying to get from SF to LA when your map has roads but not intersections, and you have to drive down each road to see whether it connects to the next one or not.  Lots of people work on testing different parts of the map at the same time, and no one's work is wasted, although the people who discover the roads that connect get nearly all the credit, and the ones who discover that certain roads don't connect get very little.

Looks like the hardest part in this model is how to " choose robustly generalizable subproblems and find robustly generalizable solutions to them", right?

How does one do that in any systematic way? What are the examples from your own research experience where this worked well, or at all?

When they're nearing the hotel, Alice gets the car's attention. And she's like, "Listen guys, I have been lying to you. My real name is Mindy. Mindy the Middlechainer.".

Elaborating on The Very General Helper Strategy: the first thing you do when planning a route by hand is find some reasonably up-to-date maps.

One thing that almost always tends to robustly generalize is improving the tools that people use to gather information and make measurements. And this also tends to snowball in unexpected ways - would anyone have guessed beforehand that the most important invention in the history of medicine would turn out to be a better magnifying glass? (And tools can include mathematical techniques, too - being able to run statistical analysis on a computer lets you find a lot of patterns you wouldn't be able to find if it was 1920 and you had to do it all by hand.)

With regards to AI, that might mean interpretability research?

I find this article confusing. So I find myself returning to fundamentals of computer science algorithms: to greedy algorithms and under what conditions they are optimal. Would anyone care to build a bridge from this terminology to what the author is trying to convey?