The Pareto Best and the Curse of Doom

post by Screwtape · 2024-02-21T23:10:01.359Z · LW · GW · 21 comments

Contents

  I.
  II.
  III.
  IV.
  V.
  VI.
None
21 comments

I.

Prerequisite reading: Being the (Pareto) Best in the World [LW · GW].

A summary of Being the (Pareto) Best in the World: Being the world's best mathematician is hard. Being the world's best musician is hard. Being the world's best mathematician/musician is much easier, especially since there are multiple slots; an amazing mathematician who is also a competent musician, someone who is good at both, and a competent mathematician who is also an amazing musician can all find a niche.

I like this concept, and have kept it in my back pocket ever since I read it. I have sometimes described myself as a software engineer who was competent at public speaking and project management. That particular overlapping skillset is, it turns out, fairly valuable. While I was attempting to become a better software engineer, I was also trying to add competence at corporate budgets and accounting to that skillset.

These days I spend a lot of time talking to the kind of person who hangs out on LessWrong a lot or spends a lot of time going to Astral Codex Ten meetups. If ever I faced a problem that required a brilliant neuroscientist, or a gifted Haskell programmer, or a world leading expert in training honeybees, well, let's just say I know somebody. There are people out there who are exemplary  at the thing they do.

Sometimes they're not very good at other things though. While Being The (Pareto) Best in the World felt optimistic when I first read it, these days I regard it as a curse of doom upon the world, blighting otherwise promising areas of effort and endeavor. I look around at places where it feels like everyone is dropping the ball and see a blasted wasteland where nothing grows because nobody has the right combination of seemingly basic skills.

II.

Imagine a toy model where everyone has a hundred points to put into being good at things.

(This is, to be clear, not just a toy model but an incorrect model. It's easy to look at your incoming university students and notice a strong inverse correlation between math and verbal SAT scores, forgetting that those get summed together during applications and anyone below a certain threshold probably has their application discarded. Still, let's use this model for the moment.)

Leading talents in a field maybe put 75 points in their area. Why not 100? Because you need points in living your life. There's an archetype of the absent minded professor, someone who can understand a complex abstract subject but who shows up to give lectures having forgotten to put their shoes on or eat breakfast. Hitting 90 points in your field requires someone else to do a lot of the upkeep for you; many FAANG jobs provide food and other amenities, and I don't think it's entirely because it's a cheap perk. Politely, I know some FAANG engineers who I suspect would forget lunch and dinner if it was not conveniently provided for them.

At sufficiently high levels of dedication, seemingly important related skills start to fall by the wayside. Many programmers are not good at documenting their code, writing or reading specifications, or estimating story points and timelines. Fiction authors vary wildly in their comfort with self-promotion, proofreading, and layout. That's what publishers and agents are for. There's a few indie musicians I enjoy whose mastery of sound mixing or recording technology is not the equal to their actual playing. You can spend 40 points on singing, 40 points on recording, and 20 points on living your life. At this point, you're giving up some noticeable quality somewhere. I'll arbitrarily draw a line at 50 points and say this is where so-called "professional" quality tends to hang out, the people you see do their thing and you think "man, they could make a living at that." 

(That line is wrong about who can get paid for doing the thing, incidentally; I can assure you that people get hired as software engineers all the time without half their skill points being in code.)

I had a slightly unusual track through my university major. The way I gloss it sometimes is that I basically took the first year of Information Security classes, the first year of Software Engineering classes, the first year of Network Administrator classes, and the first year of Electrical Engineering classes. Then I went and rounded that out with technical writing and customer support. I do not actually recommend this. Sure, there's overlap and neat tricks I discovered by moving between those disciplines, but by splitting my attention this model suggests I have something like 15 points in each of them. 

My favourite thing to do in technology is to spot some weird trick that only makes sense if you understand how the software works and how the hardware works. Getting random numbers out of the ambient static charge, reading data off a drive faster because you understand how the drive head naturally moves across the storage media, and other such chicanery. I appreciate videogames where art and misdirection conspire with the programming techniques to pull off something neither one could do alone.

This is very visible with fiction authors. The Martian by Andy Weir is one of my favourite books, but it only works because the author put points in both writing and space exploration. Ali Hazelwood's The Love Hypothesis is a cliche romance novel about a biology Ph.D. student worrying about grants and conference talks and also the tall dark and handsome brooding man who obviously isn’t into her at all, and if that sounds like your thing I suspect it absolutely is your thing. Scott Alexander’s Unsong is from an author who leveled up writing, biblical trivia, and terrible whale puns, therefore it is justly beloved. The authors needed the skillset of writing engaging stories, yes, but they also needed a whole different skillset to describe the particular details and aspects of the story they paid attention to.

III.

Those are the positive examples. 

That is, The Martian is an example of where humanity did have someone with the right points in writing and in space. Now take a moment and think about all the books like it that fail to exist because it would take someone who is skilled at writing and also, say, french cooking, or automotive repair. 

The negative examples are the things that fail to exist because there aren't enough people with that overlap of skills. The Martian for automotive repair might  exist, but I haven't heard of it. 

One piece of advice I heard in university is that if you know a lot about a topic that isn't software and you're also good at software, there might be a profitable software company waiting to happen where you write code to help people do the non-code thing. I think it's good advice, but the generalized form of it applies to more than software; Lin Manuel Miranda managed to turn "good at writing music" and "knows a lot about Alexander Hamilton" into a smash hit. I really want to listen to musicals about space travel. (There's Moon Landing. I don't think it's good. There's War of the Worlds, which has fun bits but also doesn't seem great. There's also October Sky, which is stuck in some kind of licensing hell; I genuinely considered taking a road trip to West Virginia to see it because I haven't been able to find the cast recording for love or money except it's not on their schedule anymore either.)

I often despair at how hard it is to find good, well written instructions on how to do some task or skill I want to learn. The vastness of the internet at hand, and often the best thing I can find is a youtube video with questionable lighting and camera angles. This makes sense! "Good at home repair" is one skill, and "writing clearly" is a second skill. I kind of have to hope they coincide by chance in one person to get the directions I want.

The archetypal software startup has a technical founder and a business founder. I think this is because it's very rare to find someone with lots of points in technology and also a lot of points in how to price and sell things. One of the advantages of companies whose product is software or business functions (think Replit or Stripe) is that your CTO or CEO has some domain expertise; if you have a software company that does medical imaging, you really want points in medical imaging as well as running a company and writing software. 

Look around you at all the projects you think obviously would be valuable and seem like they wouldn't be that hard. Consider how many of them involve multiple skillsets. That's the graveyard of the Pareto Best and the Curse of Doom.

IV.

Project Management and Operations Management are skills. People do not by default have these skills. At some point as you attempt to scale a project up from a few people and an afternoon to something more ambitious, I strongly suspect you will feel the lack of these skills even if you don’t know what it is you’re missing. 

Let me rephrase that in a less abstract and less corporatized way. Answering incoming emails, remembering deadlines, and making sure finished work gets filed correctly are all useful across lots of different projects. It doesn’t matter if you’re running a bank, a car repair shop, or a circus, someone probably needs to remember to invoice the customer. It turns out there’s a lot of these cross-subject skills.

There’s a lovely post about the legal structure of Effective Altruist organizations [LW · GW] which points out many separate projects are often under one legal roof, because “In the short term, forming a legally independent organization seems like a lot of money and effort for the privilege of doing more paperwork.” I think this is correct, and that it’s not just time and effort; forming a legally independent organization involves some skills which someone might not have. 

In the comments some people suggested it wasn’t that difficult, and my response to that inspired this whole post. Probably every company or formal organization wants someone who knows how to form a legal organization, but forming legal organizations is a very niche hobby. We shouldn’t be surprised if most people don’t have that skillset.

If you see lots of people working in some area, but they’re all working at small scale and they’re frequently disorganized (no project management) and they often seem inefficient (no operations management) then you are likely looking at a gap caused by the Pareto Best and the Curse of Doom. 

V.

How do we fight The Curse of Doom?

Good question. 

One answer is teamwork. Look at the example of the software startup. If you combine two people who can work well together, then they double the range of skills they have access to. Better yet, they can deliberately search out the skills they need instead. If a project needs skilled graphic design and also expert juggling, then it's much faster to find an expert juggler than for a graphic designer to learn how to juggle. When you see a gap in the world that's half in your range, consider what skill you're missing and go look for them. In the background as you go through life, keep an eye out for people with unusual or deep skillsets and remember them. They might be the missing piece, and might be just as excited to work with you as you are to work with them.

(That said, there's advantages to combining the skills in the same person, especially when it comes to noticing that a problem would be easy to solve if you combined both halves. Business majors routinely misjudge what problems are easy or hard for software to solve, and software developers routinely misjudge what problems customers would be willing to pay to solve.)

Another answer is to try and solve things at the amateur level. Going back to the simple model from before, you just won't find a single person with 60 points in Architecture and 60 points in Teaching. If the problem looks like an Architecture/Teaching problem, and you're a brilliant architect, you might genuinely be one of the best people for the job even if you only sunk 10 points in Teaching. 

(That said, there are possibilities that open up or unlock only once you're actually good at a skill. There are programming concepts I know now and didn't know at fifteen years old which make some tasks easy that would otherwise have been really hard to try. Conversely, I'm an amateur musician who does not have enough skill to pull off "Hamilton But For the Space Race" no matter how much I know about the Apollo missions.)

There's also the option of searching really hard. There are eight billion humans in the world. If there is a powerful need for a master acrobat who is also highly versed in U.S. contract law and additionally is capable of new discoveries in graph theory, they might be out there. Start with the list of top mathematicians, scan for people who also perform at circus shows, and then talk to whoever is left and see what they know about contracts.

(That could be an empty set. Reality is not obliged to offer us every combination of every skillset. People don't cap at a hundred points summed across all skills, but there is a finite amount of human lifespan and only so much time in the day. Or maybe they do exist, but there's only two of them and they're already busy with more important projects.)

Lastly, if a gap is truly important to you there's always the chance that you, yes you, could build up the missing skills you need in yourself. It would take time and effort, but many important problems are worth it. Focus on the place where you feel shocked everyone's dropping the ball [LW · GW]. 

VI.

Most of what seems like unpicked low hanging fruit to me are, when I look deeper, low hanging because the skills involved are spread across many fields. The problem involves combinations of several skills, in weird or unusual configurations. Not all problems are like this, but many are.

That's the Pareto Best and the Curse of Doom. It's a graveyard of solutions that would be quick and easy and just take 10 points in four different areas, each area low level enough that anyone in the field could do it, but spread out in such a way nobody has had all the right keys at the same time and looked in the right place. This curse frustrates me deeply.

The curse isn't going to go away. The problem won't magically solve itself. The gaps matter, some of them are things I care about a lot. This is just the state of the world. 

There is hope. As we get better at automating or outsourcing things, the required skills shrink for any given project. If I want to make a new card game, I no longer need to find someone who knows how printing on cards works (there are companies that take the image files and print them for you) and I no longer need to find a good artist (Dall-E and MidJourney are fine for prototyping and make commissions easier) and I don't need to build my own payment processor and inventory tracker (thank you Stripe and Shoprocket!) but can focus on game design. 

The best answer I have is to pick myself up and keep looking through the stack of problems, solving the ones I do have the skillset for, and keeping an eye out for when circumstances change to make things easier.

21 comments

Comments sorted by top scores.