Selective, Corrective, Structural: Three Ways of Making Social Systems Work

Problems with structural methods:

Designing structures and systems is hard. Designing structures and systems that will act upon people, and will channel their behavior into a form consonant with your goals, is very hard.

People are different. Different people will interact with your system in ways different enough that it would be impossible for your design to handle them all.

People have goals of their own. They will exert optimization pressure upon your system, to make their desired results come about; some of these results will be at odds with what you want; and you cannot foresee everything that anyone might do to, with, or in your system, to ensure that people’s behavior has only those results you want.

Make your structures rigid enough to minimize the odds of undesired outcomes, and that same rigidity will constrain you as well. You cannot build your system out of manacles and chains, if you wish anything of any consequence to come of it.

Make your structures flexible, and you make them corruptible.

There is no structure so perfect that none may be found who are so stupid or so evil as to wreck it.

Here you may do well to remember that the real rules have no exceptions [LW · GW]. But what are “the real rules” but your (and your system’s) values?

How will you ensure that those remain “the real rules”, except by never giving power within your system to anyone who isn’t aligned with your system’s values?

Now you once again face a selection task.

More problems: designing structures being hard, will you do it on your own? Or will you work with others? How will you be sure that they are suitable? If only the ideal people could be found for the task… and now you’re back to the beginning.

The regress is not infinite, surely, but any systems we have, from which to start the process—the recursive base case—already exist. They are not perfect structures, surely! So, at some point, you must rely on other methods than only structural ones.

Problems with selective methods:

What if it’s hard to tell, in advance, who is “the right sort of people”, and who is “the wrong sort”?

Well, perhaps that’s not so bad, if the decision to include is not irrevocable; if, having included someone, you can easily exclude them later.

But what if it’s not easy to exclude them later?

If you depend on having only the right sort of person within your system, then might a wrong sort of person, included even temporarily, do serious damage?

What if removing someone is very costly?

All of these are only the risks associated with false positives. What about false negatives?

What if it’s hard to identify the right sort of person, and such people are so rare (and/or you need so many of them) that only including those you’ve identified as the right sort means that you miss out on many people whose inclusion would substantially increase your system’s value, and without whom you risk failure?

Also, are there costs imposed by your selection procedure itself? (These costs might be borne by the system and those within it, by prospective members, or by uninvolved others—the latter sort usually being known as “externalities”.)

Such costs may actually be (part of) the selection mechanism—but they aren’t always.

Are there consequences of the use of selective methods, that shape your system in ways other than the simple fact of the methods’ results?

Problems with corrective methods:

To a first approximation, people do not change.

To a second approximation, people only change by their own volition, but cannot be changed otherwise.

Why should anyone who is not already “the right sort of people” to fit into your system, wish to change so as to make themselves that sort?

It can only be due to deep value alignment with your systems’s goals.

But now you have a selective task: to find those whose values are deeply aligned with those of your system. You import all the problems of selective methods [LW(p) · GW(p)], then; and add to that, the challenge of correction.

Without value alignment, you can incentivize volition toward change. But what you actually incentivize is the appearance of that change, or some approximation. Is it good enough? It might be, in some cases. Some tasks are more demanding than others; approximation may serve.

Correction requires an existing status differential. Without such, it is perceived as a bid for greater status than you (or your system) have already. People resist such bids.

Even assuming away all of the above, correction is hard. It takes a greater degree of mastery to teach a skill than to use it or to recognize it.

People are different; they vary in potential. To assume that anyone, even granting the most sincere and fervent wish to change, may become as they must be in order that they may play a role in your system, is to set yourself up for disappointment and failure. But recognizing this, you must again select. (On the other hand, it may be an easier selection task… or it may not be.)

Let's apply this framework to the Searching for Outliers [LW · GW] essay. I suppose the lesson there is that, for sufficiently heavy-tailed outcomes (where, say, some people are a 1000x better fit than the average), selective methods (i.e. searching for outliers) dominate over corrective methods (e.g. better training). And then it talks about how to do selection well.

I like this classification. Clear, crisp, easy to use. I'll see if I will utilize it in my daily thinking.

Of course, the sharp division into three types of methods should not be taken too seriously; nor, and especially, does the division imply that a choice must be made. The methods often overlap and interoperate, to a degree that makes them hard to cleanly separate in practice.

Consider the simplest of the three examples in the post: a World of Warcraft raiding guild. (See my previous [LW · GW] posts [LW · GW] about rationality lessons from WoW for some background.)

Almost nobody will care about putting in the effort to improve, to learn to play their class/role better, unless there’s some incentive to do so. Likewise, the approbation/disapprobation of their guild mates will mean nothing unless players have some reason to care about their social status within the guild. These incentives and reasons generally are “people who perform badly aren’t invited to raids, or don’t have permanent spots on raid teams; thus they have fewer chances at acquiring loot”.

In other words, what motivation do people have to submit to correction?

Thus corrective methods rely on selective and structural methods.

Meanwhile, saying “assign raid members to roles that best fit their talents and inclinations” sounds properly humane and considerate, but the reality is that some players aren’t good enough to play any useful role in a raid (in some cases, this can be corrected; in others, not).

And, likewise, some people are bad actors. In any system of incentives and rewards, they will devote most of their energy to exploiting the system (and patching to prevent exploits often makes the system worse overall for everyone else).

Thus structural methods rely on selective and corrective methods.

At the same time, it often does no good to try to select players who are “good” in some sense that refers to the final output (raid performance per se).

For one thing, there is such a thing as “culture fit” (yes, even in WoW); someone may fit well into the system—the social environment, the raid organization scheme, etc.—that you’ve created, or fit badly, resulting in a dimension of contribution effectiveness which is not perfectly correlated with “objective” measures of raid performance that can be applied across guilds.

Similarly, because a raiding guild must, by its nature, be able to adapt to new challenges (which means both “new raid content” and “new organizational and task challenges, created by shifting membership and raid composition”), adaptability and the capacity and willingness to learn and improve is very often a more important trait for a prospective raid member than performance on “objective” metrics.

Thus selective methods rely on structural and corrective methods.

And all of these patterns surely manifest in more complex, “real-world”, scenarios.

Added to my Anki. This is very clear framework to think about some problems. Thanks!

"selective, corrective, and structural" are kinda like:

• Selective: Interventions that occur before the critical event. t < T_c
• Corrective: Interventions that occur during the critical event. t ~= T_c
• Structural: Interventions that occur after the critical event. t > T_c