The Real Rules Have No Exceptions

post by Said Achmiz (SaidAchmiz) · 2019-07-23T03:38:45.992Z · score: 101 (52 votes) · LW · GW · 42 comments

(This is a comment [LW · GW] that has been turned into a post.)

From Chris_Leong’s post, “Making Exceptions to General Rules [LW · GW]”:

Suppose you make a general rule, ie. “I won’t eat any cookies”. Then you encounter a situation that legitimately feels exceptional , “These are generally considered the best cookies in the entire state”. This tends to make people torn between two threads of reasoning:

  1. Clearly the optimal strategy is to make an exception this one time and then follow the rule the rest of the time.

  2. If you break the rule this one time, then you risk dismantling the rule and ending up not following it at all.

How can we resolve this? …

This is my answer:

Consider even a single exception to totally undermine any rule. Consequently, only follow rules with no exceptions.[1]. When you do encounter a legitimate exception to a heretofore-exceptionless rule, immediately discard the rule and replace it with a new rule—one which accounts for situations like this one, which, to the old rule, had to be exceptions.

This, of course, requires a meta-rule (or, if you like, a meta-habit):

Prefer simplicity in your rules. Be vigilant that your rules do not grow too complex; make sure you are not relaxing the legitimacy criteria of your exceptions. Periodically audit your rules, inspecting them for complexity; try to formulate simpler versions of complex rules.

So, when you encounter an exception, you neither break the rule once but keep following it thereafter, nor break it once and risk breaking it again. If this is really an exception, then that rule is immediately and automatically nullified, because good rules ought not have exceptions. Time for a new rule.

And if you’re not prepared to discard the rule and formulate a new one, well, then the exception must not be all that compelling; in which case, of course, keep following the existing rule, now and henceforth.

But why do I say that good rules ought not have exceptions? Because rules already don’t have exceptions.

Exceptions are a fiction. They’re a way for us to avoid admitting (sometimes to ourselves, sometimes to others) that the rule as stated, together with the criteria for deciding whether something is a “legitimate” exception, is the actual rule.

The approach I describe above merely consists of making this fact explicit.


  1. By which I mean “only follow rules to which no legitimate exception will ever be encountered”, not “continue following a rule even if you encounter what seems like a legitimate exception”. ↩︎

42 comments

Comments sorted by top scores.

comment by Said Achmiz (SaidAchmiz) · 2019-07-23T05:18:54.696Z · score: 47 (17 votes) · LW · GW

Playing the devil’s advocate:

Consider this tumblr post by nostalgebraist, the contents of which I entirely concur with and endorse. It would seem to contradict, or at least undermine the applicability of, the approach I describe in this post.

More generally, the contradiction arises because, while this is entirely true—

the rule as stated, together with the criteria for deciding whether something is a “legitimate” exception, is the actual rule.

—the difficulty is that in some cases, the stated rule may be straightforward and legible, but the criteria for evaluating the legitimacy of exceptions is complex and illegible (and, in many or even most such cases, attempting to make the criteria legible will inevitably result in discarding important information).

Thus, e.g., in the sort of scenario described by nostalgebraist, the “actual rule” is “these are the explicit rules, but I also reserve the right to apply my own, fundamentally irreducible[1] judgment to make exceptions, and I admit of no formal/explicit rule which stands above that right”. In this case, spelling out the “actual rule” seems to have gained us very little.

Yet I think that the approach I describe withstands this challenge—because it remains the best approach, despite not being perfect; all the other solutions to the question (of what to do about apparently-compelling exceptions to apparently-reasonable rules) do no better, in such cases.

And while we gain little by spelling out the “actual rule” in these “complex and/or illegible exception-judging criteria” situations, nevertheless we do gain something—namely, making explicit (and therefore salient) the fact that unexpected exceptions (driven by irreducible judgment) are a possibility. What is explicit, can be better prepared for, and can be discussed, and problems addressed; so this is a benefit, if not a very great one.


  1. Why do I say “fundamentally irreducible”? Suppose that you offer some operationalization of my judgment criteria—one which appears to account for all of the judgments I’ve made, to instantiate any principles that seem to stand behind my judgment criteria, not to to leave unaddressed any cases I can imagine, etc. You may be tempted to call this a successful reduction—to identify my judgment with your reduction of it. Yet recall that, by construction, I have retained the right to “call bullshit” on any application of an explicit rule which I feel goes against the rule’s spirit; which means that I remain free to, e.g., reject the output of your operationalization of my judgment criteria, in any future case, no matter how closely that output has matched my judgment thus far. Since this applies to any operationalization you can construct—which must, by definition, be explicit—the “personal judgment” rule is a meta-rule of a higher order than any explicit rule, and operationalizing it is impossible. ↩︎

comment by quanticle · 2019-07-23T14:18:23.656Z · score: 30 (9 votes) · LW · GW

Paul Scharre, in his excellent book about the application of AI to military technology, Army of None, has an anecdote which I think is relevant. In the book, he talks about leading a patrol up an Afghan hillside. As he and the troops under his command ascend the hillside, they're spotted by a farmer. Realizing that they've been spotted, the patrol hunkers down and awaits the inevitable attack by Afghan insurgent forces. However, before the attackers arrive, something unexpected happens. A little girl, about 5 or 6 years of age, comes up to the position, with some goats and a radio. She reports the details of the Americans' deployment to the attacking insurgents and departs. Shortly thereafter, the insurgent attack begins in earnest, and results in the Americans being driven off the hillside.

After the failed patrol, Scharre's troop held an after-action briefing where they discussed what they might have done differently. Among the things they discussed was potentially detaining the little girl, or at least relieving her of her radio so as to limit the information being passed back to the attackers. However, at no point, did anyone suggest the alternative of shooting the girl, even though they would have been perfectly justified, under the laws of war and rules of engagement, in doing so. Under the laws of war, anyone who acts like a soldier is a soldier, and this includes 5-year-old-girls conducting reconnaissance for insurgents. However, everyone understood, on a visceral level, that there was a difference between permissible and correct and that the choice of shooting the girl, while permissible, was morally abhorrent to the point where it was discarded at an unconscious level.

That said, no one in the troop also said, "Okay, well, we need to amend our rules of engagement to say, 'Shooting at people conducting reconnaissance is permissible... except when the person is a cute little 5-year-old-girl.'" Everyone recognized, again, at an unconscious level, that there was value to having a legible rule "Shooting at people behaving in a soldierly manner is acceptable," with illegible exceptions ("Except when that person is a 5-year-old girl leading goats"). The drafters of rules cannot anticipate every circumstance in which the rule might be applied, and thus having some leeway about the specific obligations (while making the intent of the rule clear) is valuable insofar as it allows people to take actions without being paralyzed by doubt. This applies as much to rules governing as an organization as it does to rules that you make for yourself.

The application to AI is, I hope, obvious. (Unfriendly) AIs don't make a distinction between permissible and correct. Anything that is permissible is an option that can be taken, if it furthers the AI's objective. Given that, I would summarize your point about having illegible exceptions as, "You are not an unfriendly AI. Don't act like one."

comment by mr-hire · 2019-07-24T00:11:39.325Z · score: 2 (1 votes) · LW · GW
'Shooting at people conducting reconnaissance is permissible... except when the person is a cute little 5-year-old-girl.'

At least in the old war movies I've seen, that used to have the general "except for women and children" clause.

comment by quanticle · 2019-07-24T02:42:01.179Z · score: 6 (3 votes) · LW · GW

That's something you see in movies, yes, but as I understand what Paul Scharre is saying, it's not something that's actually true. According to him, the laws of war "care about what you do, not who you are." If you are behaving in a soldierly fashion, you are a soldier, whether you are a young man, old man, woman, or child.

comment by ryan_b · 2019-07-24T18:05:37.879Z · score: 23 (6 votes) · LW · GW

I affirm Scharre's interpretation.

Anecdote: during deployment when we arrive in country, we are given briefings about the latest tactics being employed in the area where we will be operating. When I went to Iraq in 2008 one of these briefings was about young girls wearing suicide vests, which was previously unprecedented.

The tactic consisted of taking a family hostage, and telling the girl that if she did not wear this vest and go to X place at Y time, her family would be killed. Then they would detonate the vest by remote.

We copped to it because sometimes we had jammers on which prevented the detonation, and one of the girls told us what happened. Of course, we didn't have jammers everywhere. Then the calculus changes from whether we can take the hit in order to spare the child, to one child or many (suicide bombings target crowds).

The obvious wrongness of killing children does not change; nor that of allowing children to die. So one guy eats the sin, and the others feel ashamed for letting him.

comment by jmh · 2019-07-24T12:18:31.086Z · score: 1 (1 votes) · LW · GW

On a more depressing note one might look into the events in the Korean War "except for women and children" was not applied. The movie is called A Little Pond (it was available on Amazon Prime a year or so back not sure if it's there now though) about the events at Nogunri.

Now, the movie also depicts the more human side of a soldier when confronted directly with that act -- rather than the impersonal shapes from hundreds of meters away -- near the end of the movie.

I would also add, regarding the whole permissible versus exception, that I suspect it is even grayer than suggested. The 5 year old with a radio is hardly and less a part of the fighting force than the civilians providing all the logistics and production supporting any of the military actions. So where is that line?

I'm not sure the AI will do much worse or much better than those making the plans and issuing the orders far from the battle ground and not exposed to the bloodshed and human carnage.

comment by orthonormal · 2019-07-23T22:17:16.796Z · score: 19 (12 votes) · LW · GW

Meta: I approve of the practice of arguing against your own post in a comment.

comment by mr-hire · 2019-07-23T18:02:11.191Z · score: 5 (3 votes) · LW · GW

See also You Don't Get To Know What You're Fighting For, which makes this sort of situation more explicit.

comment by Said Achmiz (SaidAchmiz) · 2019-07-23T18:38:49.440Z · score: 3 (2 votes) · LW · GW

Indeed. In particular I want to note Nate Soares’ point about how one of the reasons you don’t necessarily know what you’re fighting for, is that your goal(s) may change as you learn more, grow, etc. Similarly, illegible complex judgment criteria may shift over time (and for that reason will not be amenable to formalization, which is of necessity static), while still always being “my own judgment”; it is precisely that freedom to alter the criteria which I protect by resisting any proffered formalization.

comment by Gurkenglas · 2019-07-23T12:40:28.254Z · score: 1 (1 votes) · LW · GW

re your footnote: The explicit version of your judgement allows you an override, yet by construction you will never take it. So the crux behind whether the versions are semantically the same is whether we define rules to allow or disallow actions, or timelines.

comment by jbay · 2019-07-24T19:16:26.873Z · score: 25 (9 votes) · LW · GW

In spirit I agree with "the real rules have no exceptions". I believe this applies to physics just as well as it applies to decision-making.

But, while the foundational rules of physics are simple and legible, the physics of many particles -- which are needed for managing real-world situations -- includes emergent behaviours like fluid drag and turbulence. The notoriously complex behaviour of fluids can be usefully compressed into rules that are simple enough to remember and apply, such as inviscid or incompressible flow approximations, or tables of drag coefficients. But these simple rules are built on top of massively complex ones like the Navier-Stokes equation (which is itself still a simplifying assumption over quantum physics and relativity).

It is useful to remember that the equations of incompressible flow are not foundational and so will have exceptions, or else you will overconfidently predict that nobody can fly supersonic airplanes. But that doesn't mean you should discard those simplified rules when you reach an exception and proceed to always use Navier-Stokes, because the real rules might simply be too hard to apply the rest of the time and give the same answer anyway, to three significant figures. It might just be easier in practice to remember the exceptions.

Hence, when making predictive models, even astrophysicists will think of gravity in terms of "stars move according to Newton's inverse square law, except when dealing with black holes or gravitational lensing". They know that it's really relativity under the hood, but only draw on that when they know it's necessary.

OK, that's enough of an analogy. When might this happen in real life?

One case could be multi-agent, anti-inductive systems... like managing a company. As soon as anyone identifies a complete and compact formula for running a successful business it either goes horrifyingly [LW · GW] wrong [LW · GW], or the competitive landscape adapts to nullify it, or else it was too vague of a rule to allow synthesizing concrete actions. ("Successful businesses will aim to turn a profit").

comment by Said Achmiz (SaidAchmiz) · 2019-07-24T19:25:29.911Z · score: 5 (3 votes) · LW · GW

This is a very good point, thank you. I have some tentative thoughts in response, but I will have to think about it carefully.

Here’s a question in the meantime: do you think that what you say is addressed in / is essentially the same as what I write in this comment elsethread [LW · GW]? Or is this something else entirely?

comment by jbay · 2019-07-24T21:44:47.318Z · score: 1 (1 votes) · LW · GW

Thanks!

I think my point is different, although I have to admit I don't entirely grasp your objection to Nostalgebraist's objection. I think Nostalgebraist's point about rules being gameable does overlap with my example of multi-agent systems, because clear-but-only-approximately-correct rules are exploitable. But I don't think my argument is about it being hard to identify legitimate exceptions. In fact, astrophysicists would have no difficulty identifying when it's the right time to stop using Newtonian gravity.

But my point with the physics analogy is that sometimes, even if you actually know the correct rule, and even if that rule is simple (Navier-Stokes is still just one equation), you still might accomplish a lot more by using approximations and just remembering when they start to break down.

That's because Occam's-razor-simple rules like "to build a successful business, just turn a huge profit!" or "air is perfectly described by this one-line equation!" can be very hard to apply to synthesize into specific new business plans or airplane designs, or even to make predictions about existing business plans or airplane designs.

I guess a better example is: the various flavours of utilitarianism each convert complex moral judgements into simple, universal rules to maximize various measures of utility. But even with a firm belief in utilitarianism, you could still be stumped about the right action in any particular dilemma, just because it might be really hard to calculate the utility of each option. In this case, you don't feel like you've reached an "exception" to utilitarianism at all -- you still believe in the underlying principle -- but you might find it easier to make decisions using an approximation like "try not to kill anybody", until you reach edge-cases where that might break down, like in a war zone.

You might not even know if eating a cookie will increase or decrease your utility, so you stick to an approximation like "I'm on a diet" to simplify your decision-making process until you reach an exception like "this is a really delicious-looking / unusually healthy cookie", in which you decide it's worth dropping the approximation and reaching for the deeper rules of utilitarianism to make your choice.

comment by Said Achmiz (SaidAchmiz) · 2019-07-24T23:15:34.234Z · score: 2 (1 votes) · LW · GW

I think my point is different, although I have to admit I don’t entirely grasp your objection to Nostalgebraist’s objection.

Oh, I don’t object to what nostalgebraist says! I think it’s entirely right. (Also, to be clear, his post was written some time before my comment, so it’s not in any way a response to the latter.)

I say only that despite what he says seemingly being a serious challenge to (or even contradiction of) my post, nonetheless the post’s thesis survives the challenge intact, if not unscathed—mostly because no alternative approach to mine deals with the challenge any better.

I guess a better example is: the various flavours of utilitarianism …

Actually… I think this is a much worse example—because, in fact, I think such difficulties are entirely fatal to utilitarianism! (In fact I think that utilitarianism’s inadequacy as a moral theory is overdetermined—that is, that there are several reasons to reject it, each one sufficient on its own—but the sorts of problems you mention are certainly among those reasons.)

But let me return to your original examples—physics and business. Having thought about the matter a bit, it now seems to me that the position you are arguing against, which you (by implication) ascribe to me, is somewhat of a strawman.

The sort of situation I am referring to is one where you have (a) a rule that is applicable to a given class of situations, and (b) some phenomenon by which exceptions to the rule [i.e., specific situations where you don’t follow the rule, but instead do something else] arise. The claim I am making at the end of the post is that (b) is not some unfathomable black box from which, unexpectedly and unpredictably, exceptional cases spring, but rather a comprehensible set of criteria; and that (a) and (b) together constitute the actual “rule”—which, by construction, lacks exceptions. (And then there is the additional claim that there’s a benefit to making all of this explicit, and basing your decisions on it; this is the primary subject of the post.)

Now, it seems to me (and please correct me if I’m wrong here) that you are misreading me in two ways.

Firstly, it seems as if you are reading me as saying that (a) and (b) actually should be, or are, not two separate things but actually just one thing (and perhaps even that this one thing is, or should be, a simple thing). But I’m not saying anything of the sort! For instance, you say:

In fact, astrophysicists would have no difficulty identifying when it’s the right time to stop using Newtonian gravity.

Well and good! This is entirely consistent with my point. Here the “actual rule” would be something like: “relativity, plus whatever criteria we use to determine when to use Newtonian physics instead”. Clearly, this rule has no exceptions! (And if it does, well, whence those exceptions? How do physicists decide those are exceptions? However they did, whatever criteria they used—into the rule they go…)

Secondly, the situations I am referring to are, as I said, those where you have a rule that’s applicable to a given class of situations. By this I mean that you have some rule that tells you precisely what to do, but sometimes instead of doing that thing, you do (or, at least, are tempted to do) a different thing (i.e., you sometimes encounter [potentially] exceptional cases).

For example, if you have the rule “don’t eat cookies”, and you encounter a cookie, your rule is very clear on what you are to do: don’t eat the cookie. There’s no ambiguity here, no confusion or uncertainty. Should you eat this cookie? The rule says: no. You should not eat the cookie. End of story. That you are sometimes tempted to ignore, a.k.a. break, the rule, does not change the fact that the rule unambiguously dictates your actions. (The question, then, is why you’re tempted to make the exception, and exactly in what sorts of cases, etc.)

But note that this is not the case in your examples! If the rule, supposedly, is “use the Navier-Stokes equation”, but that equation is, in practice, impossible to calculate, then the rule doesn’t actually dictate your actions! It’s not that you know exactly what the answer is but you are unwilling to accept it; you just don’t have the answer! The supposed “rule” isn’t really any such thing. And in business it’s even worse: yes, “just turn a huge profit”, but what actually do I do? Specifically? I don’t know! I’m not tempted to break the rule, not at all; actually, I’d love to follow it, if only I knew how… but I don’t have any idea how! So, I have to use something other than this purported “rule”, in order to decide what to do.

comment by ryan_b · 2019-07-23T15:51:37.440Z · score: 11 (6 votes) · LW · GW
And if you’re not prepared to discard the rule and formulate a new one, well, then the exception must not be all that compelling

This feels similar to Beware Trivial Inconveniences [LW · GW] and The Amish, and Strategic Norms Around Technology [LW · GW] posts; the inconvenience of having to update the rule by itself serves as disincentive to violate the rule.

comment by Raemon · 2019-09-04T20:49:34.191Z · score: 8 (4 votes) · LW · GW

Curated.

I've found this post useful for crystallizing my own thinking, both about rules I follow (as a human taking actions), and even a bit helpful for grokking the overall Law vs Toolbox [LW · GW] distinction.

Looking over the comments, I see some people seem to have found it less crystalizing than I. I have a sense that there's a version of this post that could have bridged some inferential gulf better. But also, there's not necessarily such a thing as a universally good explanation

I have some sense that the post could be improved if it was given a second draft whose goals was specifically to find someone who didn't grok the first version of the post, and exploring various different explanations until it clicks.

But, also there's no such thing as a universally compelling explanation and maybe this is just a case where it was useful to add one more road to Rome [LW · GW] that was helpful for at least some people.

comment by Dagon · 2019-07-23T16:57:36.335Z · score: 4 (2 votes) · LW · GW

One point in favor of biasing toward non-exceptions (I still won't say "none at all") is that some parts of me are adversarial with the parts who are identifying rules. They can be very persuasive that this is a time for an exception, so it makes sense to have a pretty high bar (mostly: it contradicts another rule in the same ring) for making exceptions.

comment by Dacyn · 2019-09-13T13:46:34.119Z · score: 1 (1 votes) · LW · GW
the rule as stated, together with the criteria for deciding whether something is a “legitimate” exception, is the actual rule.
The approach I describe above merely consists of making this fact explicit.

This would be true were it not for your meta-rule. But the criteria for deciding whether something is a legitimate exception may be hazy and intuitive, and not prone to being stated in a simple form. This doesn't mean that the criteria are bad though.

For example, I wouldn't dream of formulating a rule about cookies that covered the case "you can eat them if they're the best in the state", but I also wouldn't say that just because someone is trying to avoid eating cookies means they can't eat the best-in-state cookies. It's a judgement call. If you expect your judgement to be impaired enough that following rigid explicitly stated rules will be better than making judgement calls, then OK, but it is far from obvious that this is true for most people.

comment by Said Achmiz (SaidAchmiz) · 2019-09-13T15:21:04.614Z · score: 2 (1 votes) · LW · GW

For example, I wouldn’t dream of formulating a rule about cookies that covered the case “you can eat them if they’re the best in the state”

Why?

This seems like a case that is entirely amenable to formalization (and without any great difficulty, either).

If you expect your judgement to be impaired enough that following rigid explicitly stated rules will be better than making judgement calls

“Judgment calls” are not irreducible.

One of the great insights that comes from the informal canon of best practices for GMing TTRPGs (e.g.) is that “rules” and “judgment calls” need not be contrasted with each other; on the contrary, the former can, and often does, assist and improve the latter. In other words, it’s not that following explicitly stated rules is better than making judgment calls, but rather that following explicitly stated rules is how you do better at making judgment calls.

comment by Donald Hobson (donald-hobson) · 2019-07-26T11:35:51.558Z · score: 1 (3 votes) · LW · GW

For deciding your own decisions, only a full description of your own utility function and decision theory will tell you what to do in every situation. And (work out what you would do if you were maximally smart, then do that) is a useless rule in practice. When deciding your own actions, you don't need to use rules at all.

If you are in any kind of organization that has rules, you have to use your own decision theory to work out which decision is best. To do this would involve weighing up the pros and cons of rule breaking, with one of the cons being any punishment the rule enforcers might apply.

Suppose you are in charge, you get to write the rules and no one else can do anything about rules they don't like.

You are still optimizing for more than just being correct. You want rules that are reasonably enforceable, the decision of whether or not to punish can only depend on things the enforcers know. You also want the rules to be short enough and simple enough for the rule followers to comprehend.

The best your rules can hope to do when faced with a sufficiently weird situation is not apply any restrictions at all.

comment by Said Achmiz (SaidAchmiz) · 2019-07-26T15:05:52.472Z · score: 2 (1 votes) · LW · GW

When deciding your own actions, you don’t need to use rules at all.

Even a rudimentary level of knowledge of how people behave is enough to know that this is entirely false. Act consequentialism doesn’t work for human psychology.

Suppose you are in charge, you get to write the rules and no one else can do anything about rules they don’t like.

This, too, bears no resemblance to reality. People can do all sorts of things about rules they don’t like.

comment by Wei_Dai · 2019-07-26T16:00:14.457Z · score: 12 (5 votes) · LW · GW

Act consequentialism doesn’t work for human psychology.

In what sense does it "not work"? I feel like I use act consequentialism all the time, for example when deciding what restaurant to go to for dinner (which can't be made into a rule since it depends on so many variables like where I am located on a particular day, what the weather is like, and what foods I've eaten recently), or to decide whether to say X to person Y or hold my tongue (which similarly depends on many variables). I may not be doing expected utility computations in a conscious or explicit way (at least not in most cases), but I'm guessing the neural networks implementing my intuitions have been trained to do something like that. (ETA: Because the choices I make usually respond to changing circumstances in a way that seems consistent with doing something like EU maximization.) Do you have some reason to think otherwise?

comment by Said Achmiz (SaidAchmiz) · 2019-07-26T21:13:33.565Z · score: 5 (2 votes) · LW · GW

I feel like I use act consequentialism all the time, for example when deciding what restaurant to go to for dinner …

Really? When deciding what restaurant to go to for dinner, you examine all possible consequences of all the choices at your disposal (and the probability distributions across them), evaluate or rank them all, and select one? You don’t use any rules at all?

… which can’t be made into a rule since it depends on so many variables like where I am located on a particular day, what the weather is like, and what foods I’ve eaten recently

Why do you think that there needs to be a rule, instead of, say, multiple rules (some combination of which may bear on any given situation)? And why can’t rules depend on variables? (Or contain heuristics, etc.?)

… I’m guessing the neural networks implementing my intuitions have been trained to do something like [expected utility computations]

This seems stupendously unlikely. My reason for thinking otherwise is that this just isn’t consistent with anything we know about how people make decisions.

Because the choices I make usually respond to changing circumstances in a way that seems consistent with doing something like EU maximization.

Are you just saying that the preferences revealed in your choices conform to the VNM axioms (or some other formalism—if so, which?)? (If you are, then you know that this implies nothing at all about whether your brain is actually doing any expected utility computations.) Or are you making some stronger claim? If so, what is it?

comment by Wei_Dai · 2019-07-27T01:37:27.441Z · score: 5 (2 votes) · LW · GW

Really? When deciding what restaurant to go to for dinner, you examine all possible consequences of all the choices at your disposal (and the probability distributions across them), evaluate or rank them all, and select one? You don’t use any rules at all?

No, I guess I examine some subset of consequences that seem relevant to each decision (i.e., might differ across my choices in a predictable way, and the differences make a difference for my values). I can't confidently say that I don't use any rules at all (maybe I'm using some rules in some subconscious way, or I'm doing something that counts as "using a rule") but neither can I say what those rules are.

Why do you think that there needs to be a rule, instead of, say, multiple rules (some combination of which may bear on any given situation)? And why can’t rules depend on variables? (Or contain heuristics, etc.?)

I wasn't intending to make a point about single vs multiple rules (but since you ask, having multiple rules seems to require some meta-rule to tell you which rules to use in which circumstances and how to adjudicate conflict between them, so that meta-rule would be "the rule"). My point was more that I don't see what rule(s) I could be using that would seemingly take into account so many variables in such a fluid and dynamic way, and can seemingly handle new unforeseen circumstances/variables without me having to think "how should I change my rules to handle this?"

This seems stupendously unlikely. My reason for thinking otherwise is that this just isn’t consistent with anything we know about how people make decisions.

Can you list some such inconsistencies, so I can have a better idea of what you mean?

Are you just saying that the preferences revealed in your choices conform to the VNM axioms (or some other formalism—if so, which?)?

No, I mean things like when one of my choices would predictably cause some bad consequences (and doesn't cause enough good consequences to compensate) I seem to fairly reliably avoid making that choice, even when there's enough novelty involved that it seems unlikely I would have created a rule to cover the situation ahead of time, and without having to think "how should I change my rules to handle this?"

comment by Dagon · 2019-07-26T22:26:31.648Z · score: 4 (2 votes) · LW · GW

You may need to taboo "rule" to get much further on this. I can't speak for Wei, but I use plenty of heuristics, cached ideas, and non-legible estimates of effect in choosing a dinner location. None of these are "rules" in the sense I get from this post, and I don't abandon nor reformulate them when I choose a different food than previously.

comment by Said Achmiz (SaidAchmiz) · 2019-07-26T22:33:46.264Z · score: 2 (1 votes) · LW · GW

To clarify, “rule” as used in the grandparent and “rule” as used in the OP are different concepts. (Namely, in the grandparent I was referring—following what I took to be Wei Day’s usage—to rule consequentialism.)

comment by Dagon · 2019-07-28T00:03:24.253Z · score: 2 (1 votes) · LW · GW

Can you delineate a bit of the difference between these uses of "rule", and how rule consequentialism avoids any of the problems that the post (and comments/objections to it) talks about?

comment by Said Achmiz (SaidAchmiz) · 2019-07-28T00:30:21.461Z · score: 2 (1 votes) · LW · GW

… how rule consequentialism avoids any of the problems that the post (and comments/objections to it) talks about?

It… doesn’t? Who said that it does? I’m not even sure what that would mean; it seems like an almost entirely orthogonal issue…

(As for the uses of “rule”, that’s a fine question but I hesitate to write any lengthy commentary on it, because it seems like we have some sort of weird misunderstanding, and it may not even be relevant…)

comment by jmh · 2019-07-24T12:02:09.747Z · score: 1 (1 votes) · LW · GW

At one level I can find agreement with your position. On another I find it difficult.

Well defined rules should clearly apply (much like a function with a clean mapping from domain to range) to know settings. However, that then begs the question of how do we know we have clearly defined or perceived all the possible setting the rule might seem to apply.

Is there a presumption of perfect knowledge when making the rule?

[Edit to add: Has anyone here read the article "Origins of Predictable Behavior" -- AER 1984 I think. Ron Heiner. If not I think it may offer some additional insights to this discussion. Over 20 years since I read it so don't even want to try summarizing impressionistic memory I have.]

comment by Said Achmiz (SaidAchmiz) · 2019-07-24T14:05:23.009Z · score: 5 (2 votes) · LW · GW

Is there a presumption of perfect knowledge when making the rule?

No. Of course not.

If you learn something new, or encounter some new situation, that makes some existing rule no longer make sense—you discard that rule, and make a new rule. This is no different from encountering a “legitimate exception”.

… how do we know we have clearly defined or perceived all the possible setting the rule might seem to apply

We don’t. We try our best, but we have no guarantees. That is life.

This is, in fact, the point of all that stuff about discarding and re-formulating rules when you encounter “exceptions”, periodically auditing your rules, etc. It is a way—indeed, the only sane way—of dealing with imperfect knowledge, and the inevitability of surprises.

comment by jmh · 2019-07-24T18:32:05.049Z · score: -1 (2 votes) · LW · GW
If you learn something new, or encounter some new situation, that makes some existing rule no longer make sense—you discard that rule, and make a new rule. This is no different from encountering a “legitimate exception”.

I think walking through that door puts us right back with the exception to every rule -- which has always implied the exception was in fact legitimate.

Exceptions are a fiction. They’re a way for us to avoid admitting (sometimes to ourselves, sometimes to others) that the rule as stated, together with the criteria for deciding whether something is a “legitimate” exception, is the actual rule.

Requires that we already had the criteria for deciding if the as yet unknown exception arising from the know information was in fact well formed and able to deal with the unknown information.

If one wants to argue that rules are inherently context/informationlly bound and within those bounds we can define where the rule applies and where it does not I agree. But that seems a lot different than saying we can update rules as situations arise and some how that allows us to escape the trap or temptation of claiming "exception" to escape holding ourselves to the rule.

comment by Said Achmiz (SaidAchmiz) · 2019-07-24T19:19:41.121Z · score: 2 (1 votes) · LW · GW

Requires that we already had the criteria for deciding if the as yet unknown exception arising from the know information was in fact well formed and able to deal with the unknown information.

No. It does not. Nothing that I wrote requires this.

comment by Jimdrix_Hendri · 2019-07-24T02:10:40.656Z · score: 1 (1 votes) · LW · GW

A good rule is an objective procedure that can be apply to derive a response to any foreseeable situation.

A look-up table is not a rule, for the same reason that a detailed table of planetary ephemerides is not a substitute for the law of gravity.

Nostalgebrist's suggestion cannot be considered a rule at all. It is not objective.

In the realm of psychology and politics, rules gain legitimacy when they are adhered to over a long period of time and when they are seen to consistently protect against bad outcomes.

There is a case for flexible interpretation, but an agent who abandons rules too frequently, and with slight incentive will eventually lose confidence in his ability to abide by rules. This was only hinted at in the original post, but it is a point worth making explicit.


comment by Said Achmiz (SaidAchmiz) · 2019-07-24T04:33:12.642Z · score: 4 (2 votes) · LW · GW

What is an “objective procedure”? How does it differ from a procedure which is not “objective”?

comment by Jimdrix_Hendri · 2019-07-24T21:32:04.566Z · score: 1 (1 votes) · LW · GW

One criterion for a procedure to be objective is that it can be carried out equally by anyone.

A procedure which includes a codicil: "Sometimes, I will step in and overturn the arrangement". Fails on three counts:

1 It fails to explicitly define the criteria for making interventions.

2 Nothing is said about the range of interventions that will be entertained.

3 It does not specify the means by which the type of intervention will be determined.

The name for this is dictat, and is almost always inappropriate and dangerous.

There are other ways of building in flexibility. For instance:

In cases many where the environment (causes) is very unpredictable, it is still possible to establish guidelines with reference to effects.

At the same time, the "rule" can explicitly state criteria for turning off the intervention, thereby reducing the risk that the intervention become a new normal.

Types of interventions can also be limited to a pre-existing list of alternatives, which can be criticised and vetted before the emergency is triggered.

comment by Said Achmiz (SaidAchmiz) · 2019-07-24T23:47:50.323Z · score: 2 (1 votes) · LW · GW

There are other ways of building in flexibility. For instance:

In cases many where the environment (causes) is very unpredictable, it is still possible to establish guidelines with reference to effects.

At the same time, the “rule” can explicitly state criteria for turning off the intervention, thereby reducing the risk that the intervention become a new normal.

Types of interventions can also be limited to a pre-existing list of alternatives, which can be criticised and vetted before the emergency is triggered.

Yes, all of these things can be done. And if you do any or all of them, and do not have a provision for applying judgment that stands above any of these provisions, your system will be exploitable. This is nostalgebraist’s point.

In practice, there will always exist “no, actually, we’re not following that rule in this case” exceptions. Our options are as follows:

  1. Pretend strenuously that no exceptions exist. Apply the rules as inflexibly as possible, even (in fact, especially) in cases where it really seems like we shouldn’t apply them, to maintain the illusion that no exceptions exist.

  2. Admit the fact that exceptions exist; attempt to make explicit, clarify, and rationalize the criteria for exception-making (in other words, fold them into the explicit rules); nevertheless maintain the option to exercise judgment, in contravention to the rules.

Crucially, in case #1, there will still be exceptions. They will be “snuck in” via creative interpretation of the rules, or via expansion or alteration of the rule set with rules that are bad rules (and exist only to allow a certain class of cases which the rules would otherwise forbid), or via manipulations outside the rule system that make the rule-based decision itself irrelevant, or in any number of other ways. Look to our criminal justice system for examples, and our political system also; and any number of other systems of rules.

Note three things:

First, that nostalgebraist’s advice does not in any way prevent you from formulating guidelines, and attempting to minimize and to circumscribe the scope of your “extralegal” judgments. In fact, you should do this—especially if you find yourself making such “extralegal” exceptions often! You should seek a pattern in them, and see if they (or at least some of them) can be made into a rule; or, perhaps, if the existing rules need to be altered.

Second, the “fail-safe” clause is not meant to be used only by one “side” or party in some relationship—quite the opposite! Consider what nostalgebraist says:

Of course, this option can itself be abused, particularly if it is used freely and unthinkingly. If the rules hold only until the moment you don’t like their consequences, then the rules don’t really hold at all.

But if someone does that, you can take the same option yourself: “yeah, I said you should have this option, but using it like that is wrong.” Because the option does not have specific rules (by construction), no one can use rules lawyering to take it away from you. If someone uses “calling bullshit” for bad ends, you can just call bullshit on them.

If someone is using “sometimes I will just use my judgment” to be a capricious, unpredictable dictator, then you say: “you are acting improperly”. By construction, it will do the would-be tyrant no good to say “ah, but we agreed that I can use my judgment whenever I see fit, so your objection is invalid”!

Third, you say:

In the realm of psychology and politics, rules gain legitimacy when they are adhered to over a long period of time and when they are seen to consistently protect against bad outcomes.

There is a case for flexible interpretation, but an agent who abandons rules too frequently, and with slight incentive will eventually lose confidence in his ability to abide by rules.

Here you are speaking first about legitimacy—how rules meant to bind others are perceived by the public—and then, apparently, about self-control and will. These are not the same thing, and it does no good to discuss them both in the same breath.

comment by Dagon · 2019-07-23T04:59:49.344Z · score: 1 (4 votes) · LW · GW

Alternate approach: recognize that rules (as opposed to physical laws) are always and only guidelines, or defaults, or lossy summaries of one's intent. There's no such thing as a complete and consistent ruleset, and even if we could get close, it wouldn't fit in our brains.

Rules are like models: none are true (none are fully binding or complete descriptions of desired behavior), many are useful (in that they can give good defaults and heuristics for common cases, where deeper computation is undesirable or infeasible).

There are no real rules. Exceptions may be fiction, but that's because rules are fiction in the first place.
Rules don't exist in the territory, they're just fuzzy areas on maps.


comment by Raemon · 2019-07-23T05:06:54.679Z · score: 12 (3 votes) · LW · GW

Something something Law Thinking vs Toolbox Thinking [LW · GW]. My read on Said's post here is to help people think about Lawful thinking, noticing that there *is* some actual optimal rule you can follow (even if it's computationally intractable and you don't know what the rule is)

comment by Dagon · 2019-07-23T14:38:19.291Z · score: 2 (1 votes) · LW · GW

I don't think I can pass an ITT for strict lawful thinking. I'm absolutely supportive of discovering and creating summaries of future decision intent, and of being somewhat rigorous in doing so. But I can't ignore the fundamental complexity of the real world, and the fact that these are ONLY extremely compressed expressions of a set of beliefs.

I may be stuck in toolbox thinking, though I'll definitely use lawful models as some of my tools. Or I may simply not be smart enough to identify and make legible the incredible variety of decisions I face over time. Rules (and habits, which are basically unconscious rules) make this tolerable, as I can spend very little energy on most of them. But there are daily choices where I see conflicts among rules and have to choose among rules that might apply, and also among the meta-rules to pick the right rule, and meta-meta-rules to weigh across different meta-rules, etc.

I kind of wonder if we have actual different felt experiences on the topic. I can only think of stated rules as porous and directional, and I feel good when I violate one for a good purpose. Take that, over-simplistic, condescending non-agent worldview! I also feel good when I recognize a new context in which a rule applies and find that the rule is stronger than I previously thought, so I'm not anti-rule in general, just that I think they're a convenience rather than a truth.

I've talked with other people who are horrified when they find a case that an accepted rule interferes with doing the best thing, and work hard to reconcile the situation with patches or meta-rules (and get angry when I use the word "rationalization"). They seem to feel near-physical pain from violating (some) rules without a lot of justification. I have sometimes been guilty of thinking they just need to find the right Manic Pixie Dream Person to break them out of the bonds of propriety, but I also wonder if there's something deeper in the way the world actually feels day-to-day to them and to me.

comment by Said Achmiz (SaidAchmiz) · 2019-07-23T05:20:22.754Z · score: 2 (1 votes) · LW · GW

The comment I wrote just now [LW · GW] is relevant.

comment by kithpendragon · 2019-07-23T10:37:11.304Z · score: 3 (2 votes) · LW · GW

I like to formulate this as "I intend to be the kind of person who mostly X" or "I plan to X on a [TIME]ish basis". Using these formulations removes the friction and stress I experience from "I must X every day" or "I must never X". I've found this makes habits easier to assimilate since they are intentions and not hard rules.