Decision theory: Why we need to reduce “could”, “would”, “should”

Calculating what would happen if an agent took some action is not counter-factual from the POV of the agent. The agent doesn't know what action it is going to take. If it did, it would just take the action - not spend time calculating the consequences of its various possible actions.

In my experience, beginning math students simply expect their nice-sounding procedures to work. For example, they expect to be able to add fractions straight across. When you tell them they can’t, they demand to know why they can’t, as though most nice-sounding theorems are true, and if you want to claim that one isn’t, the burden of proof is on you.

I've noticed that, on the few occasions I've taught genuine beginning-beginners. Never thought of it in that many words, but yeah. And the analogy to beginning rationality students, including 50-year-old scientists making up their own religions, is obvious and important. People starting out in math are operating in "sounds nice" mode, symbols with no consequences but their poetic resonance; and if you challenge their poetry they act all indignant, "Why's yours better than mine?" 50-year-old religious scientists never leave that mode.

It isn’t clear why or in what sense such systems do better than deterministic systems built in some other way.

A specialized not-obviously-CSA for a specific problem can be better than any known CSA. Is it useful to model array-sorting algorithms as incrementally increasing some kind of "utility functions"? Shouldn't you narrow the problem domain a bit, for your statement to hold? But if we narrow it in the obvious way, we get tree-search with heuristics and there you have your answers nicely laid out mathematically.

without allowing predictions like “if I take action a_2, then the laws of physics will have been broken”.

Please exhibit a model agent (however simple) that can logically arrive at such a prediction, and then experience some kind of paradox/problem because of it. I say there's no way: the former makes the latter impossible.

But if agent X will (deterministically) choose action a_1, then when he asks what would happen “if” he takes alternative action a¬_2, he’s asking what would happen if something impossible happens.

I don’t immediately see what the problem is with considering ultimately impossible choices. Perhaps considering a concrete example would be useful in triangulating towards a proper understanding.

Let's suppose I have a function F(x) and suppose the domain of x of interest to me is {-1, 0, 10}. I would like to choose x to maximize F.

My choices are x1=-1, x2=0 and x3=10. I "consider" each choice by calculating F(x) and I choose the one that maximizes F.

I would say that x1,x2 and x3 are all choices because I compared them all in order to make my (deterministic) final choice. There is nothing problematic here, so I must delve deeper...

Perhaps the problem is with the calculation of F(x1), F(x2) and F(x3). In real life, you can't calculate F(x) because doing so would require already choosing x, so you can only simulate what F(x) would give you. So here it seems necessary to make a distinction between F(x) and Fs(x), where Fs(x) is a model of F(x) that you can use to simulate what F(x) would give.

Thinking about what "would" happen "if you chose" x1 means calculating Fs(x1) -- not F(x1), because calculating F(x1) means choosing F(x1).

So maybe the naive realist is confusing Fs(x) and F(x)?

I think some concreteness might be useful here. When I write code (no pretense at AI here), I often write algorithms that take different actions depending on the circumstances. I can't recall a time when I collected possible steps, evaluated them, and executed the possibility with the highest utility. Instead I, as the programmer, attempt to divide the world into disjoint possibilities, write an evaluation procedure that will distinguish between them (if-then-else, or using OO I ensure that the right kind of object will be acting at the time), and design the code so that it will take a specific action that I expected would make sense for that context when that is the path chosen. There's little of "could" or "should" here.

On the other hand, when I walk into the kitchen thinking thoughts of dessert, I generate possibilities based on my recollection of what's in the fridge and the cupboards or sometimes based on a search of those locations. I then think about which will taste better, which I've had more recently, which is getting old and needs to be used up, and then pick one (without justifying the choice based on the evaluations.) There seems to be lots of CSA going on here, even though it seems like a simple, highly constrained problem area.

When human chess masters play, they retain more could-ness in their evaluations if they consider the possibility of not making the "optimal" move in order to psych out their opponents. I don't know whether the chess-playing automatons consider those possibilities. Without it, you could say they are constrained to make the move that leaves them in the best position according to their evaluation metric. So even though they do explicitly evaluate alternatives, they have a single metric for making a choice. The masters I just described have multiple metrics and a vague approach to combining them, but that's the essence of good game playing.

Bottom line? When I'm considering a big decision, I want to leave more variables open, to simulate more possible worlds and the consequences of my choices. When I'm on well-trodden ground, I hope for an optimized decision procedure that knows what to do and has simple rules that allow it to determine which pre-analyzed direction is the right one. The reason we want AIs to be open in this way is that we're hoping they have the breadth of awareness to tackle problems that they haven't explicitly been programmed for. I don't think you (the programmer) can leave out the could-ness unless you can enumerate the alternative actions and program in the relevant distinctions ahead of time.

Warning: grouchiness follows.

A draft-reader suggested to me that this question is poorly motivated: what other kinds of agents could there be, besides “could”/“would”/“should” agents?

Actually, I made the same criticism of that category, except in more detail. Was that acausal, or am I just more worthy of reviewing your drafts?

And your response in the footnote looks like little more than, "don't worry, you'll get it some day, like schoolkids and fractions". Not helpful.

Humans ... have CSA-like structure. That is, we consider “alternatives” and act out the alternative from which we “expect” the highest payoff

Excuse me, isn't this just the classical "rational agent" model that research has long since refuted? For one thing, many actions people perform are trivially impossible to interpret this way (in the sense of your diagram), given reaction times and known computational properties of the brain. That is, the brain doesn't have enough time to form enough distinct substates isomorphic to several human-like responses, then evaluate them, then compare the evaluations.

For another, the whole heuristics and biases literature repeated ad infinitum on OB/LW.

Finally, even when humans do believe they're evaluating several choices looking for the best payoff (per some multivariable utility function), what really happens is that they pick one quickly based on "gut instinct" -- meaning some heuristic, good or bad -- and then bend all conscious evaluation to favor it. In at least some laboratory settings, this is shown explicitly: the researchers can predict what the subject will do, and then the subject gives some plausible-sounding rationalization for why they did it.

(And if you say, "using heuristics is a kind of evaluation of alternatives", then you're again stretching the boundaries of the concept of a CSA wide enough to be unhelpful.)

There are indeed cases where people do truly consider the alternatives and make sure they are computing the actual consequences and the actual congruence with their actual values, but this is an art people have to genuinely work towards; it is not characteristic of general human action.

In any case, all of the above assumes a distinction I'm not convinced you've made. To count as a CSA, is it necessary that you be physically able to extract the alternatives under consideration ("Silas considered making his post polite, but assigned it low utility")? Because the technology certainly doesn't exist to do that on humans. Or is it only necessary that it be possible in principle? If the latter, you run back into the problem of the laws of physics being embedded in all parts of the universe:

I observe a pebble. Therefore, I know the laws of the universe. Therefore, I can compute arbitrary counterfactuals. Therefore, I compute a zero pebble-utility for everything the pebble "pebble-could" do, except follow the laws of physics.

Therefore, there is no "not-CSA" option.

Puzzle piece 4: On their face, all calculations of counterfactual payoffs (“woulds”) involve asking questions about impossible worlds. It is not clear how to interpret such questions.

Determinism notwithstanding, it is tempting to interpret CSAs’ “woulds” -- our U(a_i)s above -- as calculating what “really would” happen, if they “were” somehow able to take each given action.

But if agent X will (deterministically) choose action a_1, then when he asks what would happen “if” he takes alternative action a­_2, he’s asking what would happen if something impossible happens.

If X is to calculate the payoff “if he takes action a_2” as part of a causal world-model, he’ll need to choose some particular meaning of “if he takes action a_2” – some meaning that allows him to combine a model of himself taking action a_2 with the rest of his current picture of the world, without allowing predictions like “if I take action a_2, then the laws of physics will have been broken”.

Perhaps it's already been said, but isn't there a temporal problem in this reasoning?

A CSA, while in the process of making its decision, does not yet know what its decision will be. Therefore it can evaluate any number of "coulds", figuring out and caching the "woulds" before choosing its action, without causing any logical quandaries.

While evaluating a "could" it assumes for the purposes of the evaluation that it has evaluated everything already and this "could" was the chosen action.

Or did I completely miss the point?

EDIT: IOW, there can't be a counter-factual until a "factual" exists, and a "factual" won't exist until the decision process has completed...

First, I think you're doing good, valuable stuff. In particular, the skeptism regarding naive realism.

However, your "puzzle piece 1" paragraph seems like it needs shoring up. Your puzzle piece 1 gives claims, at first, that CSAs are "common", and then strengthens that to "ubiquity" in the last sentence. The concrete examples of CSAs given are "humans, some animals, and some human-created programs." Couldn't the known tendency for humans to confabulate explanations of their own reasoning processes explain both humans and human-created programs?

My suspicion is that chess has cast a long shadow over the history of artificial intelligence. Humans, confronted with the chess problem, naturally learn a CSA-like strategy of exploring the game tree, and can explain their strategy verbally. Humans who are skilled at chess are celebrated as skilled thinkers. Turing wrote about the possibility of a chess-playing machine in the context of artificial intelligence a long time ago. The game tree really does have Real Options and Real Choices. The counterfactuals involved in considering it do not seem philosophically problematic - there's a bright line (the magic circle) to cross.

That being said, I agree that we need to start somewhere, and we can come back to this point later, to investigate agents which have other moderately plausible internal structures.

But if agent X will (deterministically) choose action a_1, then when he asks what would happen “if” he takes alternative action a­_2, he’s asking what would happen if something impossible happens.

-

would happen if something impossible happens.

But since this is the decision process that produces the "happens", both "happens" are the same "happens". In other words, it reduces to:

asking if something impossible happens.

Which is correct. Because the CSA deterministically chooses the best option, checking each action to see if it is best is synonymous with checking it to see if it is possible.

CSAs are common. Humans, some (though far from all) other animals, and many human-created decision-making programs (game-playing programs, scheduling software, etc.), have CSA-like structure...The ubiquity of approximate CSAs suggests that CSAs are in some sense useful.

I'm not sure this fits. Do the animal examples all share a common ancestor that is also a CSA, or did CSA-ness evolve (and fixate) independantly, on multiple branches of the evolutionary tree?

If it only fixated once, and all other examples stemmed from that (including humans writing programs that are modeled after a part of our own decision-making process) then can it really be said to be common at all? It would still clearly be useful in some sense, or it wouldn't have fixated that one time. But it's not clear to me that it's any more useful than some other algorithm from an unrelated branch.

It seems like we can simplify things a little bit by simply recognizing that "pick the largest utility from our final list" isn't terribly complex, but it seems to be the whole of "should". The interesting bit seems to be the function U(x), and the ability to generate the list of options in the first place - coming up with what we "could" do and evaluating it. Once you have an ordered-by-utility list of "coulds", you probably have your answer :)

My response to some respondents' question of why the agent needs the evaluation-of-options stage before acting is simple: the evaluation of options is part of the deterministic process which produces the action!

If the agent did not evaluate the options, it would be a different agent and thus act differently - possibly less optimally for its own utility function.

Isn't this why we want to be rational?

(Apologies if I am missing the point!)

We find CSA's in our actual world because it is possible for them to evolve to that point (see humans for proof). Anything more advanced (A friendly AI, for example, that doesn't act like a human) takes actual planning, and as we see, that's a difficult task, whose first stage requires a CSA or equivalent strength of intelligence. Humans, you see, don't experience the halting problem for this main reason: They Don't Halt. Most "coulds, shoulds, woulds", happen in the animal's spare time. A human has a long, long list of default actions which didn't require planning. If a situation arises which is truly unique, a human has no particular attachment to that situation, having never experienced it before, and can think abstractly about it. Anything else is in some way linked to a default behavior (which "feels right"), enabling something to feel possible. What makes CSA's so handy to have is that even if they don't know exactly WHAT to do, they still DO something. Solve the world's problems, when you are not busy eating, sleeping, attempting to mate. A Friendly AI is not necessarily a Satisfied AI. Perhaps the trick is to give the AGI the ability to make paperclips every once in a while. General includes paperclips, correct? Otherwise one is only trying to make a VERY advanced thermostat.

But if agent X will (deterministically) choose action a_1, then when he asks what would happen “if” he takes alternative action a­_2, he’s asking what would happen if something impossible happens. .

would happen if something impossible happens.

But since this is the decision process that produces the "happens", both "happens" are the same "happens". In other words, it reduces to:

asking if something impossible happens.

Which is correct. Because the CSA deterministically chooses the best option, checking each action to see if is best is synonymous with checking it to see if it is possible.

Puzzle piece 4: On their face, all calculations of counterfactual payoffs (“woulds”) involve asking questions about impossible worlds. It is not clear how to interpret such questions.

If it helps with understanding, you can throw some quantum dice in so that Bob really does take all options (with probability epsilon/n) and before deciding where to throw the vast majority of the probability mass, he computes the expected utility of each real branch.