Two Alternatives to Logical Counterfactuals

I too have recently updated (somewhat) away from counterfactual non-realism. I have a lot of stuff I need to work out and write about it.

I seem to have a lot of disagreements with your post.

Given this uncertainty, you may consider material conditionals: if I take action X, will consequence Q necessarily follow? An action may be selected on the basis of these conditionals, such as by determining which action results in the highest guaranteed expected utility if that action is taken.

I don't think material conditionals are the best way to cash out counterfactual non-realism.

• The basic reason I think it's bad is the happy dance problem [LW · GW]. This makes it seem clear that the sentence to condition on should not be $obs\to act$.
• If the action can be viewed as a function of observations, conditioning on $A\left(obs\right)=act$ makes sense. But this is sort of like already having counterfactuals, or at least, being realist that there are counterfactuals about whan would do if the agent observed different things. So this response can be seen as abandoning counterfactual non-realism.
• A different approach is to consider conditional beliefs rather than material implications. I think this is more true to counterfactual non-realism. In the simplest form, this means you just condition on actions (rather than trying to condition on something like $obs\to act$ or $A\left(obs\right)=act$). However, in order to reason updatelessly, you need something like conditioning on conditionals [LW · GW], which complicates matters.
• Another reason to think it's bad is Troll Bridge [LW · GW].
• Again if the agent thinks there are basic counterfactual facts, $A\Rightarrow B$ (required to respect $A,A\Rightarrow B⊢B$ but little else -- ie entirely determined by subjective beliefs), then the agent can escape Troll Bridge by disagreeing with the relevant inference. But this, of course, rejects the kind of counterfactual non-realism you intend.
• To be more in line with counterfactual non-realism, we would like to use conditional probabilities instead. However, conditional probability behaves too much like material implication to block the Troll Bridge argument. However, I believe that there is an account of conditional probability which avoids this by rejecting the ratio analysis of conditional probability -- ie Bayes' definition -- and instead regards conditional probability as a basic entity. (Along the lines of what Alan Hájek goes on and on about.) Thus an EDT-like procedure can be immune to both 5-and-10 and Troll Bridge. (I claim.)

As for policy-dependent source code, I find myself quite unsympathetic to this view.

• If the agent is updateful, this is just saying that in counterfactuals where the agent does something else, it might have different source code. Which seems fine, but does it really solve anything? Why is this much better than counterfactuals which keep the source code fixed but imagine the execution trace being different? This seems to only push the rough spots further back -- there can still be contradictions, e.g. between the source code and the process by which programmers wrote the source code. Do you imagine it is possible to entirely remove such rough spots from the counterfactuals?
• So it seems you intend the agent to be updateless instead. But then we have all the usual issues with logical updatelessness. If the agent is logically updateless, there is absolutely no reason to think that its beliefs about the connections between source code and actual policy behavior is any good. Making those connections requires actual reasoning, not simply a good enough prior -- which means being logically updateful. So it's unclear what to do.
• Perhaps logically-updateful policy-dependent-source-code is the most reasonable version of the idea. But then we are faced with the usual questions about spurious counterfactuals, chicken rule, exploration, and Troll Bridge. So we still have to make choices about those things.

It is easy to see that this idea of logical counterfactuals is unsatisfactory. For one, no good account of them has yet been given. For two, there is a sense in which no account could be given; reasoning about logically incoherent worlds can only be so extensive before running into logical contradiction.

I've been doing some work on this topic, and I am seeing two schools of thought on how to deal with the problem of logical contradictions you mention. To explain these, I'll use an example counterfactual not involving agents and free will. Consider the counterfactual sentence: `if the vase had not been broken, the floor would not have been wet'. Now, how can we compute a truth value for this sentence?

School of thought 1 proceeds as follows: we know various facts about the world, like that the vase is broken and that the floor is wet. We also know general facts about vases, breaking, water, and floors. Now we add the extra fact that the vase is not broken to our knowledge base. Based on this extended body of knowledge, we compute the truth value of the claim 'the floor is not wet'. Clearly, we are dealing with a knowledge base that contains mutually contradictory facts: the vase is both broken and it is not broken. Under normal mathematical systems of reasoning, this will allow us to prove any claim we like: the truth value of any sentence becomes 1, which is not what we want. Now, school 1 tries to solve this by coming up with new systems of reasoning that are tolerant of such internal contradictions, systems that will make computations that will produce the 'obviously true' conclusions only, of that will derive the `obviously true' conclusions before deriving the `obviously false' ones, or that compute probabilistic truth values such a way that those of the `obviously true' conclusions are higher. In MIRI terminology, I believe this approach goes under the heading 'decision theory'. I also interpret the two alternative solutions you mention above as following this school of thought. Personally, I find this solution approach not very promising or compelling.

School of thought 2, which includes Pearl's version of counterfactual reasoning, says that if you want to reason (or if you want a machine to reason) in a counterfactual way, you should not just add facts to the body of knowledge you use. You need to delete or edit other facts in the knowledge base too, before you supply it to the reasoning engine, exactly to avoid inputting a knowledge base that has internal contradictions. For example, if you want to reason about 'if the vase had not been broken', one thing you definitely need to do is to first remove the statement (or any information leading to the conclusion that) `the vase is broken' from the knowledge base that goes into your reasoning engine. You have to do this even though the fact that the vase is broken is obviously true for the current world you are in.

So school 2 avoids the problem of having to somehow build a reasoning engine that does the right thing even when a contradictory knowledge base is input. But it trades this for the problem of having to decide exactly what edits will be made to the knowledge base to eliminate the possibility of having such contradictions. In other words, if you want a machine to reason in a counterfactual way, you have to make choices about the specific edits you will make. Often, there are many possible choices, and different choices may lead to different probability distributions in the outcomes computed. This choice problem does not bother me that much, I see it as having design freedom. But if you are a philosopher of language trying to find a single obvious system of meaning for natural language counterfactual sentences, this choice problem might bother you a lot, you might be tempted to find some kind of representation-independent Occam's razor that can be used to decide between counterfactual edits.

Overall, my feeling is that school 2 gives an account of logical counterfactuals that is good enough for my purposes in AGI safety work.

As a trivial school 1 edge case, one could design a reasoning engine that can deal with contradictory facts in its input knowledge base as follows: the engine first makes some school 2 edits on its input to remove the contradictions, and then proceeds calculating the requested truth value. So one could argue that the schools are not fundamentally different, though I do feel they are different in outlook, especially in their outlook on how necessary or useful it will be for AGI safety to resolve certain puzzles.

Suppose you learn about physics and find that you are a robot. You learn that your source code is "A". You also believe that you have free will; in particular, you may decide to take either action X or action Y.

My motivation for talking about logical counterfactuals has little to do with free will, even if the philosophical analysis of logical counterfactuals does.

The reason I want to talk about logical counterfactuals is as follows: suppose as above that I learn that I am a robot, and that my source code is "A"(which is presumed to be deterministic in this scenario), and that I have a decision to make between action X and action Y. In order to make that decision, I want to know which decision has better expected utility. The problem is that, in fact, I will either choose X or Y. Suppose without loss of generality that I will end up choosing action X. Then worlds in which I choose Y are logically incoherent, so how am I supposed to reason about the expected utility of choosing Y?

In this possible world, it is the case that "A" returns Y upon being given those same observations. But, the output of "A" when given those observations is a fixed computation, so you now need to reason about a possible world that is logically incoherent, given your knowledge that "A" in fact returns X. This possible world is, then, a logical counterfactual: a "possible world" that is logically incoherent.

Simpler solution: in that world, your code is instead A', which is exactly like A, except that it returns Y in this situation. This is the more general solution derived from Pearl's account of counterfactuals in domains with a finite number of variables (the "twin network construction").

Last year, my colleagues and I published a paper on Turing-complete counterfactual models ("causal probabilistic programming"), which details how to do this, and even gives executable code to play with, as well as a formal semantics. Have a look at our predator-prey example, a fully worked example of how to do this "counterfactual world is same except blah" construction.

http://www.zenna.org/publications/causal.pdf

I found parts of your framing quite original and I'm still trying to understand all the consequences.

Firstly, I'm also opposed to characterising the problem in terms of logical counterfactuals. I've argued before that Counterfactuals are an Answer Not a Question [LW · GW], although maybe it would have been clearer to say that they are a Tool Not a Question instead. If we're talking strictly, it doesn't make sense to ask what maths would. be like if 1+1=3 as it doesn't, but we can construct a para-consistent logic where it makes sense to do something analogous to pretending 1+1=3. And so maybe one form of "logical counterfactual" could be useful for solving these problems, but that doesn't mean asking what logical counterfactuals are, as though they were ontologically basic, as though they were in the map not the territory, as though they were a single unified concept, makes sense.

Secondly, "free will" is such a loaded word that using it in a non-standard fashion simply obscures and confuses the discussion. Nonetheless, I think you are touching upon an important point here. I have a framing which I believe helps clarify the situation. If there's only one possible decision, this gives us a Trivial Decision Problem [LW · GW]. So to have a non-trivial decision problem, we'd need a model containing at least two decisions. If we actually did have libertarian free will, then our decision problems would always be non-trivial. However, in the absence of this, the only way to avoid triviality would be to augment the factual with at least one counterfactual.

Counterfactual non-realism: Hmm... I see how this could be a useful concept, but the definition given feels a bit vague. For example, recently I've been arguing in favour of what counts as a valid counterfactual being at least partially a matter of social convention [LW · GW]. Is that counterfactual non-realism?

Further, it seems a bit strange to associate material conditions with counterfactual non-realism. Material conditions only provide the outcome when we have a consistent counterfactual. So, either a) we believe in libertarian free will b) we use something like the erasure approach to remove information such that we have multiple consistent possibilities (see https://www.lesswrong.com/posts/BRuWm4GxcTNPn4XDX/deconfusing-logical-counterfactuals). Proof-based UDT doesn't quite use material conditionals, it uses a paraconsistent version of them instead. Although, maybe I'm just being too pedantic here. In any case, we can find ways of making paraconsistent logic behave as expected in any scenario, however it would require a separate ground. That is, it isn't enough that the logic merely seems to work, but we should be able to provide a separate reason for why using a paraconsistent logic in that way is good.

Also, another approach which kind of aligns with counterfactual non-realism is to say that given the state of the universe at any particular time we can determine the past and future and that there are no counterfactuals beyond those we generate by imagining state Y at time T instead of state X. So, to imagine counterfactually taking action Y we replace the agent doing X with another agent doing Y and flow causation both forwards and backwards. (See this post [LW · GW] for more detail). It could be argued that these count as counterfactuals, but I'd align it with counterfactual non-realism as it doesn't have decision counterfactuals as seperate ontological elements.

Policy-dependent source code - this is actually a pretty interesting framing. I've always defaulted to thinking about counterfactuals in terms of actions, but when we're talking about things in terms of problems like Counterfactual Mugging, characterising counterfactuals in terms of policy might be more natural. It's strange why this feels fresh to me - I mean UDT takes this approach - but I never considered the possibility of non-UDT policy counterfactuals. I guess from a philosophical perspective it makes sense to first consider whether policy-dependent source code makes sense and then if it does further ask whether UDT makes sense.

On my current understanding of this post, I think I have a criticism. But I'm not sure if I properly understand the post, so tell me if I'm wrong in my following summary. I take the post to be saying something like the following:

'Suppose, in fact, I take the action A. Instead of talking about logical counterfactuals, we should talk about policy-dependent source code. If we do this, then we can see that initial talk about logical counterfactuals encoded an error. The error is not understanding the following claim: when asking what would have happened if I had performed some action A* $\ne$ A, observing that I do A* is evidence that I had some different source code. Thus, in analysing that counterfactual statement, we do not need to refer to incoherent 'impossible worlds'.

If my summary is right, I'm not sure how policy-dependent source code is a solution to the global accounting problem. This is because the agent, when asking what would have happened if I had done Y, still faces a global accounting problem. This is because the agent must then assume they have some different source code B, and it seems like choosing an appropriate B will be underdetermined. That is, there is no unique source code B to give you a determinate answer about what would have happened if you performed A*. I can see why thinking in terms of policy-dependent source code would be attractive if you were a nonrealist about specifically logical counterfactuals, and a realist about different kinds of counterfactuals. But that's not what I took you to be saying.

I see the problem of counterfactuals as essentially solved by quasi-Bayesianism [AF(p) · GW(p)], which behaves like UDT in all Newcomb-like situations. The source code in your presentation of the problem is more or less equivalent to Omega in Newcomb-like problems. A TRL [AF(p) · GW(p)] agent can also reason about arbitrary programs, and learn that a certain program acts as a predictor for its own actions.

This approach has some similarity with material implication and proof-based decision theory, in the sense that out of several hypothesis about counterfactuals that are consistent with observations, the decisive role is played by the most optimistic hypothesis (the one that can be exploited for the most expected utility). However, it has no problem with global accounting and indeed it solves counterfactual mugging successfully.

Short:

Hence, they would not have seen that their source code is "A".

Unless something interfered with what they saw - there need not be pure/true observations.

Instead, if the agent were to take action Y upon seeing that their source code is "A", their source code must be something else, perhaps "B".

And something might have incentive to do so if the agent were to do X if it "saw its source code was A" and were to do Y if it "saw its source code was B". While A and B may be mutually exclusive, the actual policy "might" be dependent on observations of either.

Long:

[1] If a program takes long enough to run, it may never be found that it does halt. In a sense, the fact that its output is determined does not mean it can (or will) be deduced.

there is no way for two different policies to be compatible with the same source code.

And set of inputs.

Formally, a policy is a function mapping an observation history to an action. It is distinct from source code, in that the source code specifies the implementation of the policy in some programming language, rather than itself being a policy function.

Logically, it is impossible for the same source code to generate two different policies. There is a fact of the matter about what action the source code outputs given an observation history (assuming the program halts). Hence there is no way for two different policies to be compatible with the same source code.

Overall take:

Dynamic versus static:

Consider the numbers 3, 1, 2, 4.

There exists more than one set of actions that 'transforms' the above into: 1, 2, 3, 4.

(It can also be transformed into a sorted list by deleting the 3...)

A sorting method however, does not always take a list and move the first element to the third position, or even necessarily do so in every case where the first element is three.

While deterministic, its behavior depends upon an input. Given the input, the actions it will take are known (or follow from the source code in principle[1]).

This can be generalized further, in the case of a sorting program that takes both a set of objects, and a way of ordering. Perhaps a program can even be written that reasons about some policy, and based on the results, makes an output conditional on what it finds. Thus the "logical counterfactual" does not exist per se, but is a way of thinking used in order to handle the different cases, as it is not clear which one is the case, though only one may be possible.

More specific:

Formally, a policy is a function mapping an observation history to an action. It is distinct from source code, in that the source code specifies the implementation of the policy in some programming language, rather than itself being a policy function.

Though a policy may include/specify (simpler) policies, and thus by extension, a source code may as well, though the different threads will probably be weaved together.

I'm trying t understand where exactly in your approach you sneak in the free will...

What is "linear interactive causality"?

this “what would have happened” world is logically incoherent.

There is a logical contradiction between the idea that your actions are determined, and the idea that you could have acted differently under the exact same circumstances. There is no such problem if you do not assume determinism, meaning that the "problem" of logical counterfactuals is neither unavoidable nor purely logical -- it is not purely logical because a metaphysical assumption, an assumption about the way reality works is involved.

The assumption of determinism is implicit in talking of yourself as a computer programme, and the assumption of indeterminism is implicit in talking about yourself as nonetheless having free will.

A purely logical counterfactual , a logical counterfactual properly so-called, is a hypothetical state of affairs, where a different input or set of preconditions is supposed, and a different, also hypothetical output or result obtains. Such a counterfactual is logically consistent -- it just isn't consistent with what actually occurred.

According to counterfactual nonrealism, there is no fact of the matter about what “would have happened” had a different action been taken.

People calculate logical counterfactuals all the time. You can figure out what output a programme will give in response to an input it has never received by looking at the code. But note that that is a purely epistemological issue. There may be a separate, ontological, not epistemological issue about real counterfactuals. If you have good reason to believe in determinsim, which you don't, you should disbelieve in real counterfactuals. But that says nothing about logical counterfacuals. So long as some hygiene is exercised about the epistemological/ontological distinction, and the logical/real disinticntion then there is no problem.

The apparent nondeterminism is, then, only due to the epistemic limitation of the agent at the time of making the decision, a limitation not faced by a later version of the agent (or an outside agent) with more computation power.

Note that problems agents have in introspecting their own decision making are not problems with counterfactuals (real or logical) per se.

This leads to a sort of relativism: what is undetermined from one perspective may be determined from another.

It doesn't lead to serious relativism, because the perspectives are asymmetrical. The agent that knows more is more right.

A problem that comes up is that of “spurious counterfactuals”

A "spurious" counterfactual is just a logical, as opposed to real, counterfactual. The fact that it could never have occurred means it was never a real counterfactual.

This was interesting to read but not entirely sure just what it means to me and thinking anything through.

As I was reading I started to think along the lines of you policy side -- perhaps the question is not about how to twist A code into outputting Y but rather why not just consider the agent runs some other code. (The immediate problem with my thought is that of the infinite regress.) But also when thinking about counterfactuals that is in a sense what I am exploring. But I would express that more as what if action/path Y were taken, where does that lead? Is that a better result? If so then the response is about updating some priors related to the input to A or updating A. In this sense the question is not about the logical problems of getting A to output Y when we know it has to output X but about the internal logic and decision making performance of A and if we need to update to A'.

I am also wondering if including the whole free-will aspect add value. If you just took that aspect out what changes for your thinking? Or is the whole question of free-will part of the philosophical question you need to address. If so your post did prompt a thought on my thinking about free-will, particularly in the context of rational mindsets. I don't know if someone has already followed the line of thinking (but would certainly think so) but I don't think free-will can be rationally explored and explained within the confines of pure logical consistency.