I didn't find the conclusion about the smoke-lovers and non-smoke-lovers obvious in the EDT case at first glance, so I added in some numbers and ran through the calculations that the robots will do to see for myself and get a better handle on what not being able to introspect but still gaining evidence about your utility function actually looks like.

Suppose that, out of the $N$ robots that have ever been built, $nN$ are smoke-lovers and $(1-n)N$ are non-smoke-lovers. Suppose also the smoke-lovers end up smoking with probability $p$ and non-smoke-lovers end up smoking with probability $q$.

Then $(pn+q(1-n\left)\right)N$ robots smoke, and $\left(\right(1-p)n+(1-q\left)\right(1-n\left)\right)N$ robots don't smoke. So by Bayes' theorem, if a robot smokes, there is a  $\frac{pn}{pn+q(1-n)}$ chance that it's killed, and if a robot doesn't smoke, there's a $\frac{(1-p)n}{1-(pn+q(1-n\left)\right)}$chance that it's killed.

Hence, the expected utilities are:

• An EDT non-smoke-lover looks at the possibilities. It sees that if it smokes, it expects to get$-101\frac{pn}{pn+q(1-n)}-1(1-\frac{pn}{pn+q(1-n)})$ utilons, and that if it doesn't smoke, it expects to get $-100\frac{(1-p)n}{1-(pn+q(1-n\left)\right)}$ utilons.
• An EDT smoke-lover looks at the possibilities. It sees that if it smokes, it expects to get $-90\frac{pn}{pn+q(1-n)}+10(1-\frac{pn}{pn+q(1-n)})$ utilons, and if it doesn't smoke, it expects to get $-100\frac{(1-p)n}{1-(pn+q(1-n\left)\right)}$ utilons.

Now consider some equilibria. Suppose that no non-smoke-lovers smoke, but some smoke-lovers smoke. So $q=\epsilon$ and $p\gg \epsilon$. So (taking limits as $\epsilon \to 0$ along the way):

• non-smoke-lovers expect to get $-101$ utilons if they smoke, and $-100\frac{n-pn}{1-pn}$ utilons if they don't smoke. $n<1$ so non-smoke-lovers will choose not to smoke.
• smoke-lovers expect to get $-90$ utilons if they smoke, and $-100\frac{n-pn}{1-pn}$ utilons if they don't smoke. Smoke-lovers would be indifferent between the two if $p=10-\frac{9}{n}$. This works fine if at least 90% of robots are smoke lovers, and equilibrium is achieved. But if less than 90% of robots are smoke-lovers, then there is no point at which they would be indifferent, and they will always choose not to smoke.

But wait! This is fine if more than 90% are smoke-lovers, but if fewer than 90% are smoke-lovers, then they would always choose not to smoke, that's inconsistent with the assumption that $p$ is much larger than $\epsilon$. So instead suppose that $p$ is only only a little bit bigger than $\epsilon =q$, say that $p=k\epsilon$. Then:

• non-smoke-lovers expect to get $-100(\frac{k}{1+(k-1)n}+\frac{1}{100n})n$ utilons if they smoke, and $-100n$ utilons if they don't smoke. They will choose to smoke if $k<1+\frac{1}{101n-100n^2}$, i.e. if smoke-lovers smoke so rarely that not smoking would make them believe they're a smoke-lover about to be killed by the blade runner.
• smoke-lovers expect to get  $-100(\frac{k}{1+(k-1)n}-\frac{1}{10n})n$ utilons if they smoke, and $-100n$ utilons if they don't smoke. They are indifferent between these two when $k=1+\frac{1}{9n-10n^2}$. This means that, when $k$ is at the equilibrium point, non-smoke-lovers will not choose to smoke when fewer than 90% of robots are smoke-lovers, which is exactly when this regime applies.

I wrote a quick python simulation to check these conclusions, and it was the case that $p=10-\frac{9}{n}$ for $0.9<n<1$, and $p=(1+\frac{1}{9n-10n^2})\epsilon$ for $0<n<0.9$ there as well.

From my perspective, I don’t think it’s been adequately established that we should prefer updateless CDT to updateless EDT

I agree with this.

It would be nice to have an example which doesn’t arise from an obviously bad agent design, but I don’t have one.

I’d also be interested in finding such a problem.

I am not sure whether your smoking lesion steelman actually makes a decisive case against evidential decision theory. If an agent knows about their utility function on some level, but not on the epistemic level, then this can just as well be made into a counter-example to causal decision theory. For example, consider a decision problem with the following payoff matrix:

Smoke-lover:

• Smokes:

  • Killed: 10
  • Not killed: -90

• Doesn't smoke:

  • Killed: 0
  • Not killed: 0

Non-smoke-lover:

• Smokes:

  • Killed: -100
  • Not killed: -100

• Doesn't smoke:

  • Killed: 0
  • Not killed: 0

For some reason, the agent doesn’t care whether they live or die. Also, let’s say that smoking makes a smoke-lover happy, but afterwards, they get terribly sick and lose 100 utilons. So they would only smoke if they knew they were going to be killed afterwards. The non-smoke-lover doesn't want to smoke in any case.

Now, smoke-loving evidential decision theorists rightly choose smoking: they know that robots with a non-smoke-loving utility function would never have any reason to smoke, no matter which probabilities they assign. So if they end up smoking, then this means they are certainly smoke-lovers. It follows that they will be killed, and conditional on that state, smoking gives 10 more utility than not smoking.

Causal decision theory, on the other hand, seems to recommend a suboptimal action. Let $a_1$ be smoking, $a_2$ not smoking, $S_1$ being a smoke-lover, and $S_2$ being a non-smoke-lover. Moreover, say the prior probability $P\left(S_1\right)$ is $0.5$. Then, for a smoke-loving CDT bot, the expected utility of smoking is just

$E\left[U|a_1\right]=P\left(S_1\right)\cdot U(S_1\wedge a_1)+P\left(S_2\right)\cdot U(S_2\wedge a_1)=0.5\cdot 10+0.5\cdot (-90)=-40$,

which is less then the certain $0$ utilons for $a_2$. Assigning a credence of around $1$ to $P\left(S_1|a_1\right)$, a smoke-loving EDT bot calculates

$E\left[U|a_1\right]=P\left(S_1|a_1\right)\cdot U(S_1\wedge a_1)+P\left(S_2|a_1\right)\cdot U(S_2\wedge a_1)\approx 1\cdot 10+0\cdot (-90)=10$,

which is higher than the expected utility of $a_2$.

The reason CDT fails here doesn’t seem to lie in a mistaken causal structure. Also, I’m not sure whether the problem for EDT in the smoking lesion steelman is really that it can’t condition on all its inputs. If EDT can't condition on something, then EDT doesn't account for this information, but this doesn’t seem to be a problem per se.

In my opinion, the problem lies in an inconsistency in the expected utility equations. Smoke-loving EDT bots calculate the probability of being a non-smoke-lover, but then the utility they get is actually the one from being a smoke-lover. For this reason, they can get some "back-handed" information about their own utility function from their actions. The agents basically fail to condition two factors of the same product on the same knowledge.

Say we don't know our own utility function on an epistemic level. Ordinarily, we would calculate the expected utility of an action, both as smoke-lovers and as non-smoke-lovers, as follows:

$E\left[U|a\right]=P\left(S_1|a\right)\cdot E[U|S_1,a]+P\left(S_2|a\right)\cdot E[U|S_2,a]$,

where, if $U_1$ ($U_2$) is the utility function of a smoke-lover (non-smoke-lover), $E[U|S_i,a]$ is equal to $E\left[U_i|a\right]$. In this case, we don't get any information about our utility function from our own action, and hence, no Newcomb-like problem arises.

I’m unsure whether there is any causal decision theory derivative that gets my case (or all other possible cases in this setting) right. It seems like as long as the agent isn't certain to be a smoke-lover from the start, there are still payoffs for which CDT would (wrongly) choose not to smoke.

The claim that "this isn’t changed at all by trying updateless reasoning" depends on the assumptions about updateless reasoning. If the agent chooses a policy in the form of a self-sufficient program, then you are right. On the other hand, if the agent chooses a policy in the form of a program with oracle access to the "utility estimator," then there is an equilibrium where both smoke-lovers and non-smoke-lovers self-modify into CDT. Admittedly, there are also "bad" equilibria, e.g. non-smoke-lovers staying with EDT and smoke-lovers choosing between EDT and CDT with some probability. However, it seems arguable that the presence of bad equilibria is due to the "degenerate" property of the problem that one type of agents have incentives to move away from EDT whereas another type has exactly zero such incentives.

I like this line of inquiry; it seems like being very careful about the justification for CDT will probably give a much clearer sense of what we actually want out of "causal" structure for logical facts.

First of all, it seems to me that "updateless CDT" and "updateless EDT" are the same for agents with access to their own internal states immediately prior to the decision theory computation: on an appropriate causal graph such internal states would be the only nodes with arrows leading to the nodes "output of decision theory", so if their value is known, then severing those arrows does not affect the computation for updating on an observation of the value of the "output of decision theory" node. So the counterfactual and conditional probability distributions are the same, and thus CDT and EDT are the same.

Anyway, here is another example where CDT is superior to EDT, without obviously bad agent design. Suppose an agent needs to decide whether to undertake a mission that may either succeed or fail, with utilities:

Not trying: 0

Trying and failing: -1000

Trying and succeeding: 100

The agent initially believes that its probability of success is 50%, but it can perform an expensive computation (-10 utility) to update this probability to either 1% or 99%. In any decision theory, if the new probability is 1% then it will not try, and if the new probability is 99% then it will try. However, it also has to make the decision whether to perform the computation in the first place, and if not, whether to try anyway or not. Under CDT, the choice is easy: computation gives an expected utility of

0.5(0) + 0.5(0.99(100) - 0.01(1000)) - 10 = 34.5

trying without computation gives an expected utility of

0.5(100) - 0.5(1000) = -450

and not trying without computation gives a utility of 0, so the agent performs the computation.

Under EDT, the equilibrium solution cannot be to always perform the computation. Indeed, if it were so, then the expected utility for trying without computation would be

0.99(100) - 0.01(1000) = 89

while the other two expected utilities would be the same, so the agent would try without computation. (If the agent observes itself trying, it infers that it must have done so because it computed the probability as 99%, and thus the probability of success must be 99%.)

The actual equilibrium would likely be to usually run the computation, but sometimes try without computation. But it is irrelevant, the point is that EDT recommends something different than the correct CDT solution.

Note that the computation cost is in fact irrelevant to the example, I only introduced it to motivate that a decision needs to be made about whether to make the computation or not. In fact, EDT would recommend a non-optimal solution even if there were no cost associated to running the computation.

As before, the key feature of this example is that while computing expected utilities, the agent does not have access to information about its internal states prior to choosing its action: it must choose to either update its priors about them using information from its counterfactual action (EDT) or sever this causal connection before updating (CDT).

As far as I can tell, there is no analogous setup where EDT is preferable to CDT.

Note: The reason the idea of this example can't be used in the setup of the OP is that in the OP, the inaccessible variable (the utility function) is actually used in the decision theory computation, whereas in my example the inaccessible variable (the probability of success) is inaccessible because it is hard to compute, so it wouldn't make sense to use it in the computation.