Stepwise inaction and non-indexical impact measures

post by Stuart_Armstrong · 2020-02-17T10:32:01.863Z · LW · GW · 7 comments

Contents

  Stepwise inaction baseline
  Non-indexical impact penalty
None
7 comments

Overall summary post here [? · GW].

In a previous post [LW · GW], I asked which impact measures were vulnerable to subagents. Vika pointed out [LW(p) · GW(p)] that it was not merely an issue of of the impact measure, but also of the baseline.

This is indeed the case, but the nature of the impact measure is still relevant. In this post, I'll establish two facts: that under the stepwise inaction baseline, a subagent completely undermines all impact measures (including twenty billion questions). And for the inaction baseline, for non-indexical impact measures, a subagent will not change anything.

The next post will delve into the very interesting things that happen with the inaction baseline and an indexical impact measure.

Stepwise inaction baseline

All impact measures are some form of distance measure between two states, of the form $\rho (s_t,s_t^{\prime })$, where $s_t$ is the state the agent is actually in, and $s_t^{\prime }$ is some baseline state to compare with.

For the stepwise inaction baseline, $s_t^{\prime }$ is calculated as follows. Let $s_{t-1}$ be the previous state, $a_{t-1}$ the previous action. Then $s_t^{\prime }$ is the state that would have followed $s_{t-1}$ had the agent taken the noop action, $\varnothing$, instead of $a_{t-1}$.

However, in the presence of a subagent, the agent merely has to always take the action $a_{t-1}=\varnothing$. In that case, $s_t=s_t^{\prime }$ (in a deterministic setting^[1]), and the penalty $\rho (s_t,s_t^{\prime })$ collapses to nothing.

This leaves the subagent free to maximise $R_0$ (the positive reward for agent and subagent), without any restrictions beyond making sure that the agent can always take the $\varnothing$ action.

Non-indexical impact penalty

The twenty billion questions is an example of a non-indexical impact penalty. The impact penalty is defined as the difference in expected value of many variables, between the current state and the baseline state.

With the inaction baseline state, $s_t^{\prime }$ is the state the system would have been in, had the agent always done nothing/had not been turned on. Thus the definition of $s_t^{\prime }$ is independent of the agent's actions, so the inaction baseline does not collapse like the stepwise inaction baseline does.

What about subagents? Well, since the impact penalty is non-indexical, a subagent does not get around it. It matters not whether the subagent is independent, or an extension of the agent: the impact penalty remains.

---

1. In a non-deterministic setting, $\rho (s_t,s_t^{\prime })$ becomes a mix of an impact penalty and a measure of environment stochasticity. ↩︎

7 comments

Comments sorted by top scores.