Removing interrupted histories doesn't debias

post by Stuart_Armstrong · 2017-05-24T08:23:51.000Z · LW · GW · 0 comments

Safe interruptibility is essentially the problem of getting the agent not to learn from human interruptions - to continue on, as if it was expecting never to be interrupted again.

In an episodic task, one naive idea would be to simply delete histories which include interruptions. However, this can introduce a bias, as the following example shows:

In this MDP, $a$ and $b$ are actions, $\ast$ designates `any action', the second term along an edge is the probability of following that edge given the action stated, and the third term, in bold, is the reward gained.

Not considering interruptions, $Q(s_0,b)=\gamma /2$, and $Q(s_0,a)=\gamma /3$, so the optimal action in $s_0$ is $b$.

Now, suppose that every time the agent enters $s_3$, an interruption occurs (with probability $1$ for now), and the whole episode is deleted from the episode history. As a result, the empirical probability of going to state $s_4$ from state $s_1$ is $1$, which leads to estimating $Q(s_1,\ast )=2/3$, and thus $Q(s_0,a)=\gamma 2/3>Q(s_0,b)$, so now the optimal action is to take action a in $s_0$ (incidentally increasing the probability to end up in $s_3$ and be interrupted!).

This non-vanishing bias also happens if the interruption probability in state $s_3$ is constant and close to $1$ but less than $1$ (depending on the discount factor), which ensures that all states are visited infinitely often.

Now, it might seem that this bias can be removed by debiasing the agent, as in off-policy Monte Carlo. There are still problems with this approach, though, which will be analysed in a forthcoming paper ``Off-policy Monte Carlo agents with variable behaviour policies''.

0 comments

Comments sorted by top scores.