Rationality and overriding human preferences: a combined model

post by Stuart_Armstrong · 2017-10-20T22:47:04.000Z · LW · GW · 0 comments

Contents

  Feast or heroin famine
  Overriding rewards and regret
None
No comments

A putative new idea for AI control; index here.

Previously, I presented a model in which a “rationality module” kept track of two things: how well a human was maximising their actual reward, and whether their preferences had been overridden by AI action.

The second didn’t integrate well into the first, and was tracked by a clunky extra Boolean. Since the two didn’t fit together, I was going to separate the two concepts, especially since the Boolean felt a bit too... Boolean, not allowing for grading. But then I realised that they actually fit together completely naturally, without the need for arbitrary Booleans or other tricks.

---

Feast or heroin famine

Consider the situation detailed in the following figure. An AI has the opportunity to surreptitiously inject someone with heroin ($I$) or not do so ($\neg I$). If it doesn’t, the human will choose to enjoy a massive feast ($F$); if it does, the human will instead choose more heroin ($H$).

So the human policy is given by $\pi \left(I\right)=H,\pi \left(\neg I\right)=F$. The human rationality and reward are given by a pair $(m,R)$, where $R$ is the human reward and $m$ measures their rationality - how closely their actions conform with their reward.

The module $m$ can be seen as a map from rewards to policies (or, since policies are maps from histories to actions, m can be seen as mapping histories and rewards to actions). The pair $(m,R)$ are said to be compatible if $m\left(R\right)=\pi$, the human policy.

There are three natural $R$s to consider here: $R_p$, a generic pleasure. Next, $R_e$, the ‘enjoyment’ reward, where enjoyment is pleasure endorsed as ‘genuine’ by common judgement. Assume that $R_p\left(H\right)=1$, $R_p\left(F\right)=1/3$, $R_e\left(F\right)=1/2$, and $R_e\left(H\right)=0$. Finally, there is the twisted reward $R_t$, which is $R_p$ conditional on $I$ and $R_e$

There are two natural $m$s: $m_r$, the fully rational module. And $m_f$, the module that is fully rational conditional on $I$, but always maps to $H$ if $I$ is chosen: $m\left(R\right)\left(I\right)=H$, for all $R$.

The pair $m_r\left(R_e\right)$ is not compatible with $\pi$: it predicts that the human would take action $F$ following $I$ (feast following injection). The reward $R_p$ is compatible with neither $m$: it predicts $H$ following $\neg I$ (heroin following no injection).

The other three pairs are compatible: $m_r\left(R_t\right)$, $m_f\left(R_t\right)$, and $m_f\left(R_e\right)$ all give the correct policy $\pi$.

Overriding rewards and regret

This leads to a definition of when the AI is overriding human rewards. Given a pair $(m,R)$, with $m\left(R\right)=\pi$, and AI’s action $A$ overrides the human reward if $\pi |A$ is poorly optimised for maximising $R$. If $V^{\pi }\left(R|A\right)$ is the expected reward (according to $R$) of the actual human policy, and $V^{\ast }\left(R|A\right)$ is the expected reward (according to $R$) of the human following the ideal policy for maximising $R$, then a measure of how much the AI is overriding rewards is the regret:

$V^{\ast }\left(R|A\right)-V^{\pi }\left(R|A\right)$.

One might object that this isn’t the AI overriding the reward, but reducing human rationality. But these two facets are related: $\pi |A$ is poorly fitted for maximising $R$, but there’s certainly another reward $R^{\prime }$ which $\pi |A$ is better suited to maximise. So the AI is forcing the human into maximising a different reward.

There’s also the issue that humans are poorly rational to start off with, so we have large regret for AIs that don’t do anything; but this makes sense. An AI that established our reward $R$ and didn’t intervene as we flailed and failed to maximise it, wouldn’t be a success in its role.

(An alternate, but related, measure of whether people’s reward is being overridden is whether, conditional on $A$, $m\left(R\right)$ is ‘sensitive’ to the reward $R$. A merely incompetent human would have $m\left(R\right)$ changing a lot dependent on $R$ - though never maximising it very well - while one with reward overridden would have the same behaviour whatever $R$ it was supposedly supposed to maximise).

Back to the example above. The $(m_r,R_t)$ pair means that the human is rationally maximising the twisted reward $R_t$. The $(m_f,R_t)$ is one where the injection forces the human into a very specific behaviour - specific behaviour that coincidentally is exactly the right thing for their reward. Finally, $(m_f,R_e)$ claims that the injection forces the human into specific behaviour that is detrimental to their reward. In the first two cases, the AI’s recommended action is $I$ (expected reward $1$ versus $1/2$ for $\neg I$), in the second it’s $\neg I$ (expected reward $1/2$ versus $0$ for $I$).

(Of course, it’s also possible to model humans are opiode-maximisers, whose rationality is overridden by not getting heroin injections; as already stated, rewards and rationality cannot be deduced from observations alone).

Hence the concept of overriding human preferences appears naturally and continuously within the formalism of rationality modules.

0 comments

Comments sorted by top scores.