Value Learning is only Asymptotically Safe

is there anything to aim for that is stronger than asymptotic safety?

Faster convergence?

Delegative Reinforcement Learning [AF · GW] is safe not just asymptotically. See also this [AF · GW], this [AF · GW] and (once it's uploaded) upcoming paper for SafeML 2019. In addition, there are directions for further improvment here [AF · GW] in the "value learning protocols" sections.

I sort of object to titling this post "Value Learning is only Asymptotically Safe" when the actual point you make is that we don't yet have concrete optimality results for value learning other than asymptotic safety.

We should definitely be writing down sets assumptions from which we can derive formal results about the expected behavior of an agent, but is there anything to aim for that is stronger than asymptotic safety?

In the case of value learning, given the generous assumption that "we somehow figured out how to design an agent which understood what constituted observational evidence of humanity’s reflectively-endorsed utility function", it seems like you should be able to get a PAC-type bound, where by time $T$, the agent is only $ϵ$-suboptimal with probability $1-\delta (ϵ,T)$, where $\delta (ϵ,T)$ is increasing in $ϵ$ but decreasing in $T$ -- see results on PAC bounds for Bayesian learning, which I haven't actually looked at. This gives you bounds stronger than asymptotic optimality for value leraning. Sadly, if you want your agent to actually behave well in general environments, you probably won't get results better than asymptotic optimality, but if you're happy to restrict yourself to MDPs, you probably can.

I suspect there are many more sources of risk that result in only being able to approach complete safety than cosmic rays, but this seems a reasonable argument for at least establishing that the limit exists so even if we disagree over whether something more easily controlled by AI design is a source of risk we don't get confused and think that if we eliminate all risk from the design that we suddenly get perfect safety.

"In the presence of cosmic rays, then, this agent is not safe for its entire lifetime with probability 1."

I think some readers may disagree about whether you this sentence means "with probability 1, the agent is not safe" or "with probability strictly greater than 0, the agent is not safe". In particular, I think Hibron's comment is predicated on the former interpretation and I think you meant the latter.

Even granting that it is possible for cosmic rays to flip any given bit, or any sequence of bits, in a computer's memory, it is far from clear to me that the probability of this happening approaches 1 over the lifetime of the universe. It isn't very hard to come up with cases where an event is both completely possible, and has probability 0: for instance, if I pick a number at random with uniform distribution from the closed interval [0,1], the probability I will pick 1 is 0 even though 1 is as likely a choice as any other option on the interval. And in the concrete case you're referring to, the universe has finite time to flip these bits before it sinks into entropy. Moreover, I wouldn't expect the sequence of datapoints needed to convince an AI that humans are hostile (or whatever) to be invariant across time: as the AI accrued more data, it would plausibly require more data to persuade it to change its mind.