Positive values seem more robust and lasting than prohibitions

I think the key question is how much policy-caused variance in the outcome there is.

That is, juice can chain onto itself because we assume there will be a bunch of scenarios where reinforcement-triggering depends a lot on the choices made. If you are near a juice but not in front of the juice, you can walk up to the juice, which triggers reinforcement, or you can not walk up to the juice, which doesn't trigger reinforcement. The fact that these two plausible actions differ in their reinforcement is what I am referring to with "policy-caused variance".

If you are told not to kill someone, then you can usually stay sufficiently far away from killing anyone, in such a way that the variance in killing is 0, because killing itself has a constant value of 0. (Unlike juice-drinking which might have a value of 0 or 1, depending on both scenario and actions.)

But you could also have a case with a positive value that fails to be robust/lasting, if the positive value fails to have variance. One example would be if the positive value is bounded and always achieved; for instance you might imagine that if you are always carrying a juice dispenser, juice-drinking would always have a value of 1, and therefore again there wouldn't be any variance to reinforce seeking juice.

A more subtle point is that if the task is too difficult, e.g. if you are in a desert with no juice available, then the policy-caused variance is also 0, because the juice is constant 0. This is a general point that encompasses many failures of reinforcement learning. Often, if you don't do things like reward shaping, then reinforcement learning simply fails, because the task is too complex to learn.

I think future RL algorithms will be able to succeed using less policy-caused variance than present RL algorithms require by using models to track the outcomes through many layers of interactions.

This seems related to insights from parenting (and animal training), which I mentioned, e.g., in Book Review: Kazdin's The Everyday Parenting Toolkit [LW · GW]

positive reinforcement is scientifically proven to work much better then negative reinforcement (aka punishment).

If your point is true and the analogy holds, that might indicate that other "training" methods from parenting or animal training might also work.

Finally got around to reading this, only to discover that I independently noticed a similar thing just a couple days ago, see here for how I’m thinking about it [LW(p) · GW(p)]. :)

I found it useful to compare a shard that learns to pursue juice (positive value) to one that avoids eating mouldy food (prohibition), just so they're on the same kind of framing/scale.

It feels like a possible difference between prohibitions and positive values is that positive values specify a relatively small portion of the state space that is good/desirable (there are not many states in which you're drinking juice), and hence possibly only activate less frequently, or only when parts of the state space like that are accessible, whereas prohibitions specify a large part of the state space that is bad (but not so much that the complement is a small portion - there are perhaps many potential states where you eat mouldy food, but the complement of that set is still not a similar size to the set of states of drinking juice). The first feels more suited to forming longer-term plans towards the small part of the state space (cf this definition of optimisation [AF · GW]), whereas the second is less so. Then shards that start doing optimisation like this are hence more likely to become agentic/self-reflective/meta-cognitive etc.

In effect, positive values are more likely/able to self-chain because they actually (kind of, implicitly) specify optimisation goals, and hence shards can optimise them, and hence grow and improve that optimisation power, whereas prohibitions specify a much larger desirable state set, and so don't require or encourage optimisation as much.

As an implication of this, I could imagine that in most real-world settings "don't kill humans" would act as you describe, but in environments where it's very easy to accidentally kill humans, such that states where you don't kill humans are actually very rare, then the "don't kill humans" shard could chain into itself more, and hence become more sophisticated/agentic/reflective. Does that seem right to you?

Could this be resolved by wanting to not kill, rather than not wanting to kill? My understanding is that "Don't want X" acts as a filter of generated plans, while "Want to do X" (or "Want to not do X") will either generate plans, or at least sort of act as an optimization pressure. 

People are often defined by negative values in the sense of "don't do stuff that hurt" or "don't do stuff that result in lower status", although it might be better to phrase that as "actively avoid doing things that result in pain/lower status".

My initial thoughts were:

• On one hand, if you positively reinforce, the system will seek it out, if you negatively reinforce the system will work around it.
• On the other hand, there doesn't seem to be a principled difference between positive reinforcement and negative reinforcement. Like I would assume that the zero point wouldn't affect the trade-off between two actions as long as the difference was fixed.

Having thought about it a bit more, I think I managed to resolve the tension. It seems that if at least one of the actions is positive utility, then the system has a reason to maneuver you into a hypothetical state where you choose between them, while if both are negative utility then the system has a reason to actively steer you away from having to make such a choice.

(This analysis is still naive in that it doesn't account for opportunity cost).

I'd really love to see greater formalisation of this intuition. Even what I've said above is quite ambiguous.

Major stages in my own moral development...

1. Preschool: learning "if I threaten to hit people, they can refuse to play with me, which sucks, so I guess I won't do that".  Shamefully, learning this via experience.
2. Probably early elementary school: learning "if I lie about things, then people won't believe me, so I guess I won't do that."  Again via shameful experience.  Eventually, I developed this into a practically holy commandment; not sure what the external factors were.
   1. Some kind of scientific ethic?  Feynman with the "the easiest person to fool is yourself; to maintain scientific integrity, you have to bend over backwards, naming all the potential reasons you might be wrong" and stuff.
   2. A developing notion that lying was evil, that it could mess things up really badly, that good people who tried lying quickly regretted it (probably mostly fictional examples here), and that the only sensible solution was a complete prohibition.
3. Middle school: took a game theory class at a summer camp; learned about the Prisoner's Dilemma and tragedy of the commons; threats and promises; and the hawk-dove game with evolutionarily stable strategies.  This profoundly affected me:
   1. The threats-and-promises thing showed that it was sometimes rational to (visibly) put yourself into a state (perhaps with explicit contracts, perhaps with emotions) where you would do something "irrational", because that could then change someone else's behavior.
   2. With the one-shot Prisoner's Dilemma, it seemed clear that, to get the best outcome for everyone, it was necessary for everyone to have an "irrational" module in their brain that led them to cooperate.  To a decent extent one can solve real-world situations with external mechanisms that make it no longer a one-shot Prisoner's Dilemma—reputation, private ownership rights—but it's not a complete solution.
   3. In the hawk-dove game, two birds meet and there's a resource and they have the option to fight for it; we figure each bird follows a certain strategy, dictated by their genes, and winning strategies will increase in prevalence.^[1]  Lessons I took: there are multiple different equilibria, some better than others, some more abusive than others; if the population has a high enough fraction of people who will fight back against abuse beyond the point of "rationality", this will prevent abusers from dominating, and should be considered a public service.
4. At some point I encountered the "non-aggression principle", and decided that it, coupled with common notions of what counts as aggression against person and property (and therefore what counts as property, which is perhaps contentious), was an excellent Schelling point for the fundament of an ethical system.  (I will reluctantly admit that "submit to the strongest man" is also a Schelling point people sometimes go for.)

On the subject of the post:

1. The "positive, expanding" angle on things like "don't kill"—more generally, "people have rights that shouldn't be violated"—that comes to mind is: As you learn to do more cool things with your toys, imagine if someone else could come and take your toys away, or injure you.  That would be bad, wouldn't it?  And then have a belief about a connection between the rules other people follow and the rules you follow.  Some components:
   1. Have a theory of mind.
   2. Having true peers might be important—a baby (human or AI) who interacts only with the adults who control everything and are very different from them, might find it harder to see or believe in universality of rules.
2. Another angle—similar to the above, but maybe different?—is to think positively about the rights you have to your toys, and think about the things you're thereby guaranteed to be allowed to do.
   1. I guess this would be relevant after you'd had the experience of trying to play with someone else's toys and been told a firm "no, that's not yours and you don't have permission".  For an AI, training on that seems doable.

1. ^{^}

   If two hawks meet, they fight, one gets injured, and the damage from the injury exceeds the value of the resource, so the expected value is negative to both birds (the basic scenario can be considered a game of chicken); if a hawk meets a dove, the dove runs away and gets zero, and the hawk gets the resource; if two doves meet, they waste some time on symbolic combat, one of them wins, and the expected value is positive.  Evolutionarily stable strategy is some fraction of hawks, some fraction of doves.
   
   Then there are other variations.  "Bullies" show fight and scare away doves, but will run away from a real fighter (i.e. hawk), and we figure that when two bullies meet, one gets scared and runs away first.  The bully strategy dominates the dove strategy; the equilibrium is a hawk-bully composite.

   Then we introduce the "retaliator" to defeat bullies.  Retaliators act like doves, but if the other bird shows fight, they fight back.  Against hawks and bullies they act like hawks; against doves or other retaliators they act like doves.  "Pure retaliator", or "mostly retaliator, with up to some fraction of doves", is an evolutionarily stable strategy—and so is hawk-bully.  Which one you end up with depends on your starting population.

   Further variations can be considered and explored.  For example, the most "optimal" result would be one in which all the birds understood some system by which each resource belonged to one bird and not the other, so when they met, one would act like a hawk and the other like a dove, resolving the conflict instantly.  Different systems are possible: "the biggest bird", "the bird whose territory it is", "the bird who got there first", "the bird who wins on some arbitrary visible characteristic (like tail length) not necessarily related to combat" (apparently this is a thing), and so on.  If there are multiple competing systems, then the majority will tend to push out the minority.  An evolutionarily stable (or metastable) equilibrium is "followers of one dominant system, plus up to some percentage of bullies".