The curious case of Pretty Good human inner/outer alignment

I also don't think this solution carries over very well to powerful AIs. A powerful AI has exceptionally little reason to treat its actions as correlated with ours, and will not have grown up with us in an evolutionary environment.

This seems correct, but I think that's also somewhat orthogonal to the point that I read the OP to be making. I read it to be saying something like "some alignment discussions suggest that capabilities may generalize more than alignment [LW · GW], so that when an AI becomes drastically more capable, this will make it unaligned with its original goals; however, humans seem to remain pretty well aligned with their original goals despite a significant increase in their capabilities, so maybe we could use whatever-thing-keeps-humans-aligned-with-their-original-goals to build AIs in such a way that also keeps them aligned with their original goals when their capabilities increase".

So I think the question that the post is asking is not "why did we originally evolve niceness" (the question that your comment answers) but "why have we retained our niceness despite the increase in our capabilities, and what would we need to do for an AI to similarly retain its original goals as it underwent an increase in capabilities".

There must have been some reason(s) why organisms exhibiting niceness were selected for during our evolution, and this sounds like a plausible factor in producing that selection. However, evolution did not directly configure our values. Rather, it configured our (individually slightly different) learning processes. Each human’s learning process then builds their different values based on how the human’s learning process interacts with that human’s environment and experiences.

As this post notes, the human learning process (somewhat) consistently converges to niceness. Evolution might have had some weird, inhuman reason for configuring a learning process to converge to niceness, but it still built such a learning process.

It therefore seems very worthwhile to understand what part of the human learning process allows for niceness to emerge in humans. We may not be able to replicate the selection pressures that caused evolution to build a niceness-producing learning process, but it’s not clear we need to. We still have an example of such a learning process to study. The Wright brothers learned to fly by studying birds, not by re-evolving them!

The parent comment currently stands at positive karma and negative agreement, but the comments on it seem to be saying "what you are saying is true but not exactly relevant or not the most important thing" -- which would seem to suggest the comment should have negative or low karma but positive agreement instead.

On this evidence, I suspect voters and commenters may have different ideas; any voters want to express the reasons for their votes?

As Quintin wrote, you aren't describing a mechanistic explanation for our niceness. You're describing a candidate reason why evolution selected for the mechanisms which do, in fact, end up producing niceness in humans. 

Genes being concentrated geographically is a fascinating idea, thanks for the book recommendation, I'll definitely have a look.

Niceness does seem like the easiest to explain with our current frameworks, and it makes me think about whether there is scope to train agents in shared environments where they are forced to play iterated games with either other artificial agents or us. Unless an AI can take immediate decisive action, as in a fast take-off scenario, it will, at least for a while, need to play repeated games. This does seem to be covered under the idea that powerful AI would be deceptive, and pretend to play nice until it didn't have to, but somehow our evolutionary environment led to the evolution of actual care for others' wellbeing rather than only very sophisticated long-term deception abilities.

I remember reading about how we evolved emotional reactions that are purposefully hard to fake, such as crying, in a sort of arms race against deception, I believe it's in How the Mind Works. This reminds me somewhat of that, where areas where people have genuine care for each other's well beings are more likely to propagate the genes concentrated there.

I completely agree that our behaviour doesn't maximise the outer goal. My mysteriously capitalised "Pretty Good" was intended to point in this direction - that I find it interesting that we still have some kids, even when we could have none and still have sex and do other fun things. Declining populations would also point to worse alignment. I would consider proper bad alignment to be no kids at all, or the destruction of the planet and our human race along with it, although my phrasing, and thinking on this, is quite vague.

There is an element of unsustainability in your strategy for max gene spreading, where if everyone was constantly doing everything they could to try to spread their genes as much as possible, in the ways you describe, humanity as a whole might not survive, spreading no genes at all. But, even if it would unsustainable for everyone to do the things you described, a few more people could do it, spread their genes far and wide and society would keep ticking along. Or everyone could have just a few more children and things would probably be fine in the long term. I would say that men getting very little satisfaction from sperm donation is a case of misalignment - a deep mismatch between our "training" ancestral environments and our "deployment" modern world.

So I agree we don't maximise the outer goal, especially now that know how not to. One of the things that made me curious about this whole thing is that this characteristic, some sort of robust goal following without maximising, seems like something we would desire in artificial agents. Reading through all these comments is crystallising in my head what my questions on this topic actually are:

1. Is this robust non-maximalness an emerging quality of some or all very smart agents? - I doubt it, but it would be nice as it would reduce the chances that we get turned into paperclips.
2. Do we know how to create agents that exhibit these characteristics I think are positive? - I doubt it, but might be worth figuring out. An AGI that follows their goals only some sustainable, reasonable, amount seems safer than the AGI equivalent of the habitual sperm donor.

I narrowly agree that evolution failed to align us well with inclusive genetic fitness. 

However, your comment indicates to me that you missed OP's more important points. I think humans have some pretty interesting alignment properties (e.g. blind people presumably lose access to a range of visually-activated hardcoded reward circuitry, and yet are not AFAICT less likely to care about other human beings; thus, human value formation is robust along some kinds of variation on the internal reward function; is value really that fragile after all?). Your comment focuses on evolution/human misalignment, as opposed to genome->human alignment properties (e.g. how sensitive are learned human values to mutations and modifications to the learning process, or how the genome actually mechanistically makes people care about other people). 

in favor of various inner-rewards (often of trivial magnitude)

Inner-rewards as in "the reward meted out by the human reward system"? If so, I don't think that's how people work. Otherwise, they would be wireheaders: We know how to wirehead humans; neuroscientists do not wirehead themselves, even though some probably could have it arranged; people are not inner-reward maximizers. 

just doing it freelance over Facebook anywhere.

By my impression, this is risky; you might be forced to pay child support.

Yes, that's exactly the direction this line of thought is pulling me in! Although perhaps I am less certain we can copy the mechanics of the brain, and more keen on looking at the environments that led to human intelligence developing the way it did, and whether we can do the same with AI.

I suspect (but can't prove) that most people would not upload themselves to non-biological substrate if given the choice - only 27% of philosophers[1] believe that uploading your brain would mean that you survive on the non-biological substrate. I also suspect that people would not engineer the desire to have kids out of themselves. If most people want to have kids, I don't think we can assume that they would change that desire, a bit like we don't expect very powerful AGIs to allow themselves to be modified. The closest I can think of right now would be that I could take drugs that would completely kill my sex drive, and almost no one would do that willingly, although that probably has other horrible side-effects.

If humans turn out to be misaligned in that way - we modify ourselves completely out of alignment with "evolution's wishes" - that would tell us something about the alignment of intelligent systems, but I think so far people have shown no willingness to do that sort of thing.

[1]https://survey2020.philpeople.org/survey/results/5094

It acts as though the "outer" alignment of humans is "wanting to have kids," or something

That seems correct to me; to the extent that we can talk about evolution optimizing a species for something, I think it makes the most sense to talk about it as optimizing for the specific traits under selection. When the air in Manchester got more polluted and a dark color started conferring an advantage in hiding, dark-colored moths became more common; this obviously isn't an instance of an inner optimizer within the moth's brain since there's no behavioral element involved, it's just a physical change to the moth's color. It's just evolution directly selecting for a trait that had become useful in Manchester.

Likewise, if "wanting to have kids" is useful for having more surviving descendants, then "wanting to have more kids" becomes a trait that evolution is selecting for, analogously to evolution selecting for dark color in moths. There is an inner optimizer that is executing the trait that has been selected for, but it's one that's aligned with the original selection target.

I agree with you on what is the inner optimiser. I might not have been able to make myself super clear in the OP, but I see the "outer" alignment as some version "propagate our genes", and I find it curious that that outer goal produced a very robust "want to have kids" inner alignment. I did also try to make the point that the alignment isn't maximal in some way, as in yeah, we don't have 16 kids, and men don't donate to sperm banks as much as possible and other things that might maximise gene propagation, but even that I find interesting: we fulfill evolution's "outer goal" somewhat, without going into paperclip-maximiser-style propagate genes at all cost. This seems to me like something we would want out of an AGI.

Does it, though?

Especially in the more distant past, making fifteen kids of which eight survive probably often resulted in eight half-starved, unskilled, and probably more diseased offspring that didn't find viable mates, and extinguished the branch. I don't think humans are well suited to be nearer the r-strategy end of the spectrum, despite still having some propensity to do so.

In modern times it appears much more viable, and there are some cases of humans who desired and had hundreds of children, so the "inner optimizers" obviously aren't preventing it. Would a strong desire for everyone to have dozens or hundreds of children in times of plenty benefit their survival, or not? I don't think think this is a straightforward question to answer, and therefore it's not clear how closely our inner optimizer goals match the outer.

I don't think people have shown any willingness to modify themselves anywhere close to that extent. Most people believe mind uploading would be equal to death (I've only found a survey of philosophers [1]), so I don't see a clear path for us to abandon our biology entirely. Really the clearest path I can see is us being replaced by AI in mostly unpleasant ways, but I wouldn't exactly call that humanity at that point.

I'd even argue that if given the choice to just pick a whole new set of genes for their kids unrelated to theirs most people would say no. A lot of people have a very robust desire to have biological children.

While I agree that our niceness evolved because it was beneficial, I do wonder why we didn't evolve the capacity for really long-term deception instead, like we fear AGIs will develop. A commenter above made a point about the geographic concentration of genes that I found very interesting and might explain this.

I reckon the question is whether can we replicate whatever made us nice in AGIs

[1] https://survey2020.philpeople.org/survey/results/5094