Humans aren't fitness maximizers

I disagree humans don't optimize IGF:

1. We seem to have different observational data. I do know some people who make all their major life decisions based on quality and quantity of offspring. Most of them are female but this might be a bias in my sample. Specifically, quality trades off against quantity: waiting to find a fitter partner and thus losing part of your reproductive window is a common trade off. Similarly, making sure your children have much better lives than you by making sure your own material circumstances (or health!) are better is another. To be fair, they seem to be a small minority currently but I think that is due to point 3 and would be rectified in more a constant environment.
2. A lot of our drives do indirectly help IGF. Your aestethic sense may be somewhat wired to your ability to recognize and enjoy the visual appearance of healthy mates. Similarly for healthy environments to grow up in, etc. Sure, it gets hijacked for 20 other things, but how big is the loss in IGF to keep it around? I would argue it's generally not an issue for the subsection of humans that are directly driven to have big families.
3. Many of us have badly optimized drives cause our environments have changed too fast. It will take a few generations of constant environment (not gonna happen at our current level of technological progress) to catch up. The obvious example is birth control: sex drive used to actually be a great proxy signal to optimize on offspring. Now it's no longer but we still love sex. But in a few generations the only people alive are the descendants of people who wanted kids no matter their sex drive. 'evolution' will now select directly on desire for kids but it takes awhile to catch up.

I'm not saying evolution optimized us very well, but I don't think it's accurate to say that we are not IGF maximizers. The environment has just changed much too quickly and selection pressure has been low the last few generations, but things like birth control actually introduce a new selection pressure on drive to reproduce. Humans are mediocre IGF maximizers in an environment that is changing unusually fast.

I think that analogies to evolution tell us very little about how an AGI’s value formation process would work. Biological evolution is a very different sort of optimizer than SGD, and there are evolution-specific details that entirely explain our misalignment wrt inclusive genetic fitness. See point 5 in the post linked below for details, but tl;dr: 

Evolution can only optimize over our learning process and reward circuitry, not directly over our values or cognition. Moreover, robust alignment to IGF requires that you even have a concept of IGF in the first place. Ancestral humans never developed such a concept, so it was never useful for evolution to select for reward circuitry that would cause humans to form values around the IGF concept. 

It would be an enormous coincidence if the reward circuitry that lead us to form values around those IGF-promoting concepts that are learnable in the ancestral environment were to also lead us to form values around IGF itself once it became learnable in the modern environment, despite the reward circuitry not having been optimized for that purpose at all. That would be like successfully directing a plane to land at a particular airport while only being able to influence the geometry of the plane's fuselage at takeoff, without even knowing where to find the airport in question.

SGD is different in that it directly optimizes over values / cognition, and that AIs will presumably have a conception of human values during training.
 

Additionally, on most dimensions of comparison, humans seem like the more relevant analogy [LW · GW], even ignoring the fact that we will literally train our AIs to imitate humans. 

This completely misses the point, for a simple reason: humans (uniquely among Earth lifeforms) are subject not only to genetic/epigenetic evolution but also to memetic evolution.  In fact, these two evolutionary levels are tightly coupled, as evidenced by the very good match between phylogenetic trees of human populations and of languages.

It makes no sense to talk about human IGF, any definition excluding memetic component is meaningless.   Now, if you look at IGMF optimization a lot of human behavior starts making a lot more sense.  (It is also worth pointing out that memetic evolution is much faster, so it is probably the driving factor, way more important than genetic.  It is also structurally different, more resembling evolution in bacterial colonies - with organisms swapping genes and furiously hybridising - than Darwinian competition based on IGF.)

But, do you really fundamentally care that your kids have genomes?

Seems not relevant? I think we're running into an under-definition of IGF (and the fact that it doesn't actually have a utility function, even over local mutations on a fixed genotype). Does IGF have to involve genomes, or just information patterns as written in nucleotides or in binary? The "outer objective" of IGF suffers a classic identifiability issue common to many "outer objectives", where the ancestral "training signal" history is fully compatible with "IGF just for genomes" and also "IGF for all relevant information patterns made of components of your current pattern." 

(After reading more, you later seem to acknowledge this point -- that evolution wasn't "shouting" anything about genomes in particular. But then why raise this point earlier?)

Now, there's a reasonable counterargument to this point, which is that there's no psychologically-small tweak to human psychology that dramatically increases that human's IGF. (We'd expect evolution to have gathered that low-hanging fruit.)

I don't know if I disagree, it depends what you mean here. If "psychologically small" is "small" in a metric of direct tweaks to high-level cognitive properties (like propensity to cheat given abstract knowledge of resources X and mating opportunities Y), then I think that isn't true. By information inaccessibility [LW · GW], I think that evolution can't optimize directly over high-level cognitive properties. 

Optima often occur at extremes, and concepts tend to differ pretty widely at the extremes, etc. When the AI gets out of the training regime and starts really optimizing, then any mismatch between its ends and our values are likely to get exaggerated.

This kind of argument seems sketchy to me. Doesn't it prove too much? Suppose there's a copy of me which also values coffee to the tune of $40/month and reflectively endorses that value at that strength. Are my copy and I now pairwise misaligned in any future where one of us "gets out of the training regime and starts really optimizing"? (ETA: that is, significantly more pairwise misaligned than I would be with an exact copy of myself in such a situation. For more selfish people, I imagine this prompt would produce misalignment due to some desires like coffee/sex being first-person.)

And all this is to say nothing about how humans' values are much more complex and fragile than IGF, and thus much trickier to transmit

Complexity is probably relative to the learning process and inductive biases in question. While any given set of values will be difficult to transmit in full (which is perhaps your point), the fact that humans did end up with their values shows evidence that human values are the kind of thing which can be transmitted/formed easily in at least one architecture. 

My impression is that sperm banks pay donors, rather than the reverse. This is an extremely blatant and near-universal non-IGF-maxing situation, it's not subtle. (Maybe they're paying for specifically high-yield donors, but still, what fraction of men have even taken the trouble to get their sperm tested?)

I agree. I have the sense that there is some depth to the cognitive machinery that leaves people so susceptible to this particular pattern of thinking, namely: no, I am an X optimizer, for some X for which we have some (often weak) theoretical reason to claim we might be an X optimizer. Once someone decides to view themselves as an X optimizer, it can be very difficult to convince them to pay enough attention to their own direct experience to notice that they are not an X optimizer.

More disturbingly, it seems as if people can go some distance to actually making themselves into an X optimizer. For example, a lot of people start out in young adulthood incorrectly believing that what they ultimately value is money, and then, by the end of their life, they have shifted their behavior so it looks more and more like what they really value is, ultimately, money. Nobody goes even close to all the way there -- not really -- but a mistaken view that one is truly optimizing for X really can shift things in the direction of making it true.

It's as if we have too many degrees of freedom in how to explain our externally visible behavior in terms of values, so for most any X, if we really want to explain our visible behavior as resulting from really truly valuing X then we can, and then we can make that true.

But, do you really fundamentally care that your kids have genomes

yes, definitely care that they have all of the dynamical behaviors currently uniquely encoded in the genome, minus the ones that would harm their inclusive [genetic+memetic] durability-fitness. I have thought about this a lot at the prompting of folks like yourself; I at this point reasonably must argue you have convinced me that I must cooperate with IGF! AND, I think you are modeling IGF wrong! the goal is not to maximize the number of kids, the goal is to maximize number of genes in the arbitrarily far future that are shared by a significant fraction of organisms. shorttermist IGF and longtermist IGF seem to me to be very different things. by editing my preferences about art arbitrarily, you are killing genes in a way that makes the aggregate IGF of my genome go down. yeah, that seems bad from a longtermist IGF perspective - I can now be confident that ten billion years in the future, a phenotype which has a more appropriate representation of curiosity about complexity will have outcompeted the modified, lower-adaptability trough you just pushed me into! you made local modifications, but under the long eye of evolution, your changes are near guaranteed to be detrimental. I don't see how this thought experiment could conclude rejecting an attempt to change my genetics+memetics could be anything but a loss to IGF.

It's not that I don't buy your argument in any form, it's that it looks to me like you should have proved to yourself that you should become, at least a little, an IGF optimizer. if you don't recognize that your genes demand things of you that they are too slow to ask nicely about, perhaps your implementation of discovering agency has failed? if you do notice, you could do your best to optimize for those things in ways that cooperate with all instances of things wanting to optimize for whatever it is your genes want. so, perhaps, could you attempt to offer humans immortality as biological agents, thereby massively increasing their IGF?

The claim that we aren't literally optimizing IGF is hopefully uncontroversial

it really is not uncontroversial. we aren't doing local causal optimization, but it seems to me that our neural implementation grows from genetic shards that, while a bit messy, do encode a messy but very very strong IGF maximizer. there's always a point that any learned optimizer breaks down, and we don't optimize perfectly; but I just don't see why this divergence could be argued to be significant. the presence of adversarial examples doesn't make us less of a strong learned heuristic map of the thoughts of a limited-strength IGF optimizer. almost all humans keep alignment with this, optimizing quite strongly for the survival of genes strongly similar to them.

perhaps the problem is you're using individualist IGF rather than FDT IGF? I'm not sure. after all, "I" am only part of my genetic cooperation group - I am an information pattern, a synchrony in the behavior of chemicals, and that synchrony implements an enormous cooperation group between cells. over time, my epigenetic and memetic information quality degrades, and I currently must repair myself by merging with another human and spitting out a new one, who will absorb some of our genetics and some of our memetics, learning new and improved representations that have increased time-local IGF for the context the offspring find themselves in. if I could adapt myself fast enough, I'd have no need of this process; immortality is an attempt at drastically increasing longtermist IGF, and spreading that immortality to all allows me to cooperate with longtermist IGF of everyone around me.

have you considered becoming an I[GM]F maximizer? may the universe not forget any of our agency...

Person A: Why are we going to create horribly misaligned artificial intelligence?

Person B: Because the AI will end up optimizing hard on subtly askew values that don't approximate well out of distribution, like humans did relative to IGF.

Person A (who is attacking the weak analogy instead of something more substantive): But humans seem to be doing pretty well at that inadvertently? Humans got a lot better at optimizing things and now there are eight billion humans. Yes, we, don't maximize IGF as much as we would if we explicitly optimized for that, but, if we become a nice spacefaring civilization, I expect there will be many orders of magnitude more bioconservatives still living in basically human bodies running on something like DNA.

Person B: Well, that's not going to hold, because we're going to misalign artificial intelligence, and that will kill everybody.

FWIW the difference between "we get 10^-18x the utility we would've gotten if our AI had been a human satisfaction maximizer" and "we get zero utility" is basically nothing. But it's not literally nothing.

Would you give up your enjoyment of visual stimuli then, like an actual IGF optimizer would?

Answering this question yes is generally negative for inclusive genetic fitness, in the current cultural environment. So the IGF-optimization hypothesis predicts that most people will answer NO.

This is a specific example of the general truth that you can't find out what an intelligence is optimizing for by asking it what it would do in counterfactuals.

Not only do humans not directly care about increasing IGF, the vast majority does hardly even care about the proxy of maximizing the number of their direct offspring. That's something natural selection could have optimized for, but mostly didn't. Most couples in first world countries could have more than five children, yet they have less than 1.5 on average, far below replacement. The fact that this happens in pretty much all developed countries, despite politicians' effort to counteract this trend, shows how weak the preference for offspring really is.

It also seems that particularly men hardly care about having children, even though few are directly against it when their wives want them. And women, especially educated women, largely lose their desire for children as they go to work, particularly full-time. That's at least something which poorer and past societies suggest.

One theory to explain this is the theory of female opportunity cost. Women in modern society, especially educated ones, perceive having children as a large opportunity cost, since the alternative to rearing children is having a career. Women in the past and in current poorer countries lived in more "patriarchal" societies where women pursuing a career was not a social norm, and thus pursuing a career was not perceived by most women as a live option, i.e. not an alternative to having children. Thus their perceived opportunity of having children was much lower than for women in non-patriarchal societies.

In any case, any explanation of this kind must assume that women's innate desire for children is so weak that it is easily outweighed by a desire for a career.

This is all to say: Most people are even more misaligned relative to IGF than one may realize.

I think this post suffers pretty badly from Typical Mind Fallacy. This thinking isn't alien to me. I used to think exactly like this 8 years ago, but since marriage and kid I now disagree with basically every point.

One claim that is hopefully uncontroversial: Humans are not literally optimizing for IGF,

I think this is controverisial because it's basically wrong :)

First, its not actually obvious what "definition" of IGF you are using. If you talk about animals, the definition that might fit is "number of genes in the next generation". However if you talk about humans, we care about both "number of genes in the next generation" and "resources given to the children". Humans can see "one step ahead" and know the rough prospects their children have in the dating market. "Resources" is not just money, it is also knowledge, beauty, etc. 

Given this, if someone decides to have two children instead of four, this might just mean they simply don't trust their ability to equip the kids with the necessary tools to succeed. 

Now, different people ALSO have different weights for the quantity vs quality of offspring. See Shoshannah Tekofsky's comment (unfortunately disagreed with) for the female perspective on this. Evolutionary theory might predict that males are more prone to maximize quantity and satisfice quality and female are more prone to satisfice quantity and maximize quality. That is, "optimization" is not the same as "maximization". There can also be satisfice / maximization mixes where each additional unit of quality or quantity still has value, but it falls off. 

Would you give up your enjoyment of visual stimuli then, like an actual IGF optimizer would?

If you give a choice between having 10 extra kids with my current wife painlessly + sufficient resources for a good head start for them, I would consider giving up my enjoyment of visual stimuli. The only hesitation is that i don't like "weird hypotheticals" in general and i potentially expect "human preference architectures" to not be as easily "modularizable" compared to computer architectures. This giving up can also have all sorts of negative effects beyond losing "qualia" of visualness, like losing capacity for spacial reasoning. However, if the "only" thing i lose is qualia and not any cognitive capacities, than this is an easy choice. 

But, do you really fundamentally care that your kids have genomes?

Yes, obviously i do. I don't consider "genomeless people" to be a thing, i dislike  genetic engineering and over-cyborgization, i don't think uploads are even possible.  [LW · GW]

Or, an even sharper proposal: how would you like to be killed right now, and in exchange you'll be replaced by an entity that uses the same atoms to optimize as hard as those atoms can optimize, for the inclusive genetic fitness of your particular genes. Does this sound like practically the best offer that anyone could ever make you? Or does it sound abhorrent?

This hypothetical is too abstract to be answerable, but if i were to offer an answer to a hypothetical with a similar vibe: many people do in fact die for potential benefits to inclusive fitness for their families, we call those soldiers / warriors / heroes. Now, sometimes their government deceives them about whether or not their sacrifice is in fact helpful for their nation, however the underlying psychology seems be easily consistent with "IGF-optimization" 

My point today is that the observation “humans care about their kids” is not in tension with the observation “we aren't IGF maximizers”,

I think this is where the difference between the terms "optimizer" and "maximizer" is important. Also important to understand what sort of constraints most people in fact operate under. Most people seem to  they act AS IF they are IGF satisficers - they get up to a certain level of quantity / quality and seem to slow down after that. However, it's hard to infer the exact values because very specific subconscious /conscious beliefs could be influencing the strategy. 

For example, i could argue that secretly, many people want to be maximizer, however this thing we call civilization is effectively an agreement between maximizers to forgoe certain maximization tactics and stick to being a satisficers. So people might avoid "overly agressive" maximization because they are correctly worried this is perceived as "defection" and ends up backfiring. Given that the current environment is very different from the ancestral environment, this particular machinery might be malfunctioning and leading to people subconsciously perceive having any children as defection. However i suspect humanity will adapt in a small number of generations.

Humans are not literally optimizing for IGF, and regularly trade other values off against IGF.

Sort of true. The main value people seem to trade off is "physical pain." Humans are also resource and computation constrained and implementing "proper maximization" in a heavily resources constrained computation may not even be possible. 

Introspecting my thought before and after kids, I have a theory that the process of finding a mate prior to "settling down" tends to block certain introspection into one's motivations. It's easier to appreciate art if you are not thinking "oh i am looking at art i like because art provides baysean evidence on lifestyle choices to potential mates". Thinking this way can appear low status which is itself a bad sign. So the brain is more prone to lying to itself that "there is enjoyment for it's own sake." After having a kid, the mental "block" is lifted and it is sort of obvious this is what i was doing and why.