Evolution Solved Alignment (what sharp left turn?)

If you've read The Selfish Gene, then would you agree that under Dawkins's notion of gene-level fitness, the genes composing a gene drive have high gene-level fitness? If not, why?

Not sure what kind of argument you would accept against your metric capturing the essence of evolution.

Always a good question to ask. TekhneMakre gives a good example of a situation where the two metrics disagree in this comment [LW(p) · GW(p)]. Quoting:

Say you have a species. Say you have two genes, A and B.

Gene A has two effects:

A1. Organisms carrying gene A reproduce slightly MORE than organisms not carrying A.

A2. For every copy of A in the species, every organism in the species (carrier or not) reproduces slightly LESS than it would have if not for this copy of A.

Gene B has two effects, the reverse of A:

B1. Organisms carrying gene B reproduce slightly LESS than organisms not carrying B.

B2. For every copy of B in the species, every organism in the species (carrier or not) reproduces slightly MORE than it would have if not for this copy of B.

If the relative frequency metric captures the essence of evolution, then gene A should be successful and gene B should be unsuccessful. Conversely, the total-abundance metric you suggest implies that gene B should be successful while gene A should be unsuccessful.

So one argument that would change my mind is if your showed (eg. via simulation, or just by a convincing argument) that gene B becomes prevalent in this hypothetical scenario.

Correct. In a few handfuls of generations the population shifts in those directions. This holds as long as evolutionary timescales are the important ones, but it it not at all clear to me that this is what matters today. If the subpopulations that do this move faster than even short evolutionary timescales, then the selection pressure is light enough that they can oppose it. 

I have no problem with a world where people evolve towards wanting more children or caring less about sex itself. I think if it were easy for evolution to achieve that it would have happened a long time ago, which means I think in practice we're selecting among subpopulations for cultural factors more than biological ones. That, in turn, means those subpopulations are susceptible to outside influence... which is mostly what we've been seeing for the whole timeline of fertility dropping as nations develop economically. Some communities are more resistant because they start with stronger beliefs on this front, but none are immune. And frankly I think the only way they could be immune is by enforcing kinds of rigidity that the larger world won't want to permit, seeing as they'd be seen as abusive, especially to children. 

As far as intelligence goes: a world where the average person gets dumber while a small elite becomes smarter and more powerful and wealthier (by starting companies, inventing technologies, controlling policy making, and adopting things like life extension and genetic screening as they become available and viable) is an unstable powder keg. Eventually there's conflict. Who wins? That depends on how big the gap is. Today, the smart people would probably lose, overwhelmed by numbers and lack of coordination ability. In the future, when the boundaries between populations are stark and stable cultural divides? Well by then the smarter subpopulation has all sorts of options. Like a custom virus that targets genetic characteristics of the low-intelligence subpopulation and causes infertility. Or armies of robots to supplement their numbers. Or radically longer lives so that they at least increase in absolute numbers over time and can have more children per lifetime (just slower) if they want to (note: this would enable people who love having kids to have even more of them!). Or carefully targeted propaganda campaigns (memetic warfare) to break down the cultural wall that's sustaining the differences.

Ok so the point is that the vast vast majority of optimization power coming from {selection over variation in general} is coming more narrowly from {selection for genes that increase their relative frequency in the gene pool} and not from {selection between different species / other large groups}. In arguments about misalignment, evolution refers to {selection for genes that increase their relative frequency in the gene pool}.

If you run a big search process, and then pick a really extreme actual outcome X of the search process, and then go back and say "okay, the search process was all along a search for X", then yeah, there's no such thing as misalignment. But there's still such a thing as a search process visibly searching for Y and getting some extreme and non-Y-ish outcome, and {selection for genes that increase their relative frequency in the gene pool} is an example.

The analogy from historical evolution is the misalignment between human genes and human minds, where the rise of the latter did not result in extinction of the former. It plausibly could have, but that is not what we observe.

The analogy is that the human genes thing produces a thing (human minds) which wants stuff, but the stuff it wants is different from what what the human genes want. From my perspective you're strawmanning and failing to track the discourse here to a sufficient degree that I'm bowing out.

I'm saying that the fact that you, an organism built by the evolutionary process, hope to step outside the evolutionary process and do stuff that the evolutionary process wouldn't do, is misalignment with the evolutionary process.

Obviously true but also wierd that anyone would think that's relevant? Even for a population which is perfectly aligned in expectation, you wouldn't expect any individual to be perfectly aligned.

Some people think that misalignment in RL is fundamentally impossible. I’m pretty sure you’re not in that category. But for such people, it’s nice to have an example of “RL algorithm with reward function R → at least one trained model that scores catastrophically poorly on R as soon as it goes out of distribution”.

The CoinRun thing could also work in this context, but is subject to the (misguided) rebuttal “oh but that’s just because the model’s not smart enough to understand R”. So human examples can also be useful in the course of such conversations.

For any reasonable choice of a genetic utility function …

Also what if all the DNA information still exists, but stored more compactly on digital computers, which then in simulation replicate enormously, and the simulations themselves expand as our compute spreads a bit into space and advances further..

Here are a bunch of claims that are equally correct:

• The history of evolution to date is exactly what you’d get from an RL algorithm optimizing genes for how many future copies are encoded in literal DNA molecules in basement reality.
• The history of evolution to date is exactly what you’d get from an RL algorithm optimizing genes for how many future copies are encoded in literal DNA molecules in either basement reality or accurate simulations.
• The history of evolution to date is exactly what you’d get from an RL algorithm optimizing genes for how many future copies are encoded in DNA molecules, or any other format that resembles it functionally (regardless of whether it resembles it chemically or mechanistically).
• [infinitely many more things like that]

(Related to “goal misgeneralization”). If humans switch from DNA to XNA or upload themselves into simulations or whatever, then the future would be high-reward according to some of those RL algorithms and the future would be zero-reward according to others of those RL algorithms. In other words, one “experiment” is simultaneously providing evidence about what the results look like for infinitely many different RL algorithms. Lucky us.

You’re welcome to argue that the evidence from some of these “experiments” is more relevant to AI alignment than the evidence from others of these experiments. (I figure that’s what you mean when you say “for any reasonable choice of a genetic utility function…”?) But to make that argument, I think you’d have to start talking more specifically about what you expect AI alignment to look like and thus why some of those bullet point RL algorithms are more disanalagous to AI alignment than others. You seem not to be doing that, unless I missed it, instead you seem to be just talking intuitively about what’s “reasonable” (not what’s a tight analogy).

(Again, my own position is that all of those bullet points are sufficiently disanalogous to AI alignment that it’s not really a useful example, with the possible exception of arguing with someone who takes the extreme position that misalignment is impossible in RL as discussed at the top of this comment.)

↑ comment by jacob_cannell · 2023-10-15T01:26:38.806Z · LW(p) · GW(p)

Some people think that misalignment in RL is fundamentally impossible. . for such people, it’s nice to have an example of “RL algorithm with reward function R → at least one trained model that scores catastrophically poorly on R as soon as it goes out of distribution”.

I don't actually believe that, but I also suspect that for those who do the example isn't relevant because evolution is not a RL algorithm.

Evolution is an outer optimizer that uses RL as the inner optimizer. Evolution proceeds by performing many experiments in parallel, all of which are imperfect/flawed, and the flaws are essential to the algorithms's progress. The individual inner RL optimizer can't be perfectly aligned as if they were the outer optimizer wouldn't have actually worked in the first place, and the outer optimizer can be perfectly aligned even though every single instance of the inner optimizer is at least somewhat misaligned. This is perhaps less commonly understood than I initially assumed - but true nonetheless.

Technology evolves according to an evolutionary process that is more similar/related to genetic evolution (but with greatly improved update operators). The analogy is between genetic evolution over RL brains <-> tech evolution over RL AGIs.

If humans switch from DNA to XNA or upload themselves into simulations or whatever, then the future would be high-reward according to some of those RL algorithms and the future would be zero-reward according to others of those RL algorithms. .. You’re welcome to argue that the evidence from some of these “experiments” is more relevant to AI alignment than the evidence from others of these experiments.

Again none of those future scenarios have played out, they aren't evidence yet, just speculation.

Replies from: steve2152, quetzal_rainbow

↑ comment by Steven Byrnes (steve2152) · 2023-10-15T11:30:14.414Z · LW(p) · GW(p)

I made this diagram recently—I just colored some things red to flag areas where I think we’re disagreeing.

The first thing is you think “evolution is not a RL algorithm”. I guess you think it’s a learning algorithm but not a reinforcement learning algorithm? Or do you not even think it’s a learning algorithm? I’m pretty confused by your perspective. Everyone calls PPO an RL algorithm, and I think of evolution as “kinda like PPO but with a probably-much-less-efficient learning rule because it’s not properly calculating gradients”. On the other hand, I’m not sure exactly what the definition of RL is—I don’t think it’s perfectly standardized.

(We both agree that, as a learning algorithm, evolution is very weird compared to the learning algorithms that people have been programming in the past and will continue programming in the future. For example, ML runs normally involve updating the trained model every second or sub-second or whatever, not “one little trained model update step every 20 years”. In other words, I think we both agree that the right column is a much much closer analogy for future AGI training than the middle column.)

(But as weird a learning algorithm as the middle column is, it’s still a learning algorithm. So if we want to make and discuss extremely general statements that apply to every possible learning algorithm, then it’s fair game to talk about the middle column.)

The second thing is on the bottom row, I get the impression that

• you want to emphasize that the misalignment could be much higher (e.g. if nobody cared about biological children & kin),
• I want to emphasize that the misalignment could be much less (e.g. if men were paying their life savings for the privilege of being a sperm donor).

But that’s not really a disagreement, because both of those things are true. :)

Replies from: jacob_cannell

↑ comment by jacob_cannell · 2023-10-16T16:26:33.981Z · LW(p) · GW(p)

The first thing is you think “evolution is not a RL algorithm”. I guess you think it’s a learning algorithm but not a reinforcement learning algorithm?”.

To a certain extant this is taxonomy/definitions, but there is an entire literature on genetic/evolutionary optimization algorithms and it does seem odd to categorize them under RL - I've never heard/seen those authors categorize it that way and RL is a somewhat distant branch of ML. In my mental taxonomy (which I do believe is more standard/canonical) the genetic/evolutionary search family of algorithms are combinatoric/discrete algorithms that evaluate many candidate solutions in parallel and use separate heuristics for updating params (typically not directly related to the fitness function), and instead use fitness function evaluation to select replication factor for the next generation. They approximate the full bayesian posterior with large scale but crude sampling.

Has advantages over gradient methods for exploring highly compressed/combinatoric parameter spaces, but scales poorly to higher dimensions as it doesn't solve fine grained credit assignment at all.

RL on the other hand is just a slight variation of UL/SL, and is usually (almost always?) a continuous/gradient method.

So anyway the useful analogy is not between genetic evolution and within-lifetime learning, it's between (genetic evolution, brain within-lifetime learning) and (tech/memetic evolution, ML training).

Replies from: steve2152

↑ comment by Steven Byrnes (steve2152) · 2023-10-16T17:13:39.539Z · LW(p) · GW(p)

Random @nostalgebraist [LW · GW] blog post: ““Reinforcement learning” (RL) is not a technique.  It’s a problem statement, i.e. a way of framing a task as an optimization problem, so you can hand it over to a mechanical optimizer. … Life itself can be described as an RL problem”

Russell & Norvig 3rd edition: “The task of reinforcement learning is to use observed rewards to learn an optimal (or nearly optimal) policy for the environment. …Reinforcement learning might be considered to encompass all of AI.” (emphasis in original)

If we take the Russell & Norvig definition, and plug in “reward = IGF” and “policy = a genome that (in the right environment) unfolds into a complete brain and body”, then evolution by natural selection is “using observed rewards to learn an optimal (or nearly optimal) policy for the environment”, thus qualifying as RL according to that definition. Right?

Obviously I agree that if we pick a paper on arxiv that has “RL” in its title, it is almost certainly not talking about evolutionary biology :-P

I’ll also reiterate that I think we both agree that, even if the center column of that chart picture above is “an RL algorithm” by definition, it’s mechanistically pretty far outside the distribution of RL algorithms that human AI programmers are using now and are likely to use in the future, whereas within-lifetime learning is a more central example in almost every way. I’m definitely not disagreeing about that.

RL on the other hand is just a slight variation of UL/SL, and is usually (almost always?) a continuous/gradient method.

One important difference between RL and SL / SSL is that in RL, after you take an action, there’s no ground truth about what action you counterfactually should have taken instead. In other words, SL / SSL gives you a full error gradient “for free” with each query, whereas RL doesn’t. (You can still get an error gradient in policy-optimization, but it’s more expensive—you need to query the policy a bunch of times to get one gradient, IIUC.) Thus, RL algorithms involve random exploration then exploitation, whereas things like ImageNet training and LLM pretraining do not involve any random exploration.

(Notice here that evolution-by-natural-selection does involve a version of explore-then-exploit: mutations are random, and are initially trialed in a small number of individual organisms, and only if they’re helpful are they deployed more and more widely, eventually across the whole species.)

Replies from: jacob_cannell

↑ comment by jacob_cannell · 2023-10-16T17:37:56.812Z · LW(p) · GW(p)

The task of reinforcement learning is to use observed rewards to learn an optimal (or nearly optimal) policy for the environment

If we define RL broadly as "learning/optimizing an algorithm/policy towards optimality according to some arbitrary reward/objective function", then sure it becomes indistinguishable from general optimization.

However to me it is specifically the notion of reward in RL that distinguishes it, as you say here:

One important difference between RL and SL / SSL is that in RL, after you take an action, there’s no ground truth about what action you counterfactually should have taken instead.

Which already makes it less general that general optimization. A reward function is inherently a proxy - designed by human engineers to approximate the more complex unknown utility, or evolved by natural selection as a practical approximate proxy for something like IGF.

Evolution by natural selection doesn't have any proxy reward function, so it lacks that distinguishing feature of RL. The 'optimization objective' of biological evolution is simply something like an emergent telic arrow of physics: replicators tend to replicate, similar to net entropy always increases, etc. When humans use evolutionary algorithms as evolutionary simulations, the fitness function is more or less approximating the fitness of a genotype if the physics was simulated out in more detail.

So anyway I think of RL as always an inner optimizer, using a proxy reward function that approximates the outer optimizer's objective function.

Replies from: steve2152

↑ comment by Steven Byrnes (steve2152) · 2023-10-16T22:08:07.456Z · LW(p) · GW(p)

I don’t think that, in order for an algorithm to be RL, its reward function must by definition be a proxy for something else more complicated. For example, the RL reward function of AlphaZero is not an approximation of a more complex thing—the reward function is just “did you win the game or not?”, and winning the game is a complete and perfect description of what DeepMind programmers wanted the algorithm to do. And everyone agrees that AlphaZero is an RL algorithm, indeed a central example. Anyway, AlphaZero would be an RL algorithm regardless of the motivations of the DeepMind programmers, right?

Replies from: jacob_cannell

↑ comment by jacob_cannell · 2023-10-16T22:49:01.926Z · LW(p) · GW(p)

True in games the environment itself can directly provide a reward channel, such that the perfect 'proxy' simplifies to the trivial identity connection on that channel. But that's hardly an interesting case right? A human ultimately designed the reward channel for that engineered environment, often as a proxy for some human concept.

The types of games/sims that are actually interesting for AGI, or even just general robots or self driving cars, are open ended where designing the correct reward function (as a proxy for true utility) is much of the challenge.

↑ comment by quetzal_rainbow · 2023-10-15T21:45:09.328Z · LW(p) · GW(p)

Again none of those future scenarios have played out, they aren't evidence yet, just speculation

It's very weird notion of what constitutes an evidence. If you built AGI and your interpretability tools show you that AGI is plotting to kill you, it would be pretty hard evidence in favor of sharp left turn, even if you were still alive.

Replies from: jacob_cannell

↑ comment by jacob_cannell · 2023-10-15T22:30:53.067Z · LW(p) · GW(p)

Not at all - just look at the definition of solomonoff induction: the distribution over world models/theories is updated strictly on new historical observation bits, and never on future predicted observations. If you observe mental states inside the AGI, that is naturally valid observational evidence from your pov. But that is very different from you predicting that the AGI is going to kill you, and then updating your world model based on those internal predictions - those feedback loops rapidly diverge from reality (and may be related to schizophrenia).

Replies from: quetzal_rainbow

↑ comment by quetzal_rainbow · 2023-10-16T10:53:39.240Z · LW(p) · GW(p)

I'm talking about observable evidence, like, transhumanists claiming they will drop their biological bodies on first possibility.

The broader laws of natural selection and survival of the fittest

These are distinct from biological evolution, so if our descendants end up being optimized over by them rather than by biological evolution, that feels like it's conceding the argument (that we'll have achieved escape velocity).

Evolution has succeeded at aligning homo sapiens brains to date

I'm guessing we agree on the following:

• Evolution shaped humans to have various context-dependent drives (call them Shards) and the ability to mentally represent and pursue complex goals. Those Shards were good proxies for IGF in the EEA^[1].

• Those Shards were also good^[2] enough to produce billions of humans in the modern environment. However, it is also the case that most modern humans spend at least part of their optimization power on things orthogonal to IGF.

I think our disagreement here maybe boils down to approximately the following question:

With what probability are we in each of the following worlds?

• (World A) The Shards only work^[2:1] conditional on the environment being sufficiently similar to the EEA, and humans not having too much optimization power. If the environment changes too far OOD, or if humans were to gain a lot of power^[3], then the Shards would cease to be good^[2:2] proxies.

  In this world, we should expect the future to contain only a small fraction^[4] of the "value" it would have, if humanity were fully "aligned"^[2:3]. I.e. Evolution failed to "(robustly) align humanity".

• (World B) The Shards (in combination with other structures in human DNA/brains) are in fact sufficiently robust that they will keep humanity aligned^[2:4] even in the face of distributional shift and humans gaining vast optimization power.

  In this world, we should expect the future to contain a large fraction of the "value" it would have, if humanity were fully "aligned"^[2:5]. I.e. Evolution succeeded in "(robustly) aligning humanity".

• (World C) Something else?

I think we're probably in (A), and IIUC, you think we're most likely in (B). Do you consider this an adequate characterization?

If yes, the obvious next question would be: What tests could we run, what observations could we make,^[5] that would help us discern whether we're in (A) or (B) (or (C))?

(For example: I think the kinds of observations I listed in my previous comment are moderate-to-strong evidence for (A); and the existence of some explicit-IGF-maximizing humans is weak evidence for (B).)

vast computation some of which is applied to ancestral simulations

I agree that a successful post-human world would probably involve a large amount^[1] of resources spent on simulating (or physically instantiating) things like humans engaging in play, sex, adventure, violence, etc. IOW, engaging in the things for which Evolution installed Shards in us. However, I think that is not the same as [whatever Evolution would care about, if Evolution could care about anything]. For the post-human future to be a success from Evolution's perspective, I think it would have to be full of something more like [programs (sentient or not, DNA or digital) striving to make as many copies of themselves as possible].

(If we make the notion of "DNA" too broad/vague, then we could interpret almost any future outcome as "success for Evolution".)

The natural consequence of "postbiological humans" is effective disempowerment if not extinction of humanity as a whole.

Such "transhumanists" clearly do not find the eradication of biology abhorrent, any more than any normal person would find the idea of "substrate independence"(death of all love and life) to be abhorrent.

Hmm... Reading your last comment again, it seems you make some point similar to Robin Hanson's Poor Folks Do Smile. An argument that evolution optimizes to some limited extent for happiness or satisfied preferences, to a degree significantly above neutral welfare lives. Add to that Hanson's commitment to total (rather than average) utilitarianism, evolution is pretty aligned with ethics, as it optimizes for a high number of positive welfare individuals. I'm not sure how his argument is supposed to generalize to a world with AI though. Presumably we humans don't want to be replaced with even fitter AI, while evolutionary pressures arguably do "want" this. And total utilitarianism (the Repugnant Conclusion) seems unreasonable to me anyway.

But apart from your similarity to Hanson: I think the goal of AI alignment really is ethics. Not some specific human values shaped by millennia of biological or even cultural evolution. First, I don't think cultural evolution influences our terminal values. And while biological evolution does indeed shape terminal values, it operates fundamentally on the level of individual agents. But ethics is about aggregate welfare. A society of Martians may have different terminal values from us (they may not fear death after procreation, for example), but they could still come up with the same forms of utilitarianism as we do. Ethics, at least if some sort of utilitarianism is right, isn't about labelling some idiosyncratic collection of terminal values as "good" that we historically evolved to have, but rather about maximizing an aggregate (like average or sum) of terminal values (happiness or desire satisfaction) of individuas, whatever those individual values may be. I think biological evolution doesn't very strongly optimize for the total aggregate (though Hanson may disagree), and hardly at all for the average.

Though admittedly I don't fully understand your position here.