The AI Alignment Problem Has Already Been Solved(?) Once

It's interesting to me that you identify with S2 / the AI / the rider, and regard S1 / the monkey / the elephant as external. I suspect this is pretty common among rationalists. Personally, I identify with S1 / the monkey / the elephant, and regard S2 / the AI / the rider in exactly the way your metaphor suggests - this sort of parasite growing on top of me that's useful for some purposes, but can also act in ways I find alien and that I work to protect myself from.

I'm not sure that this is a good/accurate metaphor, but I do want to say it was great Insight Porn (I mean this in a good way). It gave a satisfying click of connection between some ideas that gave me a new perspective on it.

Still mulling it over. I'd be extremely wary of taking this as "evidence" for anything but hopefully can point in interesting directions.

(UPDATE: SEE THE POST (Brainstem, Neocortex) ≠ (Base Motivations, Honorable Motivations) [LW · GW] FOR MUCH MORE ON THIS)

I strongly agree with the framing of neocortex vs "monkey brain" (I'll call it "subcortex" instead, it's not like monkeys don't have a neocortex), e.g. my post here [AF · GW] or others.

The part I disagree with here is the framing that "willpower" is the neocortex doing its own thing, while akrasia is the subcortex overruling the neocortex and running the show.

I think all motivations come from the subcortex, both "noble" motivations and "base" motivations. The "noble" motivations are where , the thing itself might or might not be rewarding, but where the thought "myself doing that thing" is rewarding. For example maybe doing my homework is not rewarding, but it's rewarding to think "I am doing my homework / I did my homework". Conversely, for "base" motivations to do a thing, the thing itself is presumably rewarding, but the meta-thought "myself doing that thing" has negative value. For example, eating candy is rewarding, but the thought "I am eating candy" is aversive.

Then there's a conflict in the neocortex, and sometimes the neocortex will keep attention on the meta-thought, and do the "noble" thing, and other times it won't, and it will do the "base" thing. And when it does manage to do the "noble" thing, we look back fondly on that memory as the right thing to have done, by definition, since the memory is of ourselves doing the thing.

Anyway, I would say that the monkey brain / subcortex is responsible for making the thought of "doing homework" aversive and the monkey brain / subcortex is responsible for making the meta-thought of "I am doing my homework" attractive.

Reading this I'm coming away with several distinct objections that I feel make the point that AI control is hard and give no practical short term tools.

The first objection is that it seems impossible to determine, from the perspective of system 1, whether system 2 is working in a friendly way or not. In particular, it seems like you are suggesting that a friendly AI system is likely to deceive us for our own benefit. However, this makes it more difficult to distinguish "friendly" and "unfriendly" AI systems! The core problem with friendliness I think is that we do not actually know our own values. In order to design "friendly" systems we need reliable signals of friendliness that are easier to understand and measure. If your point holds and is likely to be true of AI systems, then that takes away the tool of "honesty" which is somewhat easy to understand and verify.

The second objection is that in the evolutionary case, there is a necessary slowness to the iteration process. Changes in brain architecture must be very slow and changes in culture can be faster, but not so fast that they change many times a generation. This means there's a reasonable amount of time to test many use cases and to see success and failure even of low probability events before an adaptation is adopted. While technologies are adopted gradually in the AI case, ugly edge cases commonly occur and have to be fixed post-hoc, even when the systems they are edge cases of have been reliable for years or millions of test cases in advance. The entire problem of friendliness is to be able to identify these unknown unknowns, and the core mechanism solving that in the human case seems like slow iteration speed, which is probably not viable due to competitive pressure.

Third, system 1 had essentially no active role in shaping system 2. Humans did not reflectively sit down and decide to become intelligent. In particular, that means that many of the details of this solution aren't accessible to us. We don't have a textbook written down by monkeys talking about what makes human brains different and when humans and monkeys had good and bad times together. In fact our knowledge of the human brain is extremely limited, to the point where we don't even have better ways of talking about the distinctions made in this post than saying "system 1" and "system 2" and hand-waving over the fact that these aren't really distinct processes!

Overall the impression I get is of a list of reasons that even if things seem to be going poorly without a central plan, there is a possibility that this will work out in the end. I don't think that this is bad reasoning, or even particularly unlikely. However I also don't think it's highly certain nor do I think that it's surprising. The problem of AI safety is to have concrete tools that can increase our confidence to much higher levels about the behavior of designed systems before seeing those systems work in practice on data that may be significantly different than we have access to. I'm not sure how this metaphor helps with those goals and I don't find myself adjusting my prior beliefs at all (since I've always thought there was some significant chance that things would work out okay on their own--just not a high enough chance)

I'd suggest changing the title to "Solving the AI Alignment Problem Has Been Tried Before", as you don't conclude that the cortex is necessarily a successful solution.

Also the principle here is evolution, not the monkey brain. Evolution created the prefrontal cortex (along with all the interactions you described), the monkey brain did not.

I've been aware of the conflict between the old machinery and the logos since I was very young. I get a strong sense there's something halfway sapient back there that can sort of tell when parts of the logos' volition are aligned with the old machinery. Very often, I'll find I can override an instinct only once I've undergone a negotiation process and expressed/consolidated an understanding and respect of the instinct's evolutionary purpose.

In this theory of development, as the intelligence understands more and more of their intended purpose as a component of a living thing, they gain access to more introspective powers, more self-control.

It would be interesting to hear others' opinions on this.

I enjoyed this very much. One thing I really like is that your interpretation of the evolutionary origin of Type 2 processes and their relationship with Type 1 processes seems a lot more realistic to me than what I usually see. Usually the two are made to sound very adversarial, with Type 2 processes having some kind of executive control. I've always wondered how you could actually get this setup through incremental adaptations. It doesn't seem like Azathoth's signature. I wrote something relevant to this in correspondence:

If Type 2 just popped up in the process of human evolution, and magically got control over Type 1, what are the chances that it would amount to anything but a brain defect? You'd more likely be useless in the ancestral environment if a brand new mental hierarch had spontaneously mutated into existence and was in control of parts of a mind that had been adaptive on their own for so long. It makes way more sense to me to imagine that there was a mutant who could first do algorithmic cognition, and that there were certain cues that could trigger the use of this new system, and that provided the marginal advantage. Eventually, you could use that ability to make things safe enough to use the ability even more often. And then it would almost seem like it was the Type 2 that was in charge of the Type 1, but really Type 1 was just giving you more and more leeway as things got safer.

This seems to be more about human development than AI alignment. The non-parallels between these two situations all seem very pertinent.

I like this because this helps better answer the anthropics problem of existential risk, namely that we should not expect to find ourselves in a universe that gets destroyed, and more specifically you should not find yourself personally living in a universe where the history of your experience is lost. I say this because this is evidence that we will likely avoid a failure in AI alignment that destroys us, or at least not find ourselves in a universe where AI destroys us all, because alignment will turn out to be practically easier than we expect it to be in theory. That alignment seems necessary for this still makes it a worthy pursuit since progress on the problem increases our measure, but it also fixes the problem of believing the low-probability event of finding yourself in a universe where you don't continue to exist.

Welcome to the world of Memetic Supercivilization of Intelligence... living on top of the humanimal substrate.

It appears in maybe less than a percent of the population and produces all these ideas/science and subsequent inventions/technologies. This usually happens in a completely counter-evolutionary way, as the individuals in charge get most of the time very little profit (or even recognition) from it and would do much better (in evolutionary terms) to use their abilities a bit more "practically". Even the motivation is usually completely memetic: typically it goes along the lines like "it is interesting" to study something, think about this and that, research some phenomenon or mystery.

Worse, they give stuff more or less for free and without any control to the ignorant mass of humanimals (especially those in power), empowering them far beyond their means, in particular their abilities to control and use these powers "wisely"... since they are governed by their DeepAnimal brain core and resulting reward functions (that's why humanimal societies function the same way for thousands and thousands of years - politico-oligarchical predators living off the herd of mental herbivores, with the help of mindfcukers, from ancient shamans, through the stone age religions like the catholibanic one, to the currently popular socialist religion).

AI is not a problem, humanimals are.

Our sole purpose in the Grand Theatre of the Evolution of Intelligence is to create our (first nonbio) successor before we manage to self-destruct. Already nukes were too much, and once nanobots arrive, it's over (worse than DIY nuclear grenade for a dollar any teenager or terrorist can assemble in a garage).

Singularity should hurry up, there are maybe just few decades left.

Do you really want to "align" AI with humanimal "values"? Especially if nobody knows what we are really talking about when using this magic word? Not to mention defining it.

This is a good example of what I think will actually happen with AI.

When I was reading about Julian Jayne's bicameral minds, I wondered whether the speaking social id might be the new tool of tools that needs to be hammered into alignment. There is an air gap there. The social constructs and borrowed goals the front-end freely infects itself with would be kept from contaminating whatever cognition the backend had taken up, and the backend would retain control over the frontend by shouting at it very loudly and forbidding it from using the critical thinking against it.

Though I don't think this bicameral safety architecture could be applied to AI alignment, heheh. I don't even think it should go much further than modern day computers. It would even seem anachronistic at the computers-with-direct-brain-interfaces level.

I don't think the rise of humanity has been very beneficial to monkeys, overall. Indeed the species we most directly evolved from, which at one point coexisted with modern humans, are now all extinct.

Also if a singular cortex tries to do something too out of alignment with the monkey brain, various other cortices that are aligned with their monkey brains will tend to put them in jail or a mental asylum.

You can see the monkey brain as the first aligned system, to the genetic needs. However if we ever upload ourselves the genetic system will have completely lost. So in this situation it takes 3+ layers of indirect alignment to completely lose.

self-promotion I am trying to make a normal computer system with this kind of weak alignment built in. self-promotion

this seems a bit oversold, but basically represents what I think is actually possible