Neuroscience and Alignment

I am hopeful there are in fact clean values or generators of values in our brains such that we could just understand those mechanisms, and not other mechanisms. In worlds where this is not the case, I get more pessimistic about our chances of ever aligning AIs, because in those worlds all computations in the brain are necessary to do a "human mortality", which means that if you try to do, say, RLHF or DPO to your model and hope that it ends up aligned afterwards, it will not be aligned, because it is not literally simulating an entire human brain. It's doing less than that, and so it must be some necessary computation its missing.

There are two different hypotheses I feel like this is equivocating between: there being values in our brain that are represented in a clean enough fashion that you can identify them by looking at the brain bottom-up; and that there are values in our brain that can be separated out from other things that aren't necessary to "human morality", but that requires answering some questions of how to separate them. Descriptiveness vs prescriptiveness - or from a mech interp standpoint, one of my core disagreements with the mech interp people where they think we can identify values or other high-level concepts like deception simply by looking at the model's linear representations bottom-up, where I think that'll be a highly non-trivial problem.

because in those worlds all computations in the brain are necessary to do a "human mortality"

I think you meant "human morality"

Why is value alignment different from these? Because we have working example of a value-aligned system right in front of us: The human brain. This permits an entirely scientific approach, requiring minimal philosophical deconfusion. And in contrast to corrigibility solutions, biological and artificial neural-networks are based upon the same fundamental principles, so there's a much greater chance that insights from the one easily work in the other.

The similarities go even deeper, I'd say, see e.g. The neuroconnectionist research programme for a review and quite a few of my past linkposts (e.g. on representational alignment and how it could be helpful for value alignment, on evidence of [by default] (some) representational alignment between LLMs and humans, etc.); and https://www.lesswrong.com/posts/eruHcdS9DmQsgLqd4/inducing-human-like-biases-in-moral-reasoning-lms [LW · GW]. 

I've been in many conversations where I've mentioned the idea of using neuroscience for outer alignment, and the people who I'm talking to usually seem pretty confused about why I would want to do that. Well, I'm confused about why one wouldn't want to do that, and in this post I explain why.

I've had related thoughts [LW(p) · GW(p)] (and still do, though it's become more of a secondary research agenda); might be interesting to chat (more) during https://foresight.org/2024-foresight-neurotech-bci-and-wbe-for-safe-ai-workshop/. 

The brain algorithms that do moral reasoning are value-aligned in the same way a puddle is aligned with the shape of the hole it's in.

They're shaped by all sorts of forces, ranging from social environment to biological facts like how we can't make our brains twice as large. Not just during development, but on an ongoing basis our moral reasoning exists in a balance with all these other forces. But of course, a puddle always coincidentally finds itself in a hole that's perfectly shaped for it.

If you took the decision-making algorithms from my brain and put them into a brain 357x larger, that tautological magic spell might break, and the puddle that you've moved into a different hole might no longer be the same shape as it was in the original hole.

If you anticipate this general class of problems and try to resolve them, that's great! I'm not saying nobody should do neuroscience. It's just I don't think it's a "entirely scientific approach, requiring minimal philosophical deconfusion," nor does it lead to safe AIs that are just emulations of humans except smarter.

A clarification about in what sense I claim "biological and artificial neural-networks are based upon the same fundamental principles":

I would not be surprised if the reasons why neural networks "work" are also exploited by the brain.

In particular why I think neuroscience for value alignment is good is because we can expect that the values part of the brain will be compatible with these reasons, and won’t require too much extra fundamental advances to actually implement, unlike say corrigibility, which will first progress from ideal utility maximizers, and then require a mapping from that to neural networks, which seems potentially just as hard as writing an AGI from scratch.

In the case where human values are incompatible with artificial neural networks, again I get much more pessimistic about all alternative forms of value alignment of neural networks.

biological and artificial neural-networks are based upon the same fundamental principles

I'm confused by this statement. Do we know this? Do we have enough of an understanding of either to say this? Don't get me wrong, there's some level on which I totally buy this. However, I'm just highly uncertain about what is really being claimed here. 

the only aligned AIs are those which are digital emulations of human brains

I don't think this is necessarily true. I don't think emulated human brains are necessary for full alignment, nor whether emulated human brains would be more aligned than a well calibrated and scaled up version of our current alignment techniques (+ new ones to be discovered in the next few years). To emulate the entire human brain to align values seem to be not only implausible (even with neurmorphic computing, efficient neural networks and Moore's law^1000), it seems like an overkill and a misallocation of valuable computational resources.  Assuming I'm understanding "emulated human brains" correctly, emulation would mean pseudo-sentient systems solely designed to be aligned to our values. Perhaps morality can be a bit simpler than that, somewhere in the middle of static, written rules (the law) and the unpredictable human mind. Because if we do make more of people essentially, it's not really addressing the "many biases or philosophical inadequacies" of us.