Types and Degrees of Alignment

When there is a difference between what I said, what I meant, what I should have said/meant, and what is best for me... I think I would prefer the AI to explain to me this difference, if possible. If impossible to explain, at least to say that there is a difference (and let me figure out how to deal with that information).

The collective version of the same problem is more difficult, because it turns possible resulting intrapersonal conflicts into interpersonal.

From the perspective of ontology and memetic engineering, the whole "ontology" or classification of alignments that you give, "fragile, friendly, ..." is bad because it's not based on some theory but rather on the cacophony of commonsensical ideas. These "alignments" don't even belong to the same type: "Fragile" is an engineering approach (but there are also many other engineering approaches which you haven't mentioned!), 2-3 and 5-6 are black-box descriptions of some alignment characteristics (at least these seem to belong to the same type), and "Strict" looks like a description of a mathematical mechanism. Also, the names are bad.

Fragile alignment

The name is really bad you use a property of this engineering approach (its fragility) as its name. But there are infinitely many other engineering approaches that are very fragile. For example, consider post-filtering LLM's outputs for any occurrences of the words like "bomb", "kill", "poison", n-word, etc., and not passing these rollouts through, but passing all the rest. Is this an alignment technique? Yes, if we consider the filter as part of the cognitive architecture, as we should. Is it fragile? Yes.

But when multiple such approaches are combined, this may actually lead to a cognitive architecture that is robustly aligned. This is the promise of LMCAs [LW · GW] and the natural language alignment [LW · GW].

It is not clear fragile alignment is even meaningfully helpful – that it does much, survives for long, or causes actions compatible with our survival, once the AGI is smarter than we are, even if we get its details mostly right and faced relatively good conditions. There are overlapping and overdetermined reasons (the strength and validity of which are of course disputed) to expect any good properties to break down exactly when it is important they not break down.

Laissez-faire (any prompt goes), "bare", non-post-filtered LLM rollouts as a cognitive architecture is obviously doomed. OpenAI has stopped deploying and giving access to base models (GPT-4 is available only in SFT'ed and RLHF'ed form), and I expect that in the next iteration (GPT-5), they will (my wishful thinking) stop even that -- they will give access only to an LMCA from the beginning. Even "rollout + post-filtering" is already a primitive form of such LMCA.

In turn, LMCAs in general shouldn't necessarily inherit the cardinal sin of LLMs (that are "exponentially diverging diffusion process", in the words of LeCun). And SFT/RLHF (things that you have called "fragile alignment") could be a part of a robust architecture, as I already noted above.

The first four and next four kinds of alignment you propose are parallel except that they concern a single person or society as a whole.  So I suggest the following names which are more parallel.  (Not happy about 3 and 7.)

1. Personal Literal Genie: Do exactly what I say.
2. Personal Servant: Do what I intended for you to do.
3. Personal Patriot: Do what I would want you to do.
4. Personal Nanny: Be loyal to me, but do what’s best for me, not strictly what I tells you to do or what he wants or intended.
5. Public Literal Genie: Do whatever it is collectively told.
6. Public Servant: Carry out the will of the people.
7. Public Patriot: Uphold the values of the people, and do what they imply.
8. Public Nanny: Do what needs to be done, whether the people like it or not.
9. Gentle Genie: The Genie from Aladdin. Note he is not strategic.
10. Arbiter: What is the law?

Is this ‘alignment’ a natural thing you can get easily or even by default, that is essentially a normal engineering problem, or is it a highly unnatural outcome where security mindset and bulletproof approaches as yet unfound even in principle are required, with any flaws are exploited, amplified and fatal, and many lethal problems all of which one must avoid?

To answer these questions specifically, it's really important not just to consider AI--human alignment "in the abstract", but embedded in the current civilisation, with its infrastructure and incentives structures. As I wrote here [LW(p) · GW(p)]:

[...] we should address this strategic concern by rewiring the economic and action landscapes (which also interacts with the "game-theoretic, mechanism-design" alignment paradigm mentioned above). The current (internet) infrastructure and economic systems are not prepared for the emergence of powerful adversarial agents at all:

• There are no systems of trust and authenticity verification at the root of internet communication (see https://trustoverip.org/)
• The storage of information is centralised enormously (primarily in the data centres of BigCos such as Google, Meta, etc.)
• Money has no trace, so one may earn money in arbitrary malicious or unlawful ways (i.e., gain instrumental power) and then use it to acquire resources from respectable places, e.g., paying for ML training compute at AWS or Azure and purchasing data from data providers. Formal regulations such as compute governance and data governance and human-based KYC procedures can only go so far and could probably be social-engineered by a superhuman imposter or persuader AI.

In essence, we want to design civilisational cooperation systems such that being aligned is a competitive advantage. Cf. "The Gaia Attractor" by Rafael Kaufmann.

This is a very ambitious program to rewire the entirety of the internet, other infrastructure, and the economy, but I believe this must be done anyway, just expecting a "miracle" HRAD invention to be sufficient without fixing the infrastructure and system design layers doesn't sound like a good strategy. By the way, such infrastructure and economy rewiring is the real "pivotal act".

If we imagined that the world had a "right" kind of infrastructure and social structure (really decentralised, trust-first), probably alignment would be much more of an "ordinary engineering" problem. With the current economic and infrastructural vulnerabilities mentioned, however, the alignment becomes a much higher-stakes problem, requiring more of "bulletproof" solutions "on the first try", I think.

Strict alignment. The damn thing will actually follow some set of instructions to the letter subject to its optimization constraints, hopefully you like the consequences of that. It is a potentially important crux if you disagree with the claim that for almost all specified instruction sets you won’t like the consequences, and there is no known good one yet, due to various alignment difficulties.

The name is uninformative and possibly misleading. If the set of instructions is in a natural or a formal language, you push the alignment difficulty into the semantics and semiotics, which are not "strict", and the alignment ends up not "strict" either.

In the planning-as-inference frame, I guess you probably mean something like an external evaluation (perhaps with some "good old-fashioned algorithm", a-la "type checker", rather than another AI, although it's really questionable whether such "type checker" could be built) of the inferred plans in their entirety. But again, even internal AI's representations are symbols, not "ground truth", so they are subject to the same difficulty of semantic and semiotic interpretation as natural language.

It is not clear to what extent strict alignment or strawberry alignment gives us affordance to reach good outcomes, how universal and deadly the various sources of lethality involved would be, or how difficult it would be to locate such affordances, especially on the first try.

"Strict" alignment is an engineering technique or approach, that shouldn't be judged in isolation but rather as a part of a cognitive architecture as a whole, as I explained in this comment [LW(p) · GW(p)].

"Strawberry" alignment is an external evaluation criterion or characteristic. However, I would go even further and say that "strawberry" is just a thought experiment which is not meant to be an actual eval that we will actually try, and the purpose of this thought experiment is just to show that the process of reasoning (and the result of reasoning, i.e., the plan), and the resulting behaviour are the actual objects of ethical evaluation and alignment rather than simply "goals". Goals become "good" or "bad" only in the context of larger plans and behaviour. This thought could be expressed in different ways, e.g., directly, as I just did (as well as in this comment [LW(p) · GW(p)]). The latest OpenAI paper, "Let's Verify Step By Step" highlights this idea, too.

It is not clear that human-level or friendly alignment would do us much good for long either, given the nature and history of humans, and the competitive dynamics involved, and the various reasons to expect change. If AGIs are much smarter and more capable and efficient than us, is there reason to think this level of alignment might be sufficient for long?

"Human-level" is just more commonly called "value alignment" (or "alignment with human values" if you want). But I agree with the conclusion: "friendly" is an attempt at "moral fact alignment" ("humanity is valuable to preserve"), which is probably futile without considering and aligning on the underlying theory ethics, i.e., without the methodological and scientific alignment, as I described in a different comment [LW(p) · GW(p)]. Value alignment, if taken literally, i.e., as attempting to impart AI with humans' heuristics about value, is also a species of "moral fact alignment", just somewhat more concrete than just "humanity is valuable to preserve" (although the latter is also one of the human values).

Have this alignment and the surrounding dynamics cause humans to choose to remain in control over time, or somehow be unable to choose differently.

This is self-contradictory: if the surrounding dynamics strongly preclude humans from "choosing otherwise", humans are no longer "in control". Also, under certain definitions of "choosing differently", humans may be precluded from moving into different biological and computational substrates, which in itself might be a cosmic tragedy because it may forever preclude humans from realising vast amounts of potential.

It is impossible in theory to have all these different kinds of alignment simultaneously. You cannot simultaneously (without any claim of completeness):

1. Do what I say
2. Also do what I mean
3. Also do what I should have said and meant
4. Also do what is best for me
5. Also do what broader society or humanity says
6. Also do what broader society or humanity means or should have said
7. Also do what broader society or humanity should have said given their values
8. Also do what is best for everyone
9. Do some ideal friendly combination of all of it that a broadly good guy would do, in a way that is respectful of and preserves what is valuable on all levels
10. Strictly follow some other set of rules that were set up long ago, no matter the cost

I think we should already forget about items 1 and 2 (as well as 10, but that goes without saying). In the context of communicating and aligning with superhumanly smart AI, it is really hubristic and stupid to think that although we will be much stupider than AI, it will be able to write a superhumanly good theory of ethics (such that humans didn't come up with in thousands of years) in a second, and design an AI (or change oneself) to follow this theory, humans' thoughts about value will still be somehow "covered with gold" and worth adhering to, for that AI.

Given this, I disagree that "we will need to: 1) Be able to get an AGI to do something a human selects at all, rather than something not selected. Be able to retain some form of control over what it does in the future, or set it on a chosen course. At all. ...". I'm not sure that this is possible (which implies that the orthogonality thesis holds to a sufficiently strong degree, which I doubt), but if this is possible, I don't think this would be desirable.

Items 3-9 raise an important and valid concern, that of the (infinity) multiplicity of (collective) identity [LW(p) · GW(p)] and alignment subjectivity [LW(p) · GW(p)].

We must pick at most one of those, or another variation on them, or something else, as our primary target. A machine cannot serve two masters any more than a man can, and the ability to put the machine under arbitrary stress, and its additional capabilities and intelligence, makes this that much more clear.

I think this is a wrong reaction to the above concern. Humans can serve multiple masters (themselves, their family, their community, their nation/society, the whole of humanity, and Gaia), so why AIs couldn't?

In the phrase "It is impossible in theory to have all these different kinds of alignment simultaneously", it's unclear what you mean by "having alignment". If you meant something like formal, "total" alignment, then sure, it's impossible to have perfect alignment in the physical reality outside us (i.e., outside simple simulations and mathematical abstractions). But if we strive for continuously increasing alignment along various dimensions and various levels of intelligent entities (alignment subjects), it should definitely be possible in general (although some "bad" situations may call for a choice where the alignment between certain entities or along a certain dimension is worsened in favour of the alignment between some other types of entities or along other dimensions).

I think that to increase our chances to realise a good future, we must find a principled way of addressing the issue of the multiplicity of identity and alignment subjectivity, which is the essence of a scale-free theory of ethics [LW · GW].

the system has unintended (harmful or dangerous) goals or behaviors.

Note that judgements about the harmfulness and dangerousness of some goals or behaviours are themselves theory-laden. This is why Goal alignment without alignment on epistemology, ethics, and science is futile [LW · GW]. From the perspective of any theory of cognition/intelligence that includes a generative model (which is not only Active Inference [? · GW], but also LeCun's H-JEPA [LW · GW], LMCAs [LW · GW] such as the "exemplary actor [AF · GW]", and more theories of cognition and/or AI architectures) for performing planning-as-inference, I think a straightforward and useful ladder of aligned could be introduced: methodological, scientific, and fact alignment [LW · GW]:

Thus, the crux of alignment is aligning the generative models of humans and AIs. Generative models could be "decomposed", vaguely (there is a lot of intersection between these categories), into

• Methodology: the mechanics of the models themselves (i.e., epistemology, rationality, normative logic, ethical deliberation),
• Science: mechanics, or "update rules/laws" of the world (such as the laws of physics or the heuristical learnings about society, economy, markets, psychology, etc.), and
• Fact: the state of the world (facts, or inferences about the current state of the world: CO2 level in the atmosphere, the suicide rate in each country, distance from Earth to the Sun, etc.)

These, we can conceptualise, give rise to "methodological alignment", "scientific alignment", and "fact alignment" respectively. Evidently, methodological alignment is most important: it in principle allows for alignment on science, and methodology plus science helps to align on facts.

Under this framework, goals are a specific type of facts, laden by a specific theory of mind of another agent (natural or AI). A theory of mind here should be a specialised version of a general theory of cognition which itself, as noted above, includes a generative model and planning-as-inference, under which goals become future world states or some features of future world states, predicted/planned (prediction and planning is the same thing, under planning-as-inference) by the other mind (or oneself, if the agent reflects about its own goals).^[1]

Thus, goal alignment is easy (practically, automatic) when two agents are aligned on methodology and science (albeit goal alignment even between methodologically and scientifically aligned agents usually still requires communication and coordination, unless we enter the territory of logical handshakes...), but is also futile when there is no common methodological and scientific ground. 

1. ^{^}

   Incidentally, this means that RL is not a very useful framework for discussing goals, because goals couldn't be conceptualised under RL easily, which causes a lot of trouble to people in the AI safety community who tend to think that there should be a single "right" theory or framework of cognition and intelligence. There should not: For alignment, we should simultaneously use multiple theories of cognition and value [LW · GW]. And RL, although probably couldn't be deployed very usefully to discuss goal alignment specifically, could still be used to discussed some aspects of value alignment between the minds.