LessWrong 2.0 Reader

To get more comfortable with this formalism, we will translate three important voting criteria. 

You translated four criteria.

Uhm… what is the typical Ashkenazi diet?

It’s delicious, is what it is.

i'm glad that you wrote about AI sentience (i don't see it talked about so often with very much depth), that it was effortful, and that you cared enough to write about it at all. i wish that kind of care was omnipresent and i'd strive to care better in that kind of direction.

and i also think continuing to write about it is very important. depending on how you look at things, we're in a world of 'art' at the moment - emergent models of superhuman novelty generation and combinatorial re-building. art moves culture, and culture curates humanity on aggregate scales

your words don't need to feel trapped in your head, and your interface with reality doesn't need to be limited to one, imperfect, highly curated community. all communities we come across will be imperfect, and when there's scarcity: only one community to interface with, it seems like you're just forced to grant it privilege - but continued effort might just reduce that scarcity when you find where else it can be heard

your words can go further, the inferential distance your mind can cross - and the dynamic correlation between your mind and others - is increasing. that's a sign of approaching a critical point. if you'd like to be heard, there are new avenues for doing so: we're in the over-parametrized regime. 

all that means is that there's far more novel degrees of freedom to move around in, and getting unstuck is no longer limited to 'wiggling against constraints'. Is 'the feeling of smartness' or 'social approval from community x' a constraint you struggled with before when enacting your will? perhaps there's new ways to fluidly move around those constraints in this newer reality.

i'm aware that it sounds very abstract, but it's honestly drawn from a real observation regarding the nature of how information gets bent when you've got predictive AIs as the new, celestial bodies. if information you produce can get copied, mutated, mixed, curated, tiled, and amplified, then you increase your options for what to do with your thoughts

i hope you continue moving, with a growing stock pile of adaptations and strategies - it'll help. both the process of building the library of adaptations and the adaptations themselves.

in the abstract, i'd be sad if the acausal web of everyone who cared enough to speak about things of cosmic relevance with effort, but felt unheard, selected themselves away. it's not the selection process we'd want on multiversal scales 

the uneven distribution of luck in our current time, before the Future, means that going through that process won't always be rewarding and might even threaten to induce hopelessness - but hopelessness can often be a deceptive feeling, overlooking the improvements you're actually making. it's not something we can easily help by default, we're not yet gods. 

returning to a previous point about the imperfections of communities:

the minds or communities you'll encounter (the individuals who respond to you on LW,  AI's, your own mind, etc.), like any other complexity we stumble across, was evolved, shaped and mutated by any number of cost functions and mutations, and is full of path dependent, frozen accidents

nothing now is even near perfect, nothing is fully understood, and things don't yet have the luxury of being their ideals.

i'd hope that, eventually, negative feedback here (or lack of any feedback at all) is taken with a grain of salt, incorporated into your mind if you think it makes sense, and that it isn't given more qualitatively negative amplification. 

a small, curated, and not-well-trained-to-help-others-improve-in-all-regards group of people won't be all that useful for growth at the object level

ai sentience and suffering on cosmic scales in general is important and i want to hear more about it. your voice isn't screaming into the same void as before when AIs learn, compress, and incorporate your sentiments into themselves. thanks for the post and for writing genuinely 

is novel compared to... RepE

This is inaccurate, and I suggest reading our paper: https://arxiv.org/abs/2310.01405

Demonstrate full ablation of the refusal behavior with much less effect on coherence

In our paper and notebook we show the models are coherent.

Investigate projection

We did investigate projection too (we use it for concept removal in the RepE paper) but didn't find a substantial benefit for jailbreaking.

harmful/harmless instructions

We use harmful/harmless instructions.

Find that projecting away the (same, linear) feature at all layers improves upon steering at a single layer

In the RepE paper we target multiple layers as well.

Test on many different models

The paper used Vicuna, the notebook used Llama 2. Throughout the paper we showed the general approach worked on many different models.

Describe a way of turning this into a weight-edit

We do weight editing in the RepE paper (that's why it's called RepE instead of ActE).

Instead of thinking about how you can divide a discussion into two sides you can also focus on "what's actually true". In that case, it would make sense to end with an estimation of the size of the real gap.

If we, however, look at "what people argue", https://www1.udel.edu/educ/gottfredson/30years/Rushton-Jensen30years.pdf assumes the two categories culture-only (0% genetic–100% environmental) and the hereditarian (50% genetic–50% environmental).

Jay M defines the environmental model as <33% genetic and the genetic model as >66% genetic. What Rushton called the hereditarian position is right in the middle between Jay's environmental and genetic model. 

There is a serious issue with your proposed solution to problem 13. Using a random dictator policy as a negotiation baseline is not suitable for the situation, where billions of humans are negotiating about the actions of a clever and powerful AI. One problem with using this solution, in this contexts, is that some people have strong commitments to moral imperatives, along the lines of ``heretics deserve eternal torture in hell''. The combination of these types of sentiments, and a powerful and clever AI (that would be very good at thinking up effective ways of hurting heretics), leads to serious problems when one uses this negotiation baseline. A tiny number of people with sentiments along these lines, can completely dominate the outcome.

Consider a tiny number of fanatics with this type of morality. They consider everyone else to be heretics, and they would like the AI to hurt all heretics as much as possible. Since a powerful and clever AI would be very good at hurting a human individual, this tiny number of fanatics, can completely dominate negotiations. People that would be hurt as much as possible (by a clever and powerful AI), in a scenario where one of the fanatics are selected as dictator, can be forced to agree to very unpleasant negotiated positions, if one uses this negotiation baseline (since agreeing to such an unpleasant outcome, can be the only way to convince a group of fanatics, to agree to not ask the AI to hurt heretics, as much as possible, in the event that a fanatic is selected as dictator).

This post [LW · GW], explore these issues in the context of the most recently published version of CEV: Parliamentarian CEV (PCEV). PCEV has a random dictator negotiation baseline. The post shows that PCEV results in an outcome massively worse than extinction (if PCEV is successfully implemented, and pointed at billions of humans).

Another way to look at this, is to note that the concept of ``fair Pareto improvements'' has counterintuitive implications, when the question is about AI goals, and some of the people involved, has this type of morality. The concept was not designed with this aspect of morality in mind. And it was not designed to apply to negotiations about the actions of a clever and powerful AI. So, it should not be very surprising, to discover that the concept has counterintuitive implications, when used in this novel context. If some change in the world improves the lives of heretics, then this is making the world worse, from the perspective of those people, that would ask an AI to hurt all heretics as much as possible. For example: reducing the excruciating pain of a heretic, in a way that does not affect anyone else in any way, is not a ``fair Pareto improvement'', in this context. If every person is seen as a heretic by at least one group of fanatics, then the concept of ``fair Pareto improvements'' has some very counterintuitive implications, when it is used in this context.

Yet another way of looking at this, is to take the perspective of human individual Steve, who will have no special influence over an AI project. In the case of an AI, that is describable as doing what a group wants, Steve has a serious problem (and this problem is present, regardless of the details of the specific Group AI proposal). From Steve's perspective, the core problem, is that an arbitrarily defined abstract entity, will adopt preferences, that is about Steve. But, if this is any version of CEV (or any other Group AI), directed at a large group, then Steve has had no meaningful influence, regarding the adoption of those preferences, that refer to Steve. Just like every other decision, the decision of what Steve-preferences the AI will adopt, is determined by the outcome of an arbitrarily defined mapping, that maps large sets of human individuals, into the space of entities that can be said to want things. Different sets of definitions, lead to completely different such ``Group entities''. These entities all want completely different things (changing one detail can for example change which tiny group of fanatics, will end up dominating the AI in question). Since the choice of entity is arbitrary, there is no way for an AI to figure out that the mapping ``is wrong'' (regardless of how smart this AI is). And since the AI is doing what the resulting entity wants, the AI has no reason to object, when that entity wants the AI to hurt an individual. Since Steve does not have any meaningful influence, regarding the adoption of those preferences, that refer to Steve, there is no reason for him to think that such an AI will want to help him, as opposed to want to hurt him. Combined with the vulnerability of a human individual, to a clever AI that tries to hurt that individual as much as possible, this means that any group AI would be worse than extinction, in expectation.

Discovering that doing what a group wants, is bad for human individuals in expectation, should not be particularly surprising. Groups and individuals are completely different types of things. So, this should be no more surprising, than discovering that any reasonable way of extrapolating Dave, will lead to the death of every single one of Dave's cells. Doing what one type of thing wants, might be bad for a completely different type of thing. And aspects of human morality, along the lines of ``heretics deserve eternal torture in hell'' shows up throughout human history. It is found across cultures, and religions, and continents, and time periods. So, if an AI project is aiming for an alignment target, that is describable as ``doing what a group wants'', then there is really no reason for Steve to think, that the result of a successful project, would want to help him, as opposed to want to hurt him. And given the large ability of an AI to hurt a human individual, the success of such a project would be massively worse than extinction (in expectation).

The core problem, from the perspective of Steve, is that Steve has no control over the adoption of those preferences, that refer to Steve. One can give each person influence over this decision, without giving anyone any preferential treatment (see for example MPCEV in the post [LW · GW] about PCEV, mentioned above). Giving each person such influence, does not introduce contradictions, because this influence is defined in ``AI preference adoption space'', not in any form of outcome space. This can be formulated as an alignment target feature that is necessary, but not sufficient, for safety. Let's refer to this feature as the: Self Preference Adoption Decision Influence (SPADI) feature. (MPCEV is basically what happens, if one adds the SPADI feature to PCEV. Adding the SPADI feature to PCEV, solves the issue, illustrated by that thought experiment)

The SPADI feature is obviously very underspecified. There will be lots of border cases whose classification will be arbitrary. But there still exists many cases, where it is in fact clear, that a given alignment target, does not have the SPADI feature. Since the SPADI feature is necessary, but not sufficient, these clear negatives are actually the most informative cases. In particular, if an AI project is aiming for an alignment target, that clearly does not have the SPADI feature. Then the success of this AI project, would be worse than extinction, in expectation (from the perspective of a human individual, that is not given any special influence over the AI project). While there are many border cases, regarding what alignment targets could be described as having the SPADI feature, CEV is an example of a clear negative (in other words: there exists no reasonable set of definitions, according to which there exists a version of CEV, that has the SPADI feature). This is because building an AI that is describable as ``doing what a group wants'', is inherent in the core concept, of building an AI, that is describable as: ``implementing the Coherent Extrapolated Volition of Humanity''.

In other words: the field of alignment target analysis is essentially an open research question. This question is also (i): very unintuitive, (ii): very under explored, and (iii): very dangerous to get wrong. If one is focusing on necessary, but not sufficient, alignment target features. Then it is possible to mitigate dangers related to someone successfully hitting a bad alignment target, even if one does not have any idea of what it would mean, for an alignment target to be a good alignment target. This comment [LW(p) · GW(p)] outlines a proposed research effort, aimed at mitigating this type of risk.

These ideas also have implications for the Membrane concept, as discussed here [LW(p) · GW(p)] and here [LW(p) · GW(p)].

(It is worth noting explicitly that the problem is not strongly connected to the specific aspect of human morality discussed in the present comment (the ``heretics deserve eternal torture in hell'' aspect). The problem is about the lack of meaningful influence, regarding the adoption of self referring preferences. In other words, it is about the lack of the SPADI feature. It just happens to be the case, that this particular aspect of human morality is both (i): ubiquitous throughout human history, and also (ii): well suited for constructing thought experiments, that illustrates the dangers of alignment target proposals, that lack the SPADI feature. If this aspect of human morality disappeared tomorrow, the basic situation would not change (the illustrative thought experiments would change. But the underlying problem would remain. And the SPADI feature would still be necessary for safety).)

That makes sense, ty.

I'm likely not to actually quantify 'relative to' - there might be an ordered list of players if it seems reasonable to me (for example, if one submission uses 10 soldiers to get a 50% winrate and one uses 2 soldiers to get a 49% winrate, I would feel comfortable ranking the second ahead of the first - or if all players decide to submit the same number of soldiers, the rankings will be directly comparable), but more likely I'll just have a chart as in your Boojumologist scenario:

with one line added for 'optimal play'  (above or equal to all players) and one for 'random play' (hopefully below all players).

Overall, I don't think there's much optimization of the leaderboard/plot available to you - if you find yourself faced with a tough choice between an X% winrate with 9 soldiers or a Y% winrate with 8 soldiers, I don't anticipate the leaderboard taking a position on which of those is 'better'.

Are you familiar at all with the works of Christopher Alexander?  He spent about 50 years exploring the objectivity of aesthetics in Architecture (and was highly influential across several fields, including software design).  His book "The Timeless Way of Building" is available as an Audiobook and is approachable.  It is also the closest thing I have ever read to the teachings of my Tantric Teachers in India.

Basically, the book is about a "Pattern Language" by which beautiful things happen.  The hard part though is getting people to be honest about their feelings rather than lost in the intellectual games of taste.  Alexander did weird experiments like asking people "Between these two buildings, which one makes you more whole?"  People, being sophisticated and not woo, would typically say it's a stupid question.  So he would agree with them and say, "Okay, but if you had to pick one on that term, which would it be?"  He would get about 90% agreement on what is aesthetically right and what isn't.  Whereas if you get into matters of taste, you'll maybe get 10% agreement, because people need to be sophisticated and express interesting opinions about modern art, modular walls, other such things.

At the very least, he's striving to find ways to test these rather hard things, and separate points that seem impossible to tease out otherwise, such as actual feeling rather than intellectualizing.  And he was highly influential on the development of software patterns.  Most people who read the books seem to find them impactful and useful.  The downside is the thing he is finger-pointing-at-the-moon at for you is definitely "nameless" or perhaps even ineffable, yet also extremely obvious.

The book dances closely to the "Obviousness" in true creativity that the author of Impro talks about.  Another very recommendable book on both aesthetics and human dynamics in general.

Yes, I like it! Thanks for sharing that analysis, Gunnar.