Contra shard theory, in the context of the diamond maximizer problem

I appreciate you writing your quick thoughts on this. I have a few primary reactions, and then I'll detail specific reactions. 

• I agree there are difficulties in this plan template (and said so in the original post; I know you didn't say I didn't say so, but I'm adding this here for clarity). 
• I don't know why you think this isn't progress, because this plan's problems seem to just... go away, if they're solved? 
  • Like, if you figure out how to form an appropriate diamond abstraction, then that's that. Congrats, that part of the story is checked off.
  • But if you get a more robust reward model, then there's always another way to hack it. 
  • I surmise that you don't get this feeling from the post, or think I'm sweeping problems under some rug, but I don't know where the rug is supposed to be. (Maybe the "unnaturality" disagreement?)
• Of your three points, I think 
  • (1: Won't get AGI) seems wrong for this particular plan but also not cruxy to me. (Was this meant to apply to shard theory more broadly? If so, why?)
    • EDIT 11/25: Also, it doesn't seem like a big problem to me if it were true that "When you compromise and start putting it in environments where it needs to be able to think to succeed, then your new reward-signals end up promoting all sorts of internal goals that aren't particularly about diamond, but are instead about understanding the world and/or making efficient use of internal memory and/or suchlike." Seems fine if some of the agent's values are around understanding the world and suchlike.
  • (2: Proxy goal formation) seems like one of my main worries, and also quite surmountable.
  • (3: The values blow up when it gets smart) I agree that the reflection process seems sensitive in some ways. I also give the straightforward reason why the diamond-values shouldn't blow up: Because that leads to fewer diamonds. I think this a priori case is pretty strong, but agree that there should at least be a lot more serious thinking here, eg a mathematical theory of value coalitions [LW(p) · GW(p)].
• I think some of your critiques were covered in my original post. It was a long post, so no worries if you just missed them.

The first “problem” with this plan is that you don't get an AGI this way. You get an unintelligent robot that steers towards diamonds. If you keep trying to have the training be about diamonds, it never particularly learns to think. When you compromise and start putting it in environments where it needs to be able to think to succeed, then your new reward-signals end up promoting all sorts of internal goals that aren't particularly about diamond, but are instead about understanding the world and/or making efficient use of internal memory and/or suchlike.

Hm, doesn't it need to think in its curriculum I described in the OP?

Produce a curriculum of tasks, from walking to a diamond, to winning simulated chess games [self-play], to solving increasingly difficult real-world mazes, and so on. After each task completion, the agent gets to be near some diamond and receives reward.

For further detail, take an arbitrary task with a high skill ceiling and a legible end condition, give it some reward shaping and use self-play if appropriate, and put a diamond at the end and give the agent reward. I agree that even in successful stories, the agent also develops non-diamond shards. 

Here's a consideration for why training might produce an AGI, which I realized after writing the story. Given relevant features, it's often trivial for even linear models to outperform experts (see Statistical Prediction Rules Out-Perform Expert Human Judgments [LW · GW]). What I remember to be a common hypothesis: Human experts are often good at finding features to pay attention to (e.g. patient weight) but bad at setting regression coefficients to come to a decision. 

Analogously, consider an SSL+IL initialization in which the AI has imitatively learned sophisticated subroutines for perception, prediction, and action, such that the AI can imitate human-level performance on supervised training distribution (eg navigating mazes). Then PG-style RL finetuning might rearrange and reweight what subroutines to use when, efficiently finding a better subroutine arrangement for decision-making in a range of situations. And thereby doing better than human expert demonstrators. 

(Yes, this is sample inefficient, and I didn't particularly optimize the story for sample efficiency. I focused on telling any story at all which has the desired alignment outcome.)

insofar as you were able to get some sort of internalized diamond-ish goal, if you're not really careful then you end up getting lots of subgoals such as ones about glittering things, and stones cut in stylized ways, and proximity to diamond rather than presence of diamond, and so on and so forth.

Why "rather than" instead of "in addition to"? Are you just stating your belief here, or did you mean to argue for it? Maybe you're saying "It's hard to get the diamond shard to form properly", which I agree with and it's a primary way I expect the story to go wrong. I think that relatively simple interventions will plausibly solve this problem, though, and so consider this more of a research question than a fatal flaw in the training story template. 

As far as I can tell, the "reflection" section of TurnTrout’s essay says ~nothing that addresses this, and amounts to "the agent will become able to tell that it has shards". OK, sure, it has shards, but only some of them are diamond-related, and many others are cognition-related or suchlike. I don't see any argument that reflection will result in the AI settling at "maximize diamond" in-particular.

If I read you properly, that's not the relevant section. The relevant sections are the next two: The agent prevents value drift and The values handshake. EG I said [LW · GW]: 

If the agent still is primarily diamond-motivated, it now wants to stay that way by instrumental convergence. That is, if the AI considers a plan which it knows causes value drift away from diamonds, then the AI reflectively predicts the plan leads to fewer diamonds, and so the AI doesn’t choose that plan! The agent knows the consequences of value drift and it takes a more careful approach to future updating.

I think there's a very straightforward case here. In the relevant context, suppose the agent is primarily making decisions on the basis of whether they lead to more or fewer diamonds. The agent considers adopting a reflectively stable utility function which doesn't produce diamonds. The agent doesn't choose this plan because it doesn't lead to diamonds. 

I agree that there are ways this can go wrong, some of which you highlight. But the a priori argument makes me expect that, all else equal and conditional on a strong diamond shard at time of values handshake, the agent will probably equilibrate to making lots of diamonds.

I'll note that the diamond maximization problem is not in fact the problem "build an AI that makes a little diamond"

I did not claim to be solving the diamond maximization problem, but maybe you wanted to add your own take here? As I wrote in the original post [LW(p) · GW(p)], I think "maximize diamonds" is a seriously mistaken subproblem choice:

I think that pure diamond maximizers are anti-natural, and at least not the first kind of successful story we should try to tell...

I think that "get an agent which reflectively equilibrates to optimizing a single commonly considered quantity like 'diamonds'" is probably extremely hard and anti-natural [LW(p) · GW(p)]. I think MIRI should not have chosen this as a subproblem. 

I also think that relaxing the problem by assuming hypercomputation encourages thinking about argmax search, which I think is a subtle but serious trap. For specific generalizable reasons which I'll soon post about, this design pattern seems basically impossible to align compared to shard agents.

because the optimum of the shattered [LW · GW] correlates of the training objectives that it gets are likely to involve tiling the universe with something that isn't actually diamond, even if you're lucky-enough that it got a diamond-shard at all, which is dubious)

Really? That seems wrong. Suppose that the time of the values handshake, the agent has a strong diamond-shard. I understand you to predict that the agent adopts a reflective utility function which, when optimized, won't lead to actual diamond. Why? Why wouldn't the diamond-shard just bid this plan down, because it doesn't lead to actual diamond?

even if it works a little, it doesn't seem to me to be teaching us any of the insights that would be possessed by someone who knew how to robustly aim an idealized unbounded (or even hypercomputing) cognitive system in theory.

In addition to my "unbounded/hypercomputing is a red herring" response: 

Someone can say "You can reliably solve computer vision tasks by doing deep learning" isn't telling you how to write superhumanly good features into the vision model, surpassing previous hand-designed expert attempts. They don't know how the SOTA deep vision models will work internally. And yet it's still good advice. It's still telling you something about how to train good vision models. 

Similarly, if you're in a state of ignorance [LW · GW] (lethality 19) about how to reliably point any cognitive system to any latent parts of reality, and someone proposes a plan which does plausibly (for specific reasons, not as a vague "it could work" hope) produce an AI which makes lots of real-world diamonds, then that seems like progress to me. (I'm fine agreeing to disagree here, I don't think it's productive to dispute how much credit I should get.)

Smaller points:

In a similar fashion, the other “shards” that the shard theory folk want to learn are unnatural too.

I think it would make more sense to claim that niceness / other shards are "contingent" instead of "unnatural." If shard theory is correct, shards are literally natural in that they are found in nature as the predictable outcome of human value formation. Same for niceness. 

 little correlates-of-training-objectives that it latched onto in order to have a gradient up to general intelligence, blow the whole plan sky-high once it starts to reflect.

You call shards "little correlates" and, previously, "ad-hoc internalized correlates." I don't know what you intend to communicate by this. The shards are, mechanistically speaking, contextually activated influences on the agent's decision-making. What information does "ad-hoc" or "little correlate" add to that picture? I'm currently guessing that it expresses your skepticism that shards can cohere into reflectively stable caring?

Or consider the conflict "I really enjoy dunking on the outgroup (but have some niggling sense of unease about this)" — we can't conclude from the fact that the enjoyment of dunking is loud, whereas the niggling doubt is quiet, that the dunking-on-the-outgroup value will be the one left standing after reflection.

This is an interesting example. To me, the more relevant questions seem to be: How much evidence is "loudness" (e.g. if I really enjoy something which I do frequently, I sure am more likely to reflectively endorse it compared to if I didn't enjoy it, even though there are highly available counterexamples to this tendency), and how relevant is this for the diamond story? 

EDIT: As I think I wrote in the OP, it's not enough for a shard to be strongly influencing decision-making in a given context. Especially for an anti-outgroup shard which is unendorsed (eg bids for outcomes which other reflectively aware shards bid against), this shard also seemingly has to be reflectively and broadly activated in order to be retained. So, yeah, if there's an anti-outgroup shard which gets "maneuvered around and removed" by other shards, sure, that can happen. My takeaway isn't "anything can get removed for hard-to-understand reasons", but rather "one particular way shards can get removed is that they directly conflict with other powerful shards." 

I think a diamond-manufacturing subshard would resource-conflict (instrumental conflict, not terminal conflict) with eg a power-seeking subshard (manufacturing diamonds uses energy). Or even against a staple-manufacturing subshard (staples require materials and energy). But I expect the reflective utility function to reflect gains from intershard trade and specialization of different parts of the future resources towards the different decision-making influences (eg maybe one kind of comet is better specialized for making staples, and another kind for diamonds). 

Or maybe not. Maybe it goes some other way. But this kind of conflict seems different from anticorrelated terminal value (eg anti-outgroup can impinge on nice-shards, altruism-shards, empathy...) across a shard power imbalance (nonreflective anti-outgroup vs reflective niceness shard).

And my point here isn't "I have now defused the general class of objection, checkmate!"... It's still a live and legit worry to me, but I don't view this phenomenon as not comprehensible, I don't feel epistemically helpless here (not meaning to make claims about how you feel tbc).

Okay so if I'm understanding a little bit better now. What you're getting at is that self-generated true and useful philosophical insights become more and more likely to cause an ai to crash out of its domain of trained validity the smarter the ai gets, because philosophical insights are adversarial examples to many possible very smart beings, and therefore the order of philosophical insights can cause an insight to start propagating crash behavior through the rest of the network of nearby internal and external compute components starting from an agentic subnetwork?

Ok, so perhaps TurnTrout would disagree with me here, but my plan for coming up with a AGI-that-makes-diamonds using Shard theory would look more like this:

Create not one AI via this process, but a million. Each time, varying the parameters of the base instincts (the dynamic reward functions) which you have designed to try to get an AGI to care about diamond. Study the results in terms of how close each seems to get to being a 'true' diamond valuer. Then, extrapolate from these results, use your new-found knowledge to create a new batch of experiments. Examine and learn from these. Repeat this several times. Just for the sake of learning, try making it care about multiple things: diamonds, bananas, and chairs. Try using interpretability & editing tools to delete shards, or freeze some and train the others. The things you then end up learning about how to steer value systems of agents along the way turn out to be the true treasure all along. Then use this knowledge to actually try to build your diamond valuer.

The flaw I see in this plan is the question, "Can we successfully use these experiments to hill climb towards useful knowledge or would we just be fooling ourselves because even the seemingly 'better' agents would just be better liars?"

I think that then points at a dependency on reliable interpretability tools.

I am not a huge fan of shard theory, but other people seem into it a bunch. This post captured at least a bunch of my problems with shard theory (though not all of them, and it's not a perfect post). This means the post at least has saved me some writing effort a bunch of times. 

The first “problem” with this plan is that you don't get an AGI this way. You get an unintelligent robot that steers towards diamonds. If you keep trying to have the training be about diamonds, it never particularly learns to think. When you compromise and start putting it in environments where it needs to be able to think to succeed, then your new reward-signals end up promoting all sorts of internal goals that aren't particularly about diamond, but are instead about understanding the world and/or making efficient use of internal memory and/or suchlike.

Will humans stop having children as they get smarter and more powerful because they inadvertently gathered a bunch of utility function quirks like "curiosity"?

Separately, insofar as you were able to get some sort of internalized diamond-ish goal, if you're not really careful then you end up getting lots of subgoals such as ones about glittering things, and stones cut in stylized ways, and proximity to diamond rather than presence of diamond, and so on and so forth.

Will humans stop having children in the limit of intelligence and power, because we have all of these sub-shards like "make sure your children are safe", and "have lots of sex" instead of one big "spread your genes" one? Do they stop doing that when you introduce them to superstimulants via the internet or give them access to contraceptives that decouple sex from reproduction?

What the AI's shards become under reflection is very sensitive to the ways it resolves internal conflicts. For instance, in humans, many of our values trigger only in a narrow range of situations (e.g., people care about people enough that they probably can't psychologically murder a hundred thousand people in a row, but they can still drop a nuke), and whether we resolve that as "I should care about people even if they're not right in front of me" or "I shouldn't care about people any more than I would if the scenario was abstracted" depends quite a bit on the ways that reflection resolves inconsistencies.

The reason human morality is contextual and self contradictory, and we have to resolve a bunch of internal conflicts at the limit of reflectivity, is because we weren't actually trained to care about other people, the subgoal if any was "maintain the trustworthiness indicators of the people we're most likely to be able to cooperate with". So your examples are very cheesy and not at all convincing.

Do humans decide to kill or sterilize their children at higher INT and WIS scores if you change some abstract metacognition parameters that affect how they resolve (deliberately engineered) inconsistencies?

Reflection isn't easy. Humans don't seem to get it right often or at all. It is not something that gets turned on at a certain optimization strength but that grows out of precursors. Optimization power can be directed via attention mechanisms to inner and outer processes and I guess it is possible to prevent or sufficiently inhibit reflection.

For any AI that has an LLM as a component of it, I don't believe diamond-maximization is a hard problem, apart from Inner Alignment problems. The LLM knows the meaning of the word 'diamond' (GPT-4 defined it as "Diamond is a solid form of the element carbon with its atoms arranged in a crystal structure called diamond cubic. It has the highest hardness and thermal conductivity of any natural material, properties that are utilized in major industrial applications such as cutting and polishing tools. Diamond also has high optical dispersion, making it useful in jewelry as a gemstone that can scatter light in a spectrum of colors."). The LLM also knows its physical and optical properties, its social, industrial and financial value, its crystal structure (with images and angles and coordinates), what carbon is, its chemical properties, how many electrons, protons and neutrons a carbon atom can have, its terrestrial isotopic ratios, the half-life of carbon-14, what quarks a neutron is made of, etc. etc. etc. — where it fits in a vast network of facts about the world. Even if the AI also had some other very different internal world model and ontology, there's only going to be one "Rosetta Stone" optimal-fit mapping between the human ontology that the LLM has a vast amount of information about and any other arbitrary ontology, so there's more than enough information in that network of relationships to uniquely locate the concepts in that other ontology corresponding to 'diamond'. This is still true even if the other ontology is larger and more sophisticated: for example, locating Newtonian physics in relativistic quantum field theory and mapping a setup from the former to the latter isn't hard: its structure is very clearly just the large-scale low-speed limiting approximation.

The point where this gets a little more challenging is Outer Alignment, where you want to write a mathematical or pseudocode reward function for training a diamond optimizer using Reinforcement Learning (assuming our AI doesn't just have a terminal goal utility function slot that we can directly connect this function to, like AIXI): then you need to also locate the concepts in the other ontology for each element in something along the lines of "pessimizingly estimate the total number of moles of diamond (having at a millimeter-scale-average any isotopic ratio of C-12 to C-13 but no more than $N_1$ times the average terrestrial proportion of C-14, discounting any carbon atoms within $N_2$ C-C bonds of a crystal-structure boundary, or within $N_3$ bonds of a crystal -structure dislocation, or within $N_4$ bonds of a lattice substitution or vacancy, etc. …) at the present timeslice in your current rest frame inside the region of space within the future-directed causal lightcone of your creation, and subtract the answer for the same calculation in a counterfactual alternative world-history where you had permanently shut down immediately upon being created, but the world-history was otherwise unchanged apart from future causal consequences of that divergence". [Obviously this is a specification design problem, and the example specification above may still have bugs and/or omissions, but there will only be a finite number of these, and debugging this is an achievable goal, especially if you have a crystalographer, a geologist, and a jeweler helping you, and if a non-diamond-maximizing AI also helps by asking you probing questions. There are people whose jobs involve writing specifications like this, including in situations with opposing optimization pressure.]

As mentioned above, I fully acknowledge that this still leaves the usual Inner Alignment problems unsolved: applying Reinforcement Learning (or something similar such as Direct Preference Optimization) with this reward function to our AI, then how do we ensure that it actually becomes a diamond maximizer, rather than a biased estimator of diamond? I suspect we might want to look at some form of GAN, where the reward-estimating circuitry it not part of the Reinforcement Learning process, but is being trained in some other way. That still leaves the Inner Alignment problem of training a diamond maximizer instead of a hacker of reward model estimators.

In Shard Theory terms, if we reinforcement train the AI such that it has the reward-equivalent of an orgasm every time it creates a carat of diamond, show it a way to synthesize diamond and give it a taste of the effects, then if it didn't previously have a diamond shard, it's soon going to develop one.