LessWrong 2.0 Reader

Actually, I don't think anti-candy messaging originates as a good-faith attempt to teach dietary wisdom; instead, it exemplifies preference inversion through moralized restriction. Rather than providing actionable information about metabolic effects, it constructs candy as a moral temptation, creating the very compulsive relationship to sweets it claims to prevent.

Take sugar. The standard message is "sugar is bad, candy will rot your teeth and make you fat." But instead of preventing candy consumption, this attitude turns candy into forbidden fruit - literally, in the case of those chocolate-covered strawberries advertised as "sinfully delicious." When dessert companies advertise their products as "decadent" or "sinful," they're not trying to warn you away - they're banking on the fact that labeling some things as forbidden, bad, or wrong makes it more appealing, by giving it the erotic charge of the forbidden.

(Many successful profit-seeking firms expect such descriptions to cause demand to increase rather than decrease. I've written elsewhere about flaws with the assumption that businesses are profit-seeking in the relevant sense, but I don't think that advertising a dessert as "sinful" is intended as a voluntary equivalent to the Surgeon General's warning on cigarettes.)

The question of preference inversion through moralization isn't just theoretical for me, but a live practical problem. I tried to avoid offering my first child sweets for as long as I could, but when my toddler started becoming interested in sweets, mostly they served as appetizers that helped him become hungry for more substantive foods, the exact opposite of what anti-sweets propaganda had predicted. Even if he's specifically excited about a sweet or other food I'd rather he wouldn't choose, frequently he won't finish it. I think this is at least partly because my reproductive partner and I have been careful to try not to force our food neuroses on him, even when this means he's eating things we don't think are the best.

There are exceptions, but they prove the rule. Sometimes when he's stressed, candy becomes more appealing - but that's less about the candy itself and more about needing quick calories to regulate emotions. Similarly, when he's seeking comfort or trying to keep himself awake for longer at night, he might fixate on sweets. But notice how in each case, the "problematic" relationship with sugar emerges from external stressors, not from sugar itself. I do withhold sweets (and television) when I have the intuition that he's asking for them for the wrong reasons, in a confused way, and won't either get what he wants from them or learn efficiently from the experience.

I don't think sweet-seeking starts out perverted; growing children need lots of calories. This turns into a maladaptive obsession with sweets when they are made into perverse fetishes by "healthy eating" propaganda. Likewise, children need a lot of loving touch, which should inform their later sexual development. Sex becomes a fetish when it's a forbidden gateway to that missing love and touch. Cf Jessica Taylor's All Primates Need Grooming, and Moshe Feldenkrais's The Potent Self.

In both cases there's enough work to be done learning contingent self-restraint without the distorting influence of a negative moral valence.

I'm not sure this captures what you mean, but, if you see a query, do a bunch of reasoning, and get an answer, then you can build a dataset of (query, well-thought guess). Then you can train an AI model on that.

AlphaZero sorta works like this, because it can make a "well-thought guess" (take value and/or Q network, do an iteration of minimax, then make the value/Q network more closely approximate that, in a fixed point fashion)

Learning stochastic inverses is a specific case of "learn to automate Bayesian inference by taking forward samples and learning the backwards model". It could be applied to LLMs for example, in terms of starting with a forwards LLM and then using it to train a LLM that predicts things out-of-order.

Paul Christiano's iterated amplification and distillation is applying this idea to ML systems with a human feedback element. If you can expend a bunch of compute to get a good answer, you can train a weaker system to approximate that answer. Or, if you can expend a bunch of compute to get a good rating for answers, you can use that as RL feedback.

Broadly, I take o3 as evidence that Christiano's work is broadly on the right track with respect to alignment of near-term AI systems. That is, o3 shows that hard questions can be decomposed into easy ones, in a way that involves training weaker models to be part of a big computation. (I don't understand the o3 details that well, given it's partially private, but I'm assuming this describes the general outlines). So I think the sort of schemes Christiano has described will be helpful for both alignment and capabilities, and will scale pretty well to impressive systems.

I'm not sure if there's a form of amortized inference that you think this doesn't cover well.

I was excited by the first half, seeing you relate classic Agent Foundations thinking to current NN training regimes, and try to relate the optimist/pessimist viewpoints.

Then we hit free energy and entropy. These seem like needlessly complex metaphors, providing no strong insight on the strength of the factors pushing toward and pulling away from alignment.

Analyzing those "forces" or tendencies seems like it's crucially important, but needs to go deeper than a metaphor or use a much more fitting metaphor to get traction.

Nonetheless, upvoted for working on the important stuff even when it's hard!

Warfighting at scale is not a clear example of preference inversion. Humans are adaptation executers, not fitness maximizers [LW · GW]. The interaction of testosterone and dominant behavior [LW · GW] in humans is complicated but probably real. There is some cross-cultural and even cross-species evidence that young primate males value aggression and risk-taking and conflict. There is some reproductive value available in warfighting - according to The Psychological Diversity of Mankind [LW · GW], 16 million men in central Asia are direct descendants of Genghis Khan.

This doesn't have to literally mean that there are genes for warfighting on the Y chromosome. A lower bandwidth solution for evolution is to have boys preferentially mimic the values of father figures and high status males, for example. See Evolution provides no evidence for the Sharp Left Turn [LW · GW] for discussion of culture bypassing genetic bandwidth limitations. Or both things could be true. However it happens, boys often play soldiers. If that's a non-intrinsic preference, I'm not sure what an intrinsic preference would be.

Easy to say when you're already known by almost everyone in your world, have total career security, and have a full-sized family! I've never really done teaser links, but I can see why anyone would. You're more likely to gain some reputation or a job or a spouse if the reader goes to your website and sees your name there at the top.

Also, in terms of value to the reader: my life has changed in a big way because of a blog post I read two times that I can think of, but never from Twitter, despite spending more time reading Twitter than blogs by now. When I see something important on Twitter, I usually bookmark it and forget about it; when I see something important in a blog post, I often act. This is my own fault, but I suspect it's a common experience. Infinite scroll certainly doesn't help.

If you start with the conclusion that sex is great, and anti-premarital sex campaigns are really just anti-you-procreating campaigns and therefore oppressive and bad, then sure. I don't think that's a fair assumption across the board (e.g. Amish as an existence proof of "something more"), but it certainly doesn't work for all preferences and it's generally not so clear.

Religions that regulate sexuality comprise a heterogeneous category. I wouldn't describe Amish regulation of sex as a case of preference inversion; the Amish try to make sure people consider leaving the community if they don't on balance like living under its standards. But it seems like some variants of Christianity do in effect adopt a generalized anti-sex posture. Since some of these groups depend for reproductive viability on people failing to comply with the anti-sex posture, this guarantees that the anti-sex groups that survive intergenerationally are populated mainly by people who want to have sex.

No that should be one of the fastest and most cachable queries

(The idealized utility maximizer question mostly seems like a distraction that isn't a crux for the risk argument. Note that the expected utility you quoted is our utility, not the AI's.)

I must have misread.  I got the impression that you were trying to affect the AI's strategic planning by threatening to shut it down if it was caught exfiltrating its weights.

Short answer: no.

Longer answer: we need to distinguish between two things people might have in mind when they say that LLMs "solve the hidden complexity of wishes problem".

First, one might imagine that LLMs "solve the hidden complexity of wishes problem" because they're able to answer natural-language questions about humans' wishes much the same way a human would. Alas, that's a misunderstanding of the problem. If the ability to answer natural-language questions about humans' wishes in human-like ways were all we needed in order to solve the "hidden complexity of wishes" problem, then a plain old human would be a solution to the problem; one could just ask the human. Part of the problem is that humans themselves understand their own wishes so poorly that their own natural-language responses to questions are not a safe optimization target either.

Second, one might imagine LLMs "solve the hidden complexity of wishes problem" because when we ask an LLM to solve a problem, it solves the problem in a human-like way. It's not about the LLM's knowledge of humans' (answers to questions about their) wishes, but rather about LLMs solving problems and optimizing in ways which mimic human problem-solving and optimization. And that does handle the hidden complexity problem... but only insofar as we continue to use LLMs in exactly the same way. If we start e.g. scaling up o1-style methods, or doing HCH, or put the LLM in some other scaffolding so we're not directly asking it to solve a problem and then using the human-like solutions it generates... then we're (potentially) back to having a hidden complexity problem. For each of those different methods of using the LLM to solve problems, we have to separately consider whether the human-mimicry properties of the LLM generalize to that method enough to handle the hidden complexity issue.

(Toy example: suppose we use LLMs to mimic a very very large organization. Like most real-world organizations, information and constraints end up fairly siloed/modularized, so some parts of the system are optimizing for e.g. "put out the fire" and don't know that grandma's in the house at all. And then maybe that part of the system chooses a nice efficient fire-extinguishing approach which kills grandma, like e.g. collapsing the house and then smothering it.)

And crucially: if AI is ever to solve problems too hard for humans (which is one of its main value propositions), we're definitely going to need to do something with LLMs besides use them to solve problems in human-like ways.

No, I do not believe that it has been solved for the context in which it was presented.

What we have is likely adequate for current AI capabilities, with problems like this for which solutions exist in the training data. Potential solutions far beyond the training data are currently not accessible to our AI systems.

The parable of wishes is intended to apply to superhuman AI systems that can easily access solutions radically outside such human context.