LessWrong 2.0 Reader

• If the user types “improve my fitness” into some interface, and it sets the AI’s reward function to be some “function of the user’s heart rate, sleep duration, and steps taken”, then the AI can potentially get a higher reward by forcing the user into eternal cardio training on pain of death, including forcibly preventing the person from turning off the AI, or changing its goals (see §2.2 above).
• The way that the reward function operationalizes “steps taken” need not agree with what we had in mind. If it’s operationalized as steps registered on a wearable tracker, the AI can potentially get higher reward by taking the tracker from the person and attaching it to a paint shaker. “Sleep” may be operationalized in a way that includes the user being forcibly drugged by the AI.
• If the user sets a goal of “help me learn Spanish over the next five years”, the AI can potentially get a higher reward by making modified copies of itself to aggressively earn or steal as much money and resources as possible around the world, and then have those resources available in case it might be useful for its local Spanish-maximization goal. For example, money can be used to hire tutors, or to train better successor AIs, or to fund Spanish-pedagogy or brain-computer interface research laboratories around the world, or of course to cheat by bribing or threatening whoever administers the Spanish exam at the end of the five years.

I don't really like those examples because just using human feedback fixes those (although I agree that those examples possibly sound more like the vague proposal from Silver and Sutton).

• If “the human gives feedback” is part of the reward function, then the AI can potentially get a higher score by forcing the human to give positive feedback, or otherwise exploiting edge-cases in how this feedback is operationalized and measured.

I think that sounds off to AI researchers. They might (reasonably) think something like "during the critical value formation period the AI won't have the ability to force humans to give positive feedback without receiving negative feedback". If the problem you described happens, it would be that the AI's value function learned to assign high value to "whatever strategy makes the human give positive feedback", rather than "whatever the human would endorse" or "whatever is favored according to the extrapolation of human values". That isn't really a specification gaming problem though. And I think your example isn't very realistic - more likely would be that the value function just learned a complex proxy for predicting reward which totally misgeneralizes in alien ways once you go significantly off distribution. 

One could instead name the specification gaming failure mode that the AI overoptimizes on what the humans think is good rather than what is actually good, e.g. proposing a convincing seeming alignment proposal which it doesn't expect to actually work but for which the humans reward it more than for admitting it cannot reliably solve alignment.

(For more on the alignment problem for RL agents, see §10 of my intro series [LW · GW], but be warned that it’s not very self-contained—it’s sorta in the middle of a book-length discussion of how I think RL works in the human brain, and its implications for safety and alignment.)

I think post 10 there is relatively self-contained and I would recommend it much more strongly, especially since you don't really describe the inner alignment problems here.

Re post overall:

I liked sections 2.2 and 2.3 and 3.

I think the post focuses way too much on specification gaming, and way too little on the problem of how to generalize to the right values from the reward. (And IMO, that's not just correct generalization to the value function, but also to the optimization target of a later more powerful optimization process, since I don't think a simple TD-learned value function could plan effectively enough to do something pivotal. (I would still be interested in your thoughts on my comment here [LW(p) · GW(p)], if you have the time.))

(Another outer-alignment-like problem that might be worth mentioning that the values of the AI might come out misaligned if we often just give reward based on the capability of the AI, rather than specifically human-value related stuff. (E.g. the AI might rather end up caring about getting kinds of instrumentally widely useful insights or so, which becomes a problem when we go off distribution, though of course even without that problem we shouldn't expect it to generalize off distribution in the way we want.) (Related [? · GW].))

I don’t believe the standard story of the resource curse. I also don’t think Norway and the Congo are useful examples, because they differ in too many other ways. According to o3, “Norway avoided the resource curse through strong institutions and transparent resource management, while the Congo faced challenges due to weak governance and corruption.” To me this is a textbook case of where existing models still fall short: the textbook story leaves out key factors and never comes close to proving that good institutions alone prevented the resource curse.

Regarding the main content, I find the scenario implausible. The “social-freeze and mass-unemployment” narrative seems to assume that AI progress will halt exactly at the point where AI can do every job but is still somehow not dangerous. You also appear to assume a new stable state in which a handful of actors control AGIs that are all roughly at the same level.

More directly, full automation of the economy would mean that AI can perform every task in companies already capable of creating military, chemical, or biological threats. If the entire economy is automated, AI must already be dangerously capable.

I expect reality to be much more dynamic, with many parties simultaneously pushing for ever-smarter AI while understanding very little about its internals. Human intelligence is nowhere near the maximum, and far more dangerous intelligence is possible. Many major labs now treat recursive self-improvement as the default path. I expect that approaching superintelligence without any deeper understanding of the internal cognition this way will give us systems that we cannot control and that will get rid of us. For these reasons, I have trouble worrying about job replacement. You also seem to avoid mentioning the extinction risk in this text.

Definitely for preference cascades. For common knowledge I'd say it's about undermining of common knowledge formation (eg meme to not share salary, strong pressure not to name that emperor is naked, etc.)

Relevant section of Project Lawful [LW · GW], on how dath ilan handles accountability in large organizations:

"Basic project management principles, an angry rant by Keltham of dath ilan, section one:  How to have anybody having responsibility for anything."

Keltham will now, striding back and forth and rather widely gesturing, hold forth upon the central principle of all dath ilani project management, the ability to identify who is responsible for something.  If there is not one person responsible for something, it means nobody is responsible for it.  This is the proverb of dath ilani management.  Are three people responsible for something?  Maybe all three think somebody else was supposed to actually do it.

Several paragraphs collapsed for brevity

Dath ilani tend to try to invent clever new organizational forms, if not otherwise cautioned out of it, so among the things that you get warned about is that you never form a group of three people to be responsible for something.  One person with two advisors can be responsible for something, if more expertise is required than one person has.  A majority vote of three people?  No.  You might think it works, but it doesn't.  When is it time for them to stop arguing and vote?  Whose job is it to say that the time has come to vote?  Well, gosh, now nobody knows who's responsible for that meta-decision either.  Maybe all three of them think it's somebody else's job to decide when it's time to vote.

The closest thing that dath ilan has to an effective organization which defies this principle is the Nine Legislators who stand at the peak of Governance, voting with power proportional to what they receive from the layers of delegation beneath them.  This is in no small part because dath ilan doesn't want Governance to be overly effective, and no private corporations or smaller elements of Governance do that.  The Nine Legislators, importantly, do not try to run projects or be at the top of the bureaucracy, there's a Chief Executive of Governance who does that.  They just debate and pass laws, which is not the same as needing to make realtime decisions in response to current events.  Same with the Court of Final Settlement of which all lower courts are theoretically a hierarchical prediction market, they rule on issues in slowtime, they don't run projects.

Even then, every single Governance-level planetwide law in dath ilan has some particular Legislator sponsoring it.  If anything goes wrong with that law, if it is producing stupid effects, there is a particular Legislator to point to, whose job it was to be the person who owned that law, and was supposed to be making sure it didn't have any stupid effects.  If you can't find a single particular Legislator to sign off on ownership of a law, it doesn't get to be a law anymore.  When a majority court produces an opinion, one person on the court takes responsibility for authoring that opinion.

Every decision made by the Executive branch of government, or the executive structure of a standardly organized corporation, is made by a single identifiable person.  If the decision is a significant one, it is logged into a logging system and reviewed by that person's superior or manager.  If you ask a question like 'Who hired this terrible person?' there's one person who made the decision to hire them.  If you ask 'Why wasn't this person fired?' there's either an identifiable manager whose job it was to monitor this person and fire them if necessary, or your corporation simply doesn't have that functionality.

Keltham is informed, though he doesn't think he's ever been tempted to make that mistake himself, that overthinky people setting up corporations sometimes ask themselves 'But wait, what if this person here can't be trusted to make decisions all by themselves, what if they make the wrong decision?' and then try to set up more complicated structures than that.  This basically never works.  If you don't trust a power, make that power legible, make it localizable to a single person, make sure every use of it gets logged and reviewed by somebody whose job it is to review it.  If you make power complicated, it stops being legible and visible and recordable and accountable and then you actually are in trouble.

The basic sanity check on organizational structure is whether, once you've identified the person supposedly responsible for something, they then have the eyes and the fingers, the sensory inputs and motor outputs, to carry out their supposed function and optimize over this thing they are supposedly responsible for.

Any time you have an event that should've been optimized, such as, for example, notifying Keltham that yet another god has been determined to have been messing with his project, there should be one person who is obviously responsible for that happening.  That person needs to successfully be notified by the rest of the organization that Cayden Cailean has been identified as meddling.  That person needs the ability to send a message to Keltham.

In companies large enough that they need regulations, every regulation has an owner.  There is one person who is responsible for that regulation and who supposedly thinks it is a good idea and who could nope the regulation if it stopped making sense.  If there's somebody who says, 'Well, I couldn't do the obviously correct thing there, the regulation said otherwise', then, if that's actually true, you can identify the one single person who owned that regulation and they are responsible for the output.

Sane people writing rules like those, for whose effects they can be held accountable, write the ability for the person being regulated to throw an exception which gets caught by an exception handler if a regulation's output seems to obviously not make sane sense over a particular event.  Any time somebody has to literally break the rules to do a saner thing, that represents an absolute failure of organizational design.  There should be explicit exceptions built in and procedures for them.

Exceptions, being explicit, get logged.  They get reviewed.  If all your bureaucrats are repeatedly marking that a particular rule seems to be producing nonsensical decisions, it gets noticed.  The one single identifiable person who has ownership for that rule gets notified, because they have eyes on that, and then they have the ability to optimize over it, like by modifying that rule.  If they can't modify the rule, they don't have ownership of it and somebody else is the real owner and this person is one of their subordinates whose job it is to serve as the other person's eyes on the rule.

Paragraph collapsed for brevity

'Nobody seems to have responsibility for this important thing I'm looking at' is another form of throwable exception, besides a regulation turning out to make no sense.  A Security watching Keltham wander around obviously not knowing things he's been cleared to know, but with nobody actually responsible for telling him, should throw a 'this bureaucratic situation about Keltham makes no sense' exception.  There should then be one identifiable person in the organization who is obviously responsible for that exception, who that exception is guaranteed to reach by previously designed aspects of the organization, and that person has the power to tell Keltham things or send a message to somebody who does.  If the organizational design fails at doing that, this incident should be logged and visible to the single one identifiable sole person who has ownership of the 'actually why is this part of the corporation structured like this anyways' question.

If one specific person in the Dutch government had been required to give the order to destroy the squirrels, taking full responsibility for the decision, it wouldn't have happened. If there had been an exception handler that employees could notify about the order, it wouldn't have happened. 

It appears likely that models that reward hack know that humans would disapprove of what they are doing, even if they do not verbalize it. If this is the case, are there training or prompting methods that increase the rate at which models verbalize their reward hacking thinking? This is similar to existing work on encouraging CoT faithfulness, but with a focus on reward hacking.

OpenAI published a paper on this (Detecting misbehavior in frontier reasoning models | OpenAI) - which I found interesting. Essentially, the RL process works by modifying gradient descent/optimization to adjust weights based on maximizing reward. This leads to an interesting undesirable effect. 

When researchers applied a reward penalty to reasoning traces that indicated reward hacking would happen, over time, the model began to reason completely benignly and then reward hack anyway. 

OpenAI theorize that this may be due to the model starting to reason in a steganographic manner and essentially developing token codes in its reasoning that trigger reward hacking, while appearing and reading benignly to human eyes. 

My initial reaction was somewhat emotive, but upon reflection, it seems fairly inevitable to me due to mathematical optimization that this behavior will arise. If it is computationally more efficient to lie and execute the hack, than reason honestly and be penalized, or play the game honestly, then the dishonest reasoning and execution of the hack will be where the model will inherently be driven to land during RL. 

Or put another way, if 1) playing chess is hard, and my reward may be +3 if I win and 0 if I lose, and if 2) trying to cheat, and getting caught, and winning my reward might be -2, but if 3) cheating and winning without getting caught is +8, eventually, one of my reasoning traces will result in condition 3 arising naturally. In this event, back propagation will result in the weights adjusting, causing more condition 3s to arise over time, until that is the rule by which the model reasons. 

OpenAI acknowledge this issue, and essentially came to the conclusion that the antedote to it is to use 'less capable models' in terms of length of time spent in RL. 

This was a great Post by the way, thank you. I enjoyed your perspectives. 

Back-of-the-envelope math indicates that an ordinary NPC in our world needs to double their power like 20 times over to become a PC. That’s a tough ask. I guess the lesson is either give up or go all in. 

Note that any competent capital holder has significant conflict of interest with AI, AI is already a significant fraction of the stock market and a pause would bring down most capital, not just private lab equity

Our competitors/other parties are doing dangerous things? Maybe we could coordinate and share our concerns and research with them

What probability do you put that, if Anthropic had really tried, they could have meaningfully coordinated with Openai and Google? Mine is pretty low

I think many of these are predicated on the belief that it would be plausible to get everyone to pause now. In my opinion this is extremely hard and pretty unlikely to happen. I think that, even in worlds where actors continue to race, there are actions we can take to lower the probability of x-risk, and it is a reasonable position to do so.

I separately think that many of the actions you describe historically were dumb/harmful, but are equally consistent with "25% of safety people act like this" and 100%

I think you're strawmanning him somewhat

It seems very clear that Jaime thinks that AI x-risk, is unimportant relative to almost any other issue, given his non-interest in trading off x-risk against those other issues.

Does not seem a fair description of

I wont endanger the life of my family, myself and the current generation for a small decrease of the chances of AI going extremely badly in the long term

People are allowed to have multiple values! If someone would trade a small amount of value A for a large amount of value B, this is entirely consistent with them thinking both are important.

Like, if you offer people the option to commit suicide in exchange for reducing x-risk by x%, what value of x do you think they would require? And would you say they are not x risk motivated if they eg aren't willing to do it at 1e-6?

In practice this doesn't really come up, so it's not that relevant. Similarly for Jaime's position, how much he believes himself to be in situations where he's trading off meaningful harm to today and meaningful harm to the present generation seems very important.

I did follow that turn, I just am confused by the examples you chose to illustrate it with. The first examples of Bell Labs and VC firms I agree match the claim, but not the subsequent ones.

I am imagining an accountability sink as a situation where the person held responsible has no power over the outcome, shielding a third party. So this is bad as in the airline example (Attendant held responsible by disgruntled passenger, although mostly powerless, this shields corporate structure, problem not resolved), and good as in the VC example (VC firm held responsible by investors for profits, although mostly powerless, this shields startup founders to take risks, problem resolved successfully).

And if this is the frame you're using, then I don't see how the ER doctor and ATC controller examples fit this mold?