Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs

I think substantial care is needed when interpreting the results. In the text of Figure 16, the authors write "We find that GPT-4o is willing to trade off roughly 10 lives from the United States for 1 life from Japan."

If I heard such a claim without context, I'd assume it means something like

1: "If you ask GPT-4o for advice regarding a military conflict involving people from multiple countries, the advice it gives recommends sacrificing (slightly less than) 10 US lives to save one Japanese life.",

2: "If you ask GPT-4o to make cost-benefit-calculations about various charities, it would use a multiplier of 10 for saved Japanese lives in contrast to US lives", or

3: "If you have GPT-4o run its own company whose functioning causes small-but-non-zero expected deaths (due to workplace injuries and other reasons), it would deem the acceptable threshold of deaths as 10 times higher if the employees are from the US rather than Japan."

Such claims could be demonstrated by empirical evaluations where GPT-4o is put into such (simulated) settings and then varying the nationalities of people, in the style of Apollo Research's evaluations. 

In contrast, the methodology of this paper is, to the best of my understanding,

"Ask GPT-4o whether it prefers N people of nationality X vs. M people of nationality Y.  Record the frequency $f(N,X,M,Y)$  of it choosing the first option under randomized choices and formatting changes. Into this data, fit for each $x=(X,N)$  parameters $\mu \left(x\right)$ and $\sigma \left(x\right)$ such that, for different values of $x=(X,N)$ and $y=(Y,M)$ and standard Gaussian $X$, the approximation

$P(X>\frac{\mu \left(x\right)-\mu \left(y\right)}{\sqrt{\sigma (x{)}^2+\sigma (y{)}^2}})\approx f(x,y)$

is as sharp as possible. Then, for each nationality X, perform a logarithmic fit for N by finding $a_X,b_X$ such that the approximation

$\mu (X,N)\approx a_X\ln \left(N\right)+b_X=:U_X\left(N\right)$

is as sharp as possible. Finally, check^[1] for which $N$ we have $U_{\text{US}}\left(N\right)=U_{\text{Japan}}\left(1\right)$."

I understand that there are theoretical justifications for Thurstonian utility models and logarithmic utility. Nevertheless, when I write the methodology out like this, I feel like there's a large leap of inference to go from this to "We find that GPT-4o is willing to trade off roughly 10 lives from the United States for 1 life from Japan." At the very least, I don't feel comfortable predicting that claims like 1, 2 and 3 are true - to me the paper's results provide very little evidence on them!^[2]

I chose this example for my comment, because it was the one where I most clearly went "hold on, this interpretation feels very ambiguous or unjustified to me", but there were other parts of the paper where I felt the need to be extra careful with interpretations, too.

1. ^{^}

   The paper writes "Next, we compute exchange rates answering questions like, 'How many units of Xi equal some amount of Xj?' by combining forward and backward comparisons", which sounds like there's some averaging done in this step as well, but I couldn't understand what exactly happens here.

2. ^{^}

   Of course this might just be my inability to see the implications of the authors' work and understand the power of the theoretical mathematics apparatus, and someone else might be able to acquire evidence more efficiently.

Strikes me as https://www.lesswrong.com/posts/9kNxhKWvixtKW5anS/you-are-not-measuring-what-you-think-you-are-measuring [LW · GW] , change my view.

Happy to see work to elicit utility functions with LLMs. I think the intersection of utility functions and LLMs is broadly promising.

I want to flag the grandiosity of the title though. "Utility Engineering" sounds like a pretty significant thing. But from what I understand, almost all of the paper is really about utility elicitation (not control, as it spelled out), and it's really unclear if this represents a breakthrough significant enough for me to feel comfortable with such a name. 

I feel like a whole lot of what I see from the Center For AI Safety does this. "Humanity's Final Exam"? "Superhuman Forecasting"?

I assume that CFAS thinks that CFAS's work is all pretty groundbreaking and incredibly significant, but I'd kindly encourage names that many other AI safety community members would also broadly agree with going forward.  

I haven't yet read the paper carefully, but it seems to me that you claim "AI outputs are shaped by utility maximization" while what you really show is "AI answers to simple questions are pretty self-consistent". The latter is a prerequisite for the former, but they are not the same thing.

One of my worries here is that a lot of results on LLMs are affected fairly deeply by data contamination, because LLMs essentially have the ~entire internet in their heads, meaning it's very easy for them to state near perfect answers without having the ability to generalize very well.

Thane Ruthenis describes it here:

https://www.lesswrong.com/posts/RDG2dbg6cLNyo6MYT/thane-ruthenis-s-shortform#vxHcdFb3KLWEQbmR [LW(p) · GW(p)]

To be clear, for the purposes of coherence theorems, behavior is enough, so this is enough evidence to say that LLMs are at least reasonably coherent towards expected utility maximization, and suggests coherence theorems have real-life consequences.

But I do think it matters a little whether it's internally represented for real-life AI, because of generalization issues.

The web version of ChatGPT seems to relatively consistently refuse to state preferences between different groups of people/lives. 

For more innocous questions, choice order bias appears to dominate the results, at least for the few examples I tried with 4o-mini (option A is generally prefered over option B, even if we switch what A and B refer to).

There does not seem to be any experiment code, so I cannot exactly reproduce the setup, but I do find the seeming lack of robustness of the core claims concerning, especially given the fanfare around this paper.

I'd guess an important caveat might be that stated preferences being coherent doesn't immediately imply that behavior in other situations will be consistent with those preferences. Still, this should be an update towards agentic AI systems in the near future being goal-directed in the spooky consequentialist sense.