LessWrong 2.0 Reader

Hmm, I'm not sure how what you're describing (learn on a bunch of examples of (query, well-thought-out guess)) is different from other forms of supervised learning.

Based on the paper Adam shared, it seems that part of the "amortizing" picture is that instead of simple supervised learning you look at examples of the form (context1, many examples from context1), (context2, many examples from context2), etc., in order to get good at quickly performing inference on new contexts.

It sounds like in the Paul Christiano example, you're assuming access to some internal reasoning components (like activations or chain-of-thought) to set up a student-teacher context. Is this equivalent to the other picture I mentioned?

I'm also curious about what you said about o3 (and maybe have a related confusion about this). I certainly believe that NN's, including RL models, learn by parallel heuristics (there's a lot of interp and theory work that suggests this), but I don't know any special properties of o3 that make it particularly supportive of this point of view

Update: Got tested, turns out the thing I have is bacterial rather than viral (Haemophilius influenzae). Lines up with the zinc lozenges not really helping! If I remember to take zinc the next time I come down with a cold, I'll comment here again. 

The reputational downsides of bad behavior will be multiplied 100-fold+ for AI as it reflects on millions of instances

human behavior reflects on the core structure individual humans are variations on, too

Thanks! I spent a bit of time understanding the stochastic inverse paper, though haven't yet fully grokked it. My understanding here is that you're trying to learn the conditional probabilities in a Bayes net from samples. The "non-amortized" way to do this for them is to choose a (non-unique) maximal inverse factorization that satisfies some d-separation condition, then guess the conditional probabilities on the latent-generating process by just observing frequencies of conditional events -- but of course this is very inefficient, in particular because the inverse factorization isn't a general Bayes net, but must satisfy a bunch of consistency conditions; and then you can learn a generative model for these consistency conditions by a NN and then perform some MCMC sampling on this learned prior.

So is the "moral" you want to take away here then that by exploring a diversity of tasks (corresponding to learning this generative prior on inverse Bayes nets) a NN can significantly improve its performance on single-shot prediction tasks?

Great discussion! I am open to the following bet [EA · GW].

If, until the end of 2028, Metaculus' question about superintelligent AI:

• Resolves non-ambiguously, I transfer to you 10 k January-2025-$.
• Does not resolve, you transfer to me 10 k January-2025-$. As before [EA · GW], I plan to donate my profits to animal welfare [? · GW] organisations.

In addition:

• The transfer should be made in January 2029.
• The nominal amount of the transfer in $ is 10 k times the ratio between the consumer price index for all urban consumers and items in the United States, as reported by the Federal Reserve Economic Data, in December 2028 and January 2025.

I think the bet would be neutral for you in terms of purchasing power, which is what matters if you also plan to donate the profits, if:

• Your median date of superintelligent AI as defined by Metaculus was the end of 2028. If you believe the median date is later, the bet will be worse for you.
• The probability of me paying you if you win was the same as the probability of you paying me if I win. The former will be lower than the latter if you believe the transfer is less likely given superintelligent AI, in which case the bet will be worse for you.

We can agree on another resolution date such that the bet is good for you accounting for the above.

This is also a concern I have but I feel like steering / project out is kinda sufficient to understand if the model uses this feature.

BTW, another problem with the thesis "Reward is not the optimization target", even with TurnTrout's stipulation that

This post addresses the model-free policy gradient setting, including algorithms like PPO and REINFORCE.

is that it's still not true even in the model-free policy gradient setting, in any substantive sense, and cannot justify the claims that TurnTrout & Belrose make. That is because of meta-learning: the 'inner' algorithm may in fact be a model-based RL algorithm which was induced by the 'outer' algorithm of PPO/REINFORCE/etc. A model-free algorithm may learn something which optimizes the reward; and a model-based algorithm may also learn something which does not optimize the reward.

There is just no hard and fast distinction here, it is dependent on the details of the system, the environment (ie. distribution of data), the amount of compute/ data, the convergence, and so on. (A good example from an earlier comment on how reward is the optimization target [LW(p) · GW(p)] is Bhoopchand et al 2023, which is about ablating the components.)

So if the expressible set of algorithms is rich enough to include model-based RL algorithms, and there are sufficient conditions, then your PPO algorithm 'which doesn't optimize the reward' simply learns an algorithm which does optimize the reward.

A simple, neat example is given by Botvinick [LW · GW] about NNs like RNNs. (As Shah notes in the comments, this is all considered basic RL and not shocking, and there are many examples of this sort of thing in meta-RL research - although what perspective you take on what is 'outer'/'inner' is often dependent on what niche you are in, and so Table 1 here may be a helpful Rosetta stone.)

You have a binary choice (like a bandit) which yields a 0/1 reward (perhaps stochastic with probability p to make it interesting) and your NN learns which one; you train a, let's say, fully-connected MLP with REINFORCE, which takes no input and outputs a binary variable to choose an arm; it learns that the left arm yields 1 reward and to always take left. You stop training it, and the environment changes to swap it: the left arm yields 0, and now right yields 1. The MLP will still pick 'left', however, because it learned a policy which doesn't try to optimize the reward. In this case, it is indeed the case that "reward is not the optimization target" of the MLP. It just learned a myopic action which happened to be selected for. In fact, even if you resume training it, it may take a long time to learn to instead pick 'right', because you have to 'undo' all of the now-irrelevant training towards 'left'. And you can do this swapping and training several times, and it'll be about the same each time: the MLP will slowly unlearn the old arm and learn the new arm, then the swapping happens, and now it's gotta do the same thing.*

But if you instead train an RNN, and to give it an input, you feed it a history of rewards and you otherwise train the exact same way... You will instead see something entirely different. After a swap, the RNN will pick the 'wrong' arm a few times, say 5 times, and receive 0 reward - and abruptly start picking the 'right' arm even without any further training, just the same frozen RNN weights. This sort of fast response to changing rewards is a signature of model-free vs model-based. This switch is possible because it has learned a new 'policy' over its history and the sufficient statistics encoded into its hidden weights which is equivalent to a Bayesian model of the environment and where it has learned to update its posterior probability of a switch having happened and that it is utility-maximizing to, after a certain number of failures, switch. And that 5 times was just how much evidence you need to overcome the small prior of 'a switch just happened right now'. And this utility-maximizing inner algorithm is incentivized by the outer algorithm, even though the outer algorithm itself has no concept of an 'environment' to be modeling or a 'reward' anywhere inside it. Your 'reward is not the optimization target' REINFORCE algorithm has learned the Bayesian model-based RL algorithm for which reward is the optimization target, and your algorithm as a whole is now optimizing the reward target, little different from, say, AlphaZero doing a MCTS tree search.

(And this should not be a surprise, because evolutionary algorithms are often cited as examples of model-free policy gradient algorithms which cannot 'plan' or 'model the environment' or 'optimize the reward', and yet, we humans were created by evolution and we clearly do learn rich models of the environment that we can plan over explicitly to maximize the reward, such as when we play Go and 'want to win the game'. So clearly the inference from 'algorithm X is itself not optimizing the reward' to 'all systems learned by algorithm X do not optimize the reward' is an illicit one.)

And, of course, in the other direction, it is entirely possible and desirable for model-based algorithms to learn model-free ones! (It's meta-learning all the way down.) Planning is expensive, heuristics cheap and efficient. You often want to distill your expensive model-based algorithm into a cheap model-free algorithm [LW(p) · GW(p)] and amortize the cost. In the case of the RNN above, you can, after the model-based algorithm has done its work and solved the switching bandit, throw that away, and replace it by a much cheaper simple model-free algorithm like if sum(reward_history) > 5 then last_action else last_action × −1, saving millions of FLOPs per decision compared to the RNN.

* further illustrating the weakness of 'reward is not the optimization target', it's not even obvious that this must be the case, rather than usually is under most setups. Meta-learning doesn't strictly require explicit conditioning on history nor does it require the clear fast vs slow weight distinction of RNNs or Transformer self-attention. A large enough MLP, continually trained through enough switches, could potentially learn to use the gradient updates+weights themselves as an outsourced history/model, and could eventually optimize its weights into a saddle point, where after exactly k updates by the fixed SGD algorithm, it 'happens to' switch its choice, corresponding to the hidden state being fused into the MLP itself. This would be like MAML. (If you are interested in this vein of thought, you may enjoy my AUNN proposal, which tries to take this to the logical extreme.)

Today I did some analysis of the grant data from 2024 and came up with the figures in the table above. I also updated the spreadsheet and the graph in the post to use the updated values.

If Alice thinks X happens with a probability of 20% while Bob thinks it's 40%, what would be a fair bet between them? 

I created a Claude Artifact, which calculates a bet such that the expected value is the same for both.

In this case, Bob wins if X happens (he thinks it's more likely). If Alice bets $100, he should bet $42.86, and the EV of such bet for both players (according to their beliefs) is $14.29. 

You are probably right that Singer would bite the bullet and say that Unlucky Lisa is not permitted to go to the theatre (even once). This is another thing, then, that I think Singer gets wrong (as well as - as stated in the essay - that PPBO is true/necessary for the argument of FAM).

Despite disagreeing with Singer on these important points, I still see myself as defending him and his project. After all, Singer didn't simply say 'Utilitarianism is true; therefore, we ought to be doing more than we are to help those suffering and dying from a lack of food, shelter, and medical care.' Doing applies/practical ethics isn't (or at least shouldn't be, in my opinion) like this. The best arguments in practical ethics will try as much as they can to rely only on premises almost anyone would accept, or at least which people with different background convictions and beliefs regarding normative ethical theory could accept.