LessWrong 2.0 Reader

Thanks, Richard! I have updated the bet to account for that.

If, until the end of 2028, Metaculus' question about superintelligent AI:

• Resolves non-ambiguously, I transfer to you 10 k January-2025-$ in the month after that in which the question resolved.
• Does not resolve, you transfer to me 10 k January-2025-$ in January 2029. As before [EA · GW], I plan to donate my profits to animal welfare [? · GW] organisations.

The nominal amount of the transfer in $ is 10 k times the ratio between the consumer price index for all urban consumers and items in the United States, as reported by the Federal Reserve Economic Data, in the month in which the bet resolved and January 2025.

The transfer should be made in January 2029

I think you mean in January 2029 or earlier if the question resolves before the end of 2028 otherwise there would be no need to introduce the CPI into the bet to keep things fair (or predictable).

The moderators have said that it is because posts tend to cover a range of stuff, rendering a block-vote agree or disagree less useful. Comments, on the other hand, tend to be narrower, making agree/disagree more useful.

Moderators are invited to agree/disagree on the narrow topic of my characterisation of their reasoning. :)

And 117 prospective mentors applied for Summer 2025!

Hmm, I'm not sure how what you're describing (learn on a bunch of examples of (query, well-thought-out guess)) is different from other forms of supervised learning.

Based on the paper Adam shared, it seems that part of the "amortizing" picture is that instead of simple supervised learning you look at examples of the form (context1, many examples from context1), (context2, many examples from context2), etc., in order to get good at quickly performing inference on new contexts.

It sounds like in the Paul Christiano example, you're assuming access to some internal reasoning components (like activations or chain-of-thought) to set up a student-teacher context. Is this equivalent to the other picture I mentioned?

I'm also curious about what you said about o3 (and maybe have a related confusion about this). I certainly believe that NN's, including RL models, learn by parallel heuristics (there's a lot of interp and theory work that suggests this), but I don't know any special properties of o3 that make it particularly supportive of this point of view

Update: Got tested, turns out the thing I have is bacterial rather than viral (Haemophilius influenzae). Lines up with the zinc lozenges not really helping! If I remember to take zinc the next time I come down with a cold, I'll comment here again. 

The reputational downsides of bad behavior will be multiplied 100-fold+ for AI as it reflects on millions of instances

human behavior reflects on the core structure individual humans are variations on, too

Thanks! I spent a bit of time understanding the stochastic inverse paper, though haven't yet fully grokked it. My understanding here is that you're trying to learn the conditional probabilities in a Bayes net from samples. The "non-amortized" way to do this for them is to choose a (non-unique) maximal inverse factorization that satisfies some d-separation condition, then guess the conditional probabilities on the latent-generating process by just observing frequencies of conditional events -- but of course this is very inefficient, in particular because the inverse factorization isn't a general Bayes net, but must satisfy a bunch of consistency conditions; and then you can learn a generative model for these consistency conditions by a NN and then perform some MCMC sampling on this learned prior.

So is the "moral" you want to take away here then that by exploring a diversity of tasks (corresponding to learning this generative prior on inverse Bayes nets) a NN can significantly improve its performance on single-shot prediction tasks?

Great discussion! I am open to the following bet [EA · GW].

If, until the end of 2028, Metaculus' question about superintelligent AI:

• Resolves non-ambiguously, I transfer to you 10 k January-2025-$ in the month after that in which the question resolved.
• Does not resolve, you transfer to me 10 k January-2025-$ in January 2029. As before [EA · GW], I plan to donate my profits to animal welfare [? · GW] organisations.

The nominal amount of the transfer in $ is 10 k times the ratio between the consumer price index for all urban consumers and items in the United States, as reported by the Federal Reserve Economic Data, in the month in which the bet resolved and January 2025.

I think the bet would be neutral for you in terms of purchasing power, which is what matters if you also plan to donate the profits, if:

• Your median date of superintelligent AI as defined by Metaculus was the end of 2028. If you believe the median date is later, the bet will be worse for you.
• The probability of me paying you if you win was the same as the probability of you paying me if I win. The former will be lower than the latter if you believe the transfer is less likely given superintelligent AI, in which case the bet will be worse for you.

We can agree on another resolution date such that the bet is good for you accounting for the above.

This is also a concern I have but I feel like steering / project out is kinda sufficient to understand if the model uses this feature.