LessWrong 2.0 Reader

The base model will probably succeed at this task at somewhere on the order of 0.35^10 or about 0.0003% of the time, while the RL'd model should succeed about 99.8% of the time.

The interesting concept in the paper is the location of the crossover point, which seems remarkably stable (for a given task) across specific RL techniques and amount of RL training. It can be measured experimentally for a task by doing a little bit of RL training, and RL@1 performance won't get better than that with more training, so you're unlikely to get the RL model to succeed 99.8% of the time (at pass@1) ever unless the level of performance of the base model at the crossover point with a weak RL model was already higher than 99.8%.

Probably the crossover point for a task depends on things that can be changed (such as strength of the pretrained model, or size/relevance of the verifiable task dataset, or possibly the inference time reasoning budget). The issue isn't for example as straightforward as losing entropy in RL policy (as a formulation of reduced exploration), since DAPO specifically addresses this issue (otherwise present in vanilla GRPO), but the pass@k plot for DAPO (Figure 7, top) barely moves (compared to other methods), in their experiment it's even slightly worse at the crossover point.

So in the context of this paper it remains unclear how to move the plot to reach ever higher base@k performance using RL@1, higher than the ceiling of where base@k already was at the crossover point when compating with some method at only 100-500 RL steps.

Given that the default, non-quotation text is not, in general, describing sounds in the environment, why do you think a reader would interpret unquoted text as environmental sounds rather than as simply more of the author's description of goings on in the scene? I can see that presenting spoken words in some format that allows or encourages their interpretation as environmental might be artistically useful, I just don't see that removing the quotation marks from otherwise-quoted dialog accomplishes that.

I'll second that I liked this post, but it felt mostly familiar to me. I've seen these ideas discussed in other game theory posts on Lesswrong.

In any discussion of Schelling points, the natural follow-up question is always: "Wait...is everything secretly a Keynesian Beauty Contest?!?"

Interesting paper. Quick thoughts:

• I agree the benchmark seems saturated. It's interesting that the authors frame it the other way - Section 4.1 focuses on how models are not maximally goal-directed.
• It's unclear to me how they calculate the goal-directedness for 'information gathering', since that appears to consist only of 1 subtask. 

I think he would lie, or be deceptive in a way that's not technically lying, but has the same benefits to him, if not more.

i earnt more from working at a call center for about 3 months than i have in 2+ years of working in ai safety. 

And i've worked much harder in this than I did at the call center

Conjecture seems unusually good at sticking to reality across multiple domains.

I do not get this impression, why do you say this?

You can ignore for now since I need to work through whether this is still true depending on how we view the source of uncertainty in doubling time. Edit: this explanation is correct afaict and worth looking into.

The parameters for the second log-normal (doubling time at RE-Bench saturation, 10th: 0.5 mo., 90th: 18 mo.) when made equivalent for an inverse gaussian by matching mean and variance (approx. InverseGaussian[7.97413, 1.315]) are implausible. The linked paper highlights that to be representing doubling processes reasonably, the ratio of first to second parameter ought to be << 2/ln(2) (or << (1/(2ln(2)^2))). The failure to match that indicates that the "size hypothesis" of any of the growth processes is violated, indicating that the distribution is no longer modeling uncertainty around such a process.

Ok, so that's too many functions, what does it mean? In general, it means that our "uncertainty" is actually the main driver of fast timelines now rather than reflecting a lack of knowledge in any way. The distribution is so stretched that the mode and median are wildly smaller than the mean entirely due to the possibility that a random unknown event causes foom, unrelated to the estimated "growth rate" of the process. It's like cranking up a noise term on a stock market model and being surprised that some companies are estimated to go to the moon tomorrow, then claiming it is due to estimating those stocks as potentially having huge upsides.

There is not a good solution that keeps the model intact (and the same basic issue is that the model is working in domains that are outcomes like time and frequency rather than inputs like time horizons, compute, or effective compute). If one were to use the same mean and scale up the second parameter, the left side of the pdf would collapse, and the mode and median would jump much higher resulting in a much later estimate of SC. That doesn't mean that's how to fix the model, but it does indicate fast timelines are incidental to and reflective of other issues in the model.

I think:

1. Finding principles for AI "behavioural engineering" that reduces people's desire to engage in risky races (e.g. because they find the principles acceptable) seems highly valuable
2. To the extent permitted by 1, pursuing something CEV like ("we're happier with the outcome in hindsight than we would've been with other outcomes") seems desirable also

I sort of see the former as potentially encouraging diversity (because different groups want different things, and are most likely to agree to "everyone gets what they want"), but the latter may in fact suggest convergence (because, perhaps, there are fairly universal answers to "what makes people happy with the benefit of hindsight?").

You stress the importance of having robust feedback procedures, but having overall goals like this can help to judge which procedures are actually doing what we want.

Is there a name for the phenomenon of increased intelligence or increased awareness leading to increased selfishness? It sounds like something that a psychologist would have named.