LessWrong 2.0 Reader

Finally, none of the frameworks make provisions for putting probabilities on anything. The framework itself needn’t include the hard numbers—indeed, such numbers might best be continually updated—but a risk management framework should point to a process that outputs well-calibrated risk estimates for known and unknown pathways before and after mitigation.

AI companies should publish numerical predictions, open Metaculus questions for each of their predictions, and provide large monetary prizes for predictors.

I'd guess this paper doesn't have the actual optimal methods.

Intuitively, this shouldn't matter much. They use some RL-on-CoTs method that works, and I expect its effects are not fundamentally different from optimal methods'. Thus, optimal methods might yield better quantitative results, but similar qualitative results: maybe they'd let elicit pass@800 capabilities instead of "just" pass@400, but it'd still be just pass@k elicitation for not-astronomical k.

Not strongly convinced of that, though.

Huh. This is roughly what I'd expected [LW(p) · GW(p)], but even I didn't expect it to be so underwhelming.^[1]

I weakly predict that the situation isn't quite as bad for capabilities as this makes it look. But I do think something-like-this is likely the case.

1. ^{^}

   Of course, moving a pass@400 capability to pass@1 isn't nothing, but it's clearly astronomically short of a Singularity-enabling technique that RL-on-CoTs is touted as.

If true, would this imply you want a base model to generate lots of solutions and a reasoning model to identify the promising ones and train on those?

If Anthropic’s position is that the current “alignment faking” behavior is applicable to the second scenario, I think that argument needs to be more explicitly laid out.

Aella estimates (in rather noisy fashion) that 3.2% of active, in-person sex workers in America are actively being sex trafficked.

From the linked post (emphasis in original):

So: given my estimated sex trafficking prevalence, I estimate about 3.2% of active, in-person sex workers in the US are currently being sex trafficked. This is higher than I anticipated. But I didn’t thoroughly check definitions of trafficking, and criticisms of sex trafficking concern often point out that ‘trafficking’ is a loose term with some reports including “anyone who crosses a border in order to do sex work, even if voluntary,” so who actually knows.

And indeed there is, in the sources cited in the post, very little in the way of consistent (or any!) clarification of just what the heck “sex trafficked” (or, for that matter, “sex workers”) is supposed to mean, thus no basis for assuming that the sources mean the same thing by the term, or even mean anything coherent by it…

In short, the only conclusion that can be drawn from Aella’s post is “how many sex workers are actively being sex trafficked? idk, what even are words and what do they mean”.

Story of a mostly homeless guy who scammed Isaac King out of $300. Isaac sued in small claims court on principle, did all the things, and none of it mattered.

This link goes to Sarah's tweet, not to Isaac's story.

British Columbia is recriminalizing marijuana

This is not what the article says. It says that BC is re-criminalizing hard drugs.

I am in BC, and have not heard anything about decriminalizing marijuana. I get the sense that it being legal is generally popular. Complaints about drug users are common here, but they are usually not talking about weed.

I very much appreciate you offering this concrete bet! I probably am not interested in taking this exact proposal and would need to set aside the time to do an investigation into your thread with Ryan [LW(p) · GW(p)] and similar to see how close current models are to resolving this, before taking it. I'll add to my to-do list to look into this and perhaps propose an alternative, if you think that might be useful.

See also my comments about how what you're saying is our 0th percentile is not my actual 0th percentile [LW(p) · GW(p)], and how I disagree with you regarding whether the metric you're proposing underestimates software progress [LW(p) · GW(p)], for the reasons I probably don't want to take the bet.

Perhaps it would be simpler and more cruxy to make a bet more directly on downstream tasks we care about, or perhaps metrics of usefulness like revenue? Perhaps looking at capabilities in early 2027, for which I have ~15-20% of having achieved superhuman coder and Daniel has ~25-30%? For example, what the trend on the METR suite will look like or whatever the closest equivalent will be, whether a panel we choose says they think it's plausible the leading AI company as a superhuman coder internally, what AI companies' revenue will look like, etc.?

Is your issue that it shouldn't be this determinate, or that it should be more clearly explained? I'm guessing both? As I've said I'm happy to make how important various parameters are more salient in non-high-effort ways.