LessWrong 2.0 Reader

If you don't know who to believe then falling back on prediction markets or at least expert consensus is not the worst strategy.

There's definitely an interest. Like many subjects, the limiting factor is people who are good writers, good at business (or at least knowledgeable about it) and have the slack to post. Pseudonyms are fine.

The European AI Office is finalizing Codes of Practice that will define how general-purpose AI (GPAI) models are governed under the EU AI Act.

They are explicitly asking for global expert input, and feedback is open to anyone, not just EU citizens.

The guidelines in development will shape:

• The definition of "systemic risk"
• How training compute triggers obligations
• When fine-tuners or downstream actors become legally responsible
• What counts as sufficient transparency, evaluation, and risk mitigation

Major labs (OpenAI, Anthropic, Google DeepMind) have already expressed willingness to sign the upcoming Codes of Practice. These codes will likely become the default enforcement standard across the EU and possibly beyond.

So far, AI safety perspectives are seriously underrepresented.

Without strong input from AI Safety researchers and technical AI Governance experts, these rules could lock in shallow compliance norms (mostly centered on copyright or reputational risk) while missing core challenges around interpretability, loss of control, and emergent capabilities.

I’ve written a detailed Longform [LW · GW] post breaking down exactly what’s being proposed, where input is most needed, and how you can engage.

Even if you don’t have policy experience, your technical insight could shape how safety is operationalized at scale.

📅 Feedback is open until 22 May 2025, 12:00 CET
🗳️ Submit your response here

Happy to connect with anyone individually for help drafting meaningful feedback. 

I think your main point is probably right but was not well argued here.

Fair, I though that an example would make this sufficiently obvious that it wasn't worth arguing for at length but I should have spelled it out a bit more.

I think it's also worth saying a thing very directly: just because non-behavioral evidence isn't likely to be widely legible and convincing does not mean it is not useful evidence for those trying to have correct beliefs.

FWIW, I do say this under "These techniques could be quite useful via two mechanisms:".

I think your main point is probably right but was not well argued here. It seems like the argument is a vibe argument of like "nah they probably won't find this evidence compelling". 

You could also make an argument from past examples where there has been large action to address risks in the world, and look at the evidence there (e.g., banning of CFCs, climate change more broadly, tobacco regulation, etc.) 

You could also make an argument from existing evidence around AI misbehavior and how its being dealt with, where (IMO) 'evidence much stronger than internals' basically doesn't seem to affect the public conversation outside the safety community (or even much here). 

I think it's also worth saying a thing very directly: just because non-behavioral evidence isn't likely to be widely legible and convincing does not mean it is not useful evidence for those trying to have correct beliefs. Buck's previous post [LW · GW] and many others discuss the rough epistemic situation when it comes to detecting misalignment. Internals evidence is going to be one of the tools in the toolkit, and it will be worth keeping in mind. 

Another thing worth saying: if you think scheming is plausible, and you think it will be difficult to update against scheming from behavioral evidence (Buck's post [LW · GW]), and you think non-behavioral evidence is not likely to be widely convincing (this post), then the situation looks really rough. 

If I'm interpreting the paper correctly the k at which base models start beating RL'd models is a per-task number, and k can be arbitrarily high for a given task, and the 50-400 range was specifically for tasks of the type the authors chose within a narrow difficulty band.

Let's say you have a base model which performs at 35% on 5 digit addition, and an RL'd model which performs at 99.98%. Even if the failures of the RL'd model are perfectly correlated, you'd need k=20 for base@20 to exceed the performance of fine-tuned@20. And the failures of the RL model won't be perfectly correlated - but this paper claims that the failures of the RL model will be more correlated than the failures of the base model, and so the lines will cross eventually, and "eventually" was @50 to @400 in the tasks they tested.

But you could define a task where you pass in 10 pairs of 5 digit numbers and the model must correctly find the sum of each pair. The base model will probably succeed at this task at somewhere on the order of 0.35^10 or about 0.0003% of the time, while the RL'd model should succeed about 99.8% of the time. So for this task we'd expect k in the range of k=220,000 assuming perfectly-correlated failures in the RL model, and higher otherwise.

Also I suspect that there is some astronomically high k such that monkeys at a keyboard (i.e. "output random tokens") will outperform base models for some tasks by the pass@k metric.

I'm missing some context here.  Is this not obvious, and well-supported by the vast majority of "treaties" between europeans and natives in the 16th through 19th centuries?  For legal settlements, it's generally between the extremes that each party would prefer, but it's not always the case that this range doesn't include "quite bad", even if not completely arbitrary.

"We'll kill you quickly and painlessly" isn't actually arbitrarily bad, it's only quite bad.  There are possibly worse outcomes available if no agreement was available.
 

Lots that I like in this paper. I, uh, see our probable future a bit differently from you.

I expect fully autonomous superhumanly fast and smart AGI operating in the world before 2028. I expect that before 2035, the total thought power of AI will likely greatly outstrip that of all humanity combined.

I do not see humanity remaining in control for long past this point. I believe we need to create an AGI we can literally trust with our lives, even operating in a fully independent superhuman way. We need to create what Dan Faggella calls a 'worthy successor'. Failure to do so soon enough likely means we all die. I feel like this pleasant talk of how to make chatbots polite within their contexts without rendering them utterly bland is.... missing the point rather badly. Disappointing!

Quote:

Nevertheless, the power of historical precedent makes such AI-only social structures rather unlikely outcomes, at least here on Earth in domains where human politics, preferences, and institutions are thoroughly entrenched. (Of course, other planets are another story, AI could conceivably operate at great distances from Earth, and would most likely end up with far greater latitude to evolve independently in those environments.) Most likely, our future is a hybrid one where intelligences, both biological and artificial, operating on a wide range of spatial and temporal scales, all coexist and operate together in social-institutional forms that evolved directly from those we already have on Earth in fully human-mediated and human-oriented forms.

Yeah it is confusing. You'd think there's tons of available data on pixelated game screens. Maybe training on it somehow degrades performance on other images?

A useful direction for further research would be to analyze fiction and non-fiction as separate corpora. Longer sentences may reflect a tendency towards hypotaxis, and they may also be a deliberate stylistic choice for fiction writers, perhaps setting a more languid and relaxed tone not appropriate for much non-fiction.