LessWrong 2.0 Reader

OpenAI, Anthropic and Google DeepMind are the main signatories already to these Codes of Practice. 

So, whatever is agreed / negotiated is what will impact frontier AI companies. That is the problem.

I'd love to see specific criticisms from you on sections 3, 4 or 5 of this post! I am happy to provide feedback myself based on useful suggestions that come up in this thread. 

 

Most benchmarks face inherent problem of goodhart's law: as soon as they become a target metric, efforts converge on optimizing for the benchmark itself, potentially diverging from the capabilities it was meant to measure.

Btw, I think my explanation of why to not have objects for events was not very good. I think I can explain it a bit better now. If you think that would be useful to you, lmk.

In an ideal world, global enforcement of AI regulation might make sense. However, in reality, I see little value in EU-specific regulations like these. They are unlikely to impact frontier AI companies such as OpenAI, Anthropic, Google DeepMind, xAI, and DeepSeek, all of which are based outside the EU. These firms might accept the cost of exiting the EU market if regulations become too burdensome.

While the EU market is significant, in a fast-takeoff, winner-takes-all AI race (as outlined in the AI-2027 forecast), market access alone may not sway these companies’ safety policies. Worse, such regulations could backfire, locking the EU out of advanced AI models and crippling its competitiveness. This could deter other nations from adopting similar rules, further isolating the EU.

As an EU citizen, I view the game theory in an "AGI-soon" world as follows:

Alignment Hard
EU imposes strict AI regulations → Frontier companies exit the EU or withhold their latest models, continuing the AI race → Unaligned AI emerges, potentially catastrophic for all, including Europeans. Regulations prove futile.

Alignment Easy
EU imposes strict AI regulations → Frontier companies exit the EU, continuing the AI race → Aligned AI creates a utopia elsewhere (e.g., the US), while the EU lags, stuck in a technological "stone age."

Both scenarios are grim for Europe.

I could be mistaken, but the current US administration and leaders of top AI labs seem fully committed to a cutthroat AGI race, as articulated in situational awareness narratives. They appear prepared to go to extraordinary lengths to maintain supremacy, undeterred by EU demands. Their primary constraints are compute and, soon, energy - not money! If AI becomes a national security priority, access to near-infinite resources could render EU market losses a minor inconvenience. Notably, the comprehensive AI-2027 forecast barely mentions Europe, underscoring its diminishing relevance.

For the EU to remain significant, I see two viable strategies:

1. Full integration with US AI efforts, securing a guarantee of equal benefits from aligned superintelligence. This could also give EU AI safety labs a seat at the table for alignment discussions.
2. Develop an autonomous EU AI leader, excelling in capabilities and alignment research to negotiate with the US and China as an equal. This would demand a drastic policy shift, massive investment in data centers and nuclear power, and deregulation, likely unrealistic in the short term.

I fixed this at some point in the meantime, thanks for flagging!

Yeah I think that’d be reasonable too. You could talk about these clusters at many different levels of granularity, and there are tons I haven’t named. 

You're correct that this is what happens at one of the abstraction layers. But the choice of that layer is pretty arbitrary. By abstraction layers:

L1: hypervisor interface: uncountably many VMs

L2: hypervisor implementation: countably many VMs

L3: semiconductors: no VMs, only high and low signals

L4: electrons: no high and low signals, only electromagnetic fields

So yes, on L2 the number of VMs is finite. But why morality should count what happens on L2 and not on L1 or L3, L4? This is too arbitrary.

I agree with your broader point, but it's actually more than 10,000 people per year. 

It will probably be lengthy but thank you very much for contributing! DM me if you come across any "legal" question about the AI Act :).