LessWrong 2.0 Reader

But most people on Earth don't want "an artificial system to light the Earth in such a way as to mimic the sun", they want the actual sun to go on existing.

Yeah, agreed - the allocation of compute per human would likely become even more skewed if AI agents (or any other tooling improvements) allow your very top people to get more value out of compute than the marginal researcher currently gets.

And notably this shifting of resources from marginal to top researchers wouldn't require achieving "true AGI" if most of the time your top researchers spend isn't spent on "true AGI"-complete tasks.

The argument is (I assume):

• Once centuries have passed, you've already sent out huge amounts of space probes that roughly saturate reachable resources. (Because you can convert Proxima Centauri fully into probes within <20 years probably.)
• It doesn't take that much energy to pretty much fully saturate on probes. In particular, Eternity in six hours claims getting the energy for most of the probes you want is possible with just 6 hours of solar output (let alone eating 0.1% of the sun). Even if we assume this off by 2 OOMs (e.g. to be confident you get everywhere you need), that still means we can saturate on energy after 1 month of solar output. If we're willing to eat 0.1% of the sun (presumably at least millions of years of solar output?), the situation isn't even close. In fact, the key bottleneck based on Eternity in six hours is disassembling mercury (I think on heat dissipation) though it is hard to be confident in advance.

I've tried speaking with a few teams doing AI safety work, including:
 • assistant professor leading an alignment research group at a top university who is starting a new AI safety org
 • anthropic independent contractor who has coauthored papers with the alignment science team
 • senior manager at nvidia working on LLM safety (NeMo-Aligner/NeMo-Guardrails)
 • leader of a lab doing interoperability between EU/Canada AI standards
 • ai policy fellow at US Senate working on biotech strategies
 • executive director of an ai safety coworking space who has been running weekly meetups for ~2.5 years
 • startup founder in stealth who asked not to share details with anyone outside CAISI
 • chemistry olympiad gold medalist working on a dangerous capabilities evals project for o3
 • mats alumni working on jailbreak mitigation at an ai safety & security org
 • ai safety research lead running a mechinterp reading group and interning at EleuthrAI

Some random brief thoughts:
• CAISI's focus seems to be on stuff other than x-risks (i.e, misinformation, healthcare, privacy).
 • I'm afraid of being too unfiltered and causing offence.
 • Some of the statements made in the interviews are bizarrely devoid of content, such as:

"AI safety work is not only a necessity to protect our social advances, but also essential for AI itself to remain a meaningful technology."

 • Others seem to be false as stated, such as:

 "our research on privacy-preserving AI led us to research machine unlearning — how to remove data from AI systems — which is now an essential consideration for deploying large-scale AI systems like chatbots."

 • (I think a lot of unlearning research is bullshit, but besides that, is anyone deploying large models doing unlearning?)
 • The UK AISI research agendas seemed a lot more coherent with better developed proposals and theories of impact.
 • They're only recruiting for 3 positions for a research council that meets once a month?
 • CAD 27m of CAISI's initial funding is ~15% of the UK AISI's GBP 100m initial funding, but more than the U.S AISI's initial funding (USD $10m).
 • Another source says $50m CAD, but that's distributed over 5 years compared to a $2.4b budget for AI in general, so about 2% of the AI budget goes to safety?
 • I was looking for scientific advancements which would be relevant at the national scale. I read through every page of anthropic/redwood's alignment faking paper, which is considered the best empirical alignment research paper of 2024, but it was a firehose of info and I don't have clear recommendations that can be put into a slide deck.
 • Instead of learning more about what other people were doing on a shallow level it might've been more beneficial to focus on my own research questions or practice training project relevant skills.

Yes. Though notably, if your employees were 10x faster you might want to adjust your workflows to have them spend less time being bottlenecked on compute if that is possible. (And this sort of adaption is included in what I mean.)

The uncomputability of AIXI is a bigger problem than this post makes it out to be. This uncomputability inserts a contradiction into any proof that relies on AIXI -- the same contradiction as in Goedel's Theorem. You can get around this contradiction instead by using approximations of AIXI, but the resulting proofs will be specific to those approximations, and you would need to prove additional theorems to transfer results between the approximations.

I think I misunderstood what you were saying there - I interpreted it as something like

Currently, ML-capable software developers are quite expensive relative to the cost of compute. Additionally, many small experiments provide more novel and useful insights than a few large experiments. The top practically-useful LLM costs about 1% as much per hour to run as a ML-capable software developer, and that 100x decrease in cost and the corresponding switch to many small-scale experiments would likely result in at least a 10x increase in the speed at which novel, useful insights were generated.

But on closer reading I see you said (emphasis mine)

I was trying to argue (among other things) that scaling up basically current methods could result in an increase in productivity among OpenAI capabilities researchers at least equivalent to the productivity you'd get as if the human employees operated 10x faster. (In other words, 10x'ing this labor input.)

So if the employees spend 50% of their time waiting on training runs which are bottlenecked on company-wide availability of compute resources, and 50% of their time writing code, 10xing their labor input (i.e. the speed at which they write code) would result in about an 80% increase in their labor output. Which, to your point, does seem plausible.

I think a lot of thinking around multipolar scenarios suffers from heuristic "solution in the shape of the problem", i.e. "multipolar scenario is when we have kinda aligned AI, but still die due to coordination failures, therefore, solution for multipolar scenarios should be about coordination".

I think the correct solution is to leverage available superintelligence in nice unilateral way:

1. D/acc - use superintelligence to put as much defence as you can, starting from formal software verification and ending in spreading biodefence nanotech;
2. Running away - if you set up Moon/Mars/Jovian colony of nanotech-upgraded humans/uploads and pour available resources into defence, even if Earth explodes, humanity as a species survives. 

Do you have any thoughts on mechanism & whether prevention is actually worse independent of inconvenience?

And I've received an email from Mieux Donner confirming Lucie's leg has been executed for 1,000€. Thanks to everyone involved!

If if anyone else is interested in a similar donation swap, from either side, I'd be excited to introduce people or maybe even do this trick again :D