LessWrong 2.0 Reader

Yes, I wanted to argue something like this. 

"Rogue celestial body" should also include more commonly encountered interstellar objects, such as Oumuamua. Attempting to detect these objects (and to deflect potential impactors) does not seem obviously futile.

(I think a lot of unlearning research is bullshit, but besides that, is anyone deploying large models doing unlearning?)

Why is unlearning research bullshit? It seems to me that knowing how effectively we can remove something from a model is just the flip-side of understanding how a model changes when we say, fine-tune to instill values.

Although not the primary goal of unlearning, work into how to 'remove' should also equally benefit attempts to 'instill' robust values into the model. If fine-tuning for value alignment just patches over 'bad facts' with 'good facts' any 'aligned' model will be less robust than one with harmful knowledge properly removed. If the alignment faking paper and peripheral alignment research are important at a meta level, then perhaps unlearning will be important because it can tell us something about 'how deep' our value installation really is. Lack of current practical use isn't really important, we should be able to develop theory that will tell us something important about model internals. I think there is a lot of very interesting mech-interp of unlearning work waiting to be done that can help us here.

Thanks for the comment - the reason I focus on cosmic unfairness here is because I addressed local unfairness in a previous post [LW · GW] in the sequence - apologies this wasn't clear, I've now added a hyperlink to clarify. 

I don't agree that the challenges of Dostoevsky etc are only about local unfairness though: as I say I think it's typically a mixture of local and cosmic unfairness that are not clearly distinguished. 

The 'problem from the lack of intervention' that you mention is much discussed by people in this context, so presumably they think it is relevant to the challenges they are considering, even if there is no easy solution.

But most people on Earth don't want "an artificial system to light the Earth in such a way as to mimic the sun", they want the actual sun to go on existing.

Yeah, agreed - the allocation of compute per human would likely become even more skewed if AI agents (or any other tooling improvements) allow your very top people to get more value out of compute than the marginal researcher currently gets.

And notably this shifting of resources from marginal to top researchers wouldn't require achieving "true AGI" if most of the time your top researchers spend isn't spent on "true AGI"-complete tasks.

The argument is (I assume):

• Once centuries have passed, you've already sent out huge amounts of space probes that roughly saturate reachable resources. (Because you can convert Proxima Centauri fully into probes within <20 years probably.)
• It doesn't take that much energy to pretty much fully saturate on probes. In particular, Eternity in six hours claims getting the energy for most of the probes you want is possible with just 6 hours of solar output (let alone eating 0.1% of the sun). Even if we assume this off by 2 OOMs (e.g. to be confident you get everywhere you need), that still means we can saturate on energy after 1 month of solar output. If we're willing to eat 0.1% of the sun (presumably at least millions of years of solar output?), the situation isn't even close. In fact, the key bottleneck based on Eternity in six hours is disassembling mercury (I think on heat dissipation) though it is hard to be confident in advance.

I've tried speaking with a few teams doing AI safety work, including:
 • assistant professor leading an alignment research group at a top university who is starting a new AI safety org
 • anthropic independent contractor who has coauthored papers with the alignment science team
 • senior manager at nvidia working on LLM safety (NeMo-Aligner/NeMo-Guardrails)
 • leader of a lab doing interoperability between EU/Canada AI standards
 • ai policy fellow at US Senate working on biotech strategies
 • executive director of an ai safety coworking space who has been running weekly meetups for ~2.5 years
 • startup founder in stealth who asked not to share details with anyone outside CAISI
 • chemistry olympiad gold medalist working on a dangerous capabilities evals project for o3
 • mats alumni working on jailbreak mitigation at an ai safety & security org
 • ai safety research lead running a mechinterp reading group and interning at EleuthrAI

Some random brief thoughts:
• CAISI's focus seems to be on stuff other than x-risks (i.e, misinformation, healthcare, privacy).
 • I'm afraid of being too unfiltered and causing offence.
 • Some of the statements made in the interviews are bizarrely devoid of content, such as:

"AI safety work is not only a necessity to protect our social advances, but also essential for AI itself to remain a meaningful technology."

 • Others seem to be false as stated, such as:

 "our research on privacy-preserving AI led us to research machine unlearning — how to remove data from AI systems — which is now an essential consideration for deploying large-scale AI systems like chatbots."

 • (I think a lot of unlearning research is bullshit, but besides that, is anyone deploying large models doing unlearning?)
 • The UK AISI research agendas seemed a lot more coherent with better developed proposals and theories of impact.
 • They're only recruiting for 3 positions for a research council that meets once a month?
 • CAD 27m of CAISI's initial funding is ~15% of the UK AISI's GBP 100m initial funding, but more than the U.S AISI's initial funding (USD $10m).
 • Another source says $50m CAD, but that's distributed over 5 years compared to a $2.4b budget for AI in general, so about 2% of the AI budget goes to safety?
 • I was looking for scientific advancements which would be relevant at the national scale. I read through every page of anthropic/redwood's alignment faking paper, which is considered the best empirical alignment research paper of 2024, but it was a firehose of info and I don't have clear recommendations that can be put into a slide deck.
 • Instead of learning more about what other people were doing on a shallow level it might've been more beneficial to focus on my own research questions or practice training project relevant skills.

Yes. Though notably, if your employees were 10x faster you might want to adjust your workflows to have them spend less time being bottlenecked on compute if that is possible. (And this sort of adaption is included in what I mean.)

The uncomputability of AIXI is a bigger problem than this post makes it out to be. This uncomputability inserts a contradiction into any proof that relies on AIXI -- the same contradiction as in Goedel's Theorem. You can get around this contradiction instead by using approximations of AIXI, but the resulting proofs will be specific to those approximations, and you would need to prove additional theorems to transfer results between the approximations.