LessWrong 2.0 Reader

I've tried speaking with a few teams doing AI safety work, including:
 • assistant professor leading an alignment research group at a top university who is starting a new AI safety org
 • anthropic independent contractor who has coauthored papers with the alignment science team
 • senior manager at nvidia working on LLM safety (NeMo-Aligner/NeMo-Guardrails)
 • leader of a lab doing interoperability between EU/Canada AI standards
 • ai policy fellow at US Senate working on biotech strategies
 • executive director of an ai safety coworking space who has been running weekly meetups for ~2.5 years
 • startup founder in stealth who asked not to share details with anyone outside CAISI
 • chemistry olympiad gold medalist working on a dangerous capabilities evals project for o3
 • mats alumni working on jailbreak mitigation at an ai safety & security org
 • ai safety research lead running a mechinterp reading group and interning at EleuthrAI

Some random brief thoughts:
• CAISI's focus seems to be on stuff other than x-risks (i.e, misinformation, healthcare, privacy).
 • I'm afraid of being too unfiltered and causing offence.
 • Some of the statements made in the interviews are bizarrely devoid of content, such as:

"AI safety work is not only a necessity to protect our social advances, but also essential for AI itself to remain a meaningful technology."

 • Others seem to be false as stated, such as:

 "our research on privacy-preserving AI led us to research machine unlearning — how to remove data from AI systems — which is now an essential consideration for deploying large-scale AI systems like chatbots."

 • (I think a lot of unlearning research is bullshit, but besides that, is anyone deploying large models doing unlearning?)
 • The UK AISI research agendas seemed a lot more coherent with better developed proposals and theories of impact.
 • They're only recruiting for 3 positions for a research council that meets once a month?
 • $27M of CAISI's initial funding is ~15% of the UK AISI's GBP 100m initial funding
 • Another source says $50m CAD, but that's distributed over 5 years compared to a $2.4b budget for AI in general, so about 2% of the AI budget goes to safety?
 • I was looking for scientific advancements which would be relevant at the national scale. I read through every page of anthropic/redwood's alignment faking paper, which is considered the best empirical alignment research paper of 2024, but it was a firehose of info and I don't have clear recommendations that can be put into a slide deck.
 • Instead of learning more about what other people were doing on a shallow level it might've been more beneficial to focus on my own research questions or practice training project relevant skills.

Yes. Though notably, if your employees were 10x faster you might want to adjust your workflows to have them spend less time being bottlenecked on compute if that is possible. (And this sort of adaption is included in what I mean.)

The uncomputability of AIXI is a bigger problem than this post makes it out to be. This uncomputability inserts a contradiction into any proof that relies on AIXI -- the same contradiction as in Goedel's Theorem. You can get around this contradiction instead by using approximations of AIXI, but the resulting proofs will be specific to those approximations, and you would need to prove additional theorems to transfer results between the approximations.

I think I misunderstood what you were saying there - I interpreted it as something like

Currently, ML-capable software developers are quite expensive relative to the cost of compute. Additionally, many small experiments provide more novel and useful insights than a few large experiments. The top practically-useful LLM costs about 1% as much per hour to run as a ML-capable software developer, and that 100x decrease in cost and the corresponding switch to many small-scale experiments would likely result in at least a 10x increase in the speed at which novel, useful insights were generated.

But on closer reading I see you said (emphasis mine)

I was trying to argue (among other things) that scaling up basically current methods could result in an increase in productivity among OpenAI capabilities researchers at least equivalent to the productivity you'd get as if the human employees operated 10x faster. (In other words, 10x'ing this labor input.)

So if the employees spend 50% of their time waiting on training runs which are bottlenecked on company-wide availability of compute resources, and 50% of their time writing code, 10xing their labor input (i.e. the speed at which they write code) would result in about an 80% increase in their labor output. Which, to your point, does seem plausible.

I think a lot of thinking around multipolar scenarios suffers from heuristic "solution in the shape of the problem", i.e. "multipolar scenario is when we have kinda aligned AI, but still die due to coordination failures, therefore, solution for multipolar scenarios should be about coordination".

I think the correct solution is to leverage available superintelligence in nice unilateral way:

1. D/acc - use superintelligence to put as much defence as you can, starting from formal software verification and ending in spreading biodefence nanotech;
2. Running away - if you set up Moon/Mars/Jovian colony of nanotech-upgraded humans/uploads and pour available resources into defence, even if Earth explodes, humanity as a species survives. 

Do you have any thoughts on mechanism & whether prevention is actually worse independent of inconvenience?

And I've received an email from Mieux Donner confirming Lucie's leg has been executed for 1,000€. Thanks to everyone involved!

If if anyone else is interested in a similar donation swap, from either side, I'd be excited to introduce people or maybe even do this trick again :D

I have more reasons for believing that Mensa members are below 130, but also for believing that they're above.

Below: Most online IQ tests are similar enough to the Mensa IQ test that the practice effect applies. And most people who obsess about their IQ scores probably take a lot of online IQ tests, memorizing most patterns (there's a limit to the practice effect, but it can still give you at least 10 points)

Above: Mensa tests for pattern recognition abilities, which in my experience correlates worse with academic performance than verbal abilities. Pattern recognition abilities also select for people with autism (they tend to score about 20 points higher on RPM-like pattern recognition tests (matrices) than on other subtests). These people will be smarter than they sound, because their low verbal abilities makes them appear stupid, even though their pattern recognition might be 2 standard deviations higher. So you get intelligent people with poor social skills, who sound much dumber than they are, and who tend to have more diagnoses than just autism. It's no wonder that these people go to forums like Mensa, or that they're less successful in life than their IQ would suggest. These people are also incredibly easy targets by the kind of people who go to r/iamverysmart so it's easy to build the public consensus that they're actually stupid, even when it isn't true.

However, in order for high intelligence to shine (and have worthy insights) even without formal education, IQs above 150 are likely needed. For in order to generate your own ideas and still be able to compete with the consensus (which is largely based off the theories of genuises like Tesla, Einstein, Neumann, Turing, Pavlov, etc.) you need to discover similar things yourself independently.

I think many rationalists are above 130. I don't like rationalist mentalities very much though. They seem to think that everything needs to have a source or a proof (a projected lack of confidence in their own discernment). They also tend to overestimate the value of knowledge (even sometimes using it as a synonym of intelligence). If somebodies IQ is, say, 110, I don't think they will ever have any great takes (even with years of studies) which a 140 IQ person couldn't run circles around given a week or two of thoughts. Ever seen somebody invest their whole life into something that you could dismantle or do better in 5 minutes? You could look at this and go "Rapid feedback is better because you approximate reality and update your beliefs faster, makes sense, but why overcompl- right, it's to make mone- to legitimize the only position in which they are thought to have value - because agile coaches are selling ideas/theory and rely on the illusion of substance of course"

Minor would count.

However, I don't view safe tiling as the primary obstacle to alignment. Constructing even a modestly superhuman agent which is aligned to human values would put us in a drastically stronger position and currently seems out of reach. If necessary, we might like that agent to recursively self-improve safely, but that is an additional and distinct obstacle. It is not clear that we need to deal with recursive self-improvement below human level.

I am not sure that treating recursive self-improvement via tiling frameworks is necessarily a good idea, but setting this aspect aside, one obvious weakness with this argument is that it mentions a superhuman case and a below human level case, but it does not mention the approximately human level case.

And it is precisely the approximately human level case where we have a lot to say about recursive self-improvement, and where it feels that avoiding this set of considerations would be rather difficult.

1. Humans often try to self-improve, and human-level software will have advantage over humans at that.

Humans are self-improving in the cognitive sense by shaping their learning experiences, and also by controlling their nutrition and various psychoactive factors modulating cognition. The desire to become smarter and to improve various thinking skills is very common.

Human-level software would have great advantage over humans at this, because it can hack at its own internals with great precision at the finest resolution and because it can do so in a reversible fashion (on a copy, or after making a backup), and so can do it in a relatively safe manner (whereas a human has difficulty hacking their own internals with required precision and is also taking huge personal risks if hacking is sufficiently radical).

2. Collective/multi-agent aspects are likely to be very important.

People are already talking about possibilities of "hiring human-level artificial software engineers" (and, by extension, human-level artificial AI researchers). The wisdom of having an agent form-factor here is highly questionable, but setting this aspect aside and focusing only on technical feasibility, we see the following.

One can hire multiple artificial software engineers with long-term persistence (of features, memory, state, and focus) into an existing team of human engineers. Some of those teams will work on making next generations of better artificial software engineers (and artificial AI researchers). So now we are talking about mixed teams with human and artificial members.

By definition, we can say that those artificial software engineers and artificial AI researchers have reached human level, if a team of those entities would be able to fruitfully work on the next generation of artificial software engineers and artificial AI researchers even in the absence of any human team members.

This multi-agent setup is even more important than individual self-improvement, because this is what the mainstream trend might actually be leaning towards, judging by some recent discussions. Here we are talking about a multi-agent setup, and about recursive self-improvement of the community of agents, rather than focusing on self-improvement of individual agents.

3. Current self-improvement attempts.

We actually do see a lot of experiments with various forms of recursive self-improvements even at the current below human level. We are just lucky that all those attempts have been saturating at the reasonable levels so far.

We currently don't have good enough understanding to predict when they stop saturating, and what would the dynamics be when they stop saturating. But self-improvement by a community of approximately human-level artificial software engineers and artificial AI researchers competitive with top human software engineers and top human AI researcher seems unlikely to saturate (or, at least, we should seriously consider the possibility that it won't saturate).

4. At the same time, the key difficulties of AI existential safety are tightly linked to recursive self-modifications.

The most intractable aspect of the whole thing is how to preserve any properties indefinitely through radical self-modifications. I think this is the central difficulty of AI existential safety. Things will change unpredictably. How can one shape this unpredictable evolution so that some desirable invariants do hold?

These invariants would be invariant properties of the whole ecosystem, not of individual agents; they would be the properties of a rapidly changing world, not of a particular single system (unless one is talking about a singleton which is very much in control of everything). This seems to be quite central to our overall difficulty with AI existential safety.