LessWrong 2.0 Reader

The extensive discussion of trends in global datacenter/Nvidia revenue shows that the framing considers human economy as a whole as the system driving eventual AI takeoff, that there are always essential complementary inputs that can't be abstracted out.

Software-only singularity is about considering scaling laws for a different system [LW(p) · GW(p)] that is not the entire economy and whose relevant inputs are specific AIs (varying in their capabilities and compute efficiency) and the novel cultural knowledge they are producing rather than more material forms of capital or compute. An intermediate construction is an isolated AI/robot economy that's highly decoupled from the human economy and does its own thing at its own pace.

Early trends of an algal bloom shouldn't be about the total mass of organic matter in the ocean. The choice of the system to consider relevant carries more of the argument than detailed analyses of any given system. In the post, Ege Erdil makes a point that we know very little about the system where a possible software-only singularity takes place:

It’s just hard to be convinced in a domain where the key questions about the complexity of the job of a researcher and the complementarity between cognitive and compute/data inputs remain unanswered.

This is a reason disagreement about which systems are relevant persists, as those who see recursive self-improvement as possible and therefore a relevant system will fail to convince those who don't. But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.

In the current regime, pretraining scaling laws tether AI capabilities to compute of a single training system, but not to the total amount of compute (or revenue) in datacenters worldwide. This in turn translates to relevance of finances of individual AI companies and hardware improvements, which will remain similarly crucial if long reasoning training takes over from pretraining, the difference being that AI company money will be buying inference compute for RL training from many datacenters, rather than time on a single large training system. A pivot to RL lifts some practical constraints on the extent of scaling, and the need to coordinate construction of increasingly large and expensive training systems that are suboptimal for other purposes, which might let the current scaling regime extend for another 3-4 years, until 2030-2032.

I think something like "alignment features" are plausibly a huge part of the story for why AI goes well. 

At least, I think it is refreshing to take the x-risk goggles off for a second sometimes and remember that there is actually a huge business incentive to eg. solve "indirect prompt injections", perfect robust AI decision making in high stakes contexts, or find the holy grail of compute-scalable oversight. 

Like, a lot of times there seems genuine ambiguity and overlap b/w "safety" research and normal AI research. The clean "capabilities"/"alignment" distinction is more map than territory sometimes.

Also, isn't this already basically a thing?? Companies already compete to have the "special sauce"; a lot of this is post-training stuff, so massively overlaps with "alignment-coded" stuff. When does the RL post training go from being safety to "special sauce" to "alignment feature" y'know?

I agree with the top part. I think it's naive to believe that AI is helping anyone, but what I want to talk about is why this problem might be unsolvable (except by avoiding it entirely). 

If you hate something and attempt to combat it, you will get closer to it rather than further away, in the manner which people refer to when they say "You actually love what you say you hate". When I say "don't think about pink elephants", the more you try, the more you will fail, and this is because the brain doesn't have subtraction and division, but only addition and multiplication.

You cannot learn about how to defend yourself against a problem without learning how to also cause the problem. When you learn self-defense you will also learn attacks. You cannot learn how to argue effectively with people who hold stupid worldviews without first understanding them and thus creating a model of the worldview within yourself as well.

Due to mechanics like these, it may be impossible to research "AI safety" in isolation. It's probably better to use a neutral word like "AI capabilities" which include both the capacity for harm and defense against harm so that we don't mislead ourselves with words. It can cause untold damage, much like viewing "good and evil" as opposites, rather than two sides of the same thing, has.

I also want to warn everyone that there seems to be an asymmetry in warfare which makes it so that attacking is strictly easier than defending. This ratio seems to increase as technology improves.

If you are $p_2$, you can change logic itself such that $p_2$, despite its source code, somehow outputs "take 1 box." If you are the programmer, you can't do that.

Following news of Anthropic allowing Claude to decide to terminate conversations, I find myself thinking about when Microsoft did the same with the misaligned Sydney in Bing Chat.

You can freely wait a year before replying.

Or more! (I was delighted to receive this reply.)

I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments.

I kind of like feel the opposite way, in that a lot of people seem to think we'll have short timelines but the arguments for that seem weak! They seem to mostly be based on something like trend extrapolation and assuming that e.g. models getting improving scores on benchmarks means they're actually getting better on real-world tasks. E.g. somebody like Leopold Aschenbrenner will write that GPT-4 is "on the level of a smart high schooler" while at the same time, language models require extensive additional scaffolding [LW · GW] to even get started on a simple game like Pokemon (and none of them have managed to beat it yet). 

There seems to be a general and unjustified assumption that merely because language models perform on some specific narrow problems on the level of a "smart high schooler", you can say that they have that level of intelligence overall. But that seems clearly false, somewhat analogous to saying that a calculator is a superintelligence just because it's superhuman at quickly operating on large numbers. Rather, the AI we have so far seems to succeed at the kinds of things it's been specifically trained at, but fail to generalize [LW · GW] to more novel situations. People also haven't been able to point at much in the way of convincing novel discoveries [LW · GW] made by LLMs.

I asked for the strongest arguments in favor of short timelines [LW · GW] some time ago, and didn't feel like any of them were very compelling. By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years. But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability. o3 is said to perform "on par with elite human competitors" on CodeForces, but recent characterizations of its programming ability [LW · GW] are that while it produces code that "almost certainly works", that code is "verbose, brittle, hard to read". 

Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking [LW · GW] and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don't meet the spirit of it.

The benchmarks also do not take into account the fact that the vast majority of them measure a model's performance in a situation where the model is only given one task at a time, and it can completely focus on solving that. If you want the models to act agentically in situations where they have multiple conflicting priorities and need to decide what kinds of approaches to try, then you need them to have something like executive function. Much of Claude's difficulties in playing Pokemon come from severe problems with executive function [LW · GW]. 

This is an entirely separate capability that we're only now starting to make progress on, one which is more difficult to solve because the nature of good executive function is environmentally dependent rather than something with one objectively correct solution. E.g. sometimes it's good if you're able to determinedly work on a problem without getting frustrated or bored, but some of Claude's problems in playing Pokemon are because it doesn't ever get frustrated or bored, so keeps trying the same hopeless approaches over and over again. Sometimes getting utterly obsessed with something that seems unpromising makes you into Charles Darwin, sometimes it causes you to just obsess over something nobody else cares about for your whole life. That's one huge crucial capability that any real-world agents would need to be effective, that I've seen minimal discussion over.

Does any of this make me certain about longer timelines? No. AI progress has been surprisingly fast in the past, and maybe it will be that in the future too. But I don't see any obvious reason to be very confident in short ones, either.

(Also none of this is considering the possibility of increased regulation slowing down the pace of AI - we've already seen regulators starting to get nervous about AI, and widespread anti-AI sentiment.)

If many independent actors are working on AI capabilities, even if each team has decent safety intentions within their own project, is there a fundamental coordination problem that makes the overall landscape unsafe? A case where the sum of the whole is flawed, unsafe, and/or dangerous and thus doesn't equal collective safety?

while I think most people aren't quite tackling this with the particular taste I'd apply, it does sure seem like everyone is working on "do stuff with LLMs" and it's not where the underpicked fruit is

I disagree, I think pretty much nobody is attempting anything useful with LLM-based interfaces. Almost all projects I've seen in the wild are terrible and there are tons of unpicked low-hanging fruits.

I'd been thinking, on and off, about ways to speed up agent-foundations research using LLMs. An LLM-powered exploratory medium for mathematics is one possibility.

A big part of highly theoretical research is flipping between different representations of the problem: viewing it in terms of information theory, in terms of Bayesian probability, in terms of linear algebra; jumping from algebraic expressions to the visualizations of functions or to the nodes-and-edges graphs of the interactions between variables; et cetera.

The key reason behind it is that research heuristics bind to representations. E. g., suppose you're staring at some graph-theory problem. Certain problems of this type are isomorphic to linear-algebra problems, and they may be trivial in linear-algebra terms. But unless you actually project the problem into the linear-algebra ontology, you're not necessarily going to see the trivial solution when staring at the graph-theory representation. (Perhaps the obvious solution is to find the eigenvectors of the adjacency matrix of the graph – but when you're staring at a bunch of nodes connected by edges, that idea isn't obvious in that representation at all.)

This is a bit of a simplified example – the graph theory/linear algebra connection is well-known, so experienced mathematicians may be able to translate between those representations instinctively – but I hope it's illustrative.^[1]

As a different concrete example, consider John Wentworth's Bayes Net Algebra [LW · GW]. This is essentially an interface for working with factorizations of joint probability distributions. The nodes-and-edges representation is more intuitive and easy to tinker with than the "formulas" representation, which means that having concrete rules for tinkering with graph representations without committing errors would significantly speed up how quickly you can reason through related math problems. Imagine if the derivation of such frameworks was automated: if you could set up a joint PD in terms of formulas, automatically project the setup into graph terms, start tinkering with it by dragging nodes and edges around, and get errors if and only if back-projecting the changed "graph" representation into the "formulas" representations results in a setup that's non-isomorphic to the initial one.

(See also this video, and the article linked above.)

A related challenge are refactors. E. g., suppose you're staring at some complicated algebraic expression with an infinite sum. It may be the case that a certain no-loss-of-generality change of variables would easily collapse that expression into a Fourier series, or make some Obscure Theorem #418152/Weird Trick #3475 trivially applicable. But unless you happen to be looking at the problem through those lens, you're not going to be able to spot it. (Especially if you don't know the Obscure Theorem #418152/Weird Trick #3475.)

It's plausible that the above two tasks is what 90% of math research consists of (the "normal-science" part of it), in terms of time expenditure. Flipping between representations in search of a representation-chain where every step is trivial.

Those problems would be ameliorated by (1) reducing the friction costs of flipping between representations, and (2) being able to set up an automated searches for simplifying refactors of the problem.

Can LLMs help with (1)? Maybe. They can write code and they can, more or less, reason mathematically, as long as you're not asking them for anything creative. One issue is that they're also really sloppy and deceptive when writing proofs... But that problem can potentially be ameliorated by fine-tuning e. g. r1 to justify all its conclusions using rigorous Lean code, which could be passed to automated proof-checkers before being shown to you.^[2]

Can LLMs help with (2)? Maybe. I'm thinking something like the Pantheon interface [LW · GW], where you're working through the problem on your own, and in a side window LLMs offer random ideas regarding how to simplify the problem.

LLMs have bad research taste, which would extend to figuring out what refactorings they should try. But they also have a superhuman breadth of knowledge regarding theorems/math results. A depths-first search might thus be productive here. Most of LLM suggestions would be trash, but as long as complete nonsense is screened off by proof-checkers, and the ideas are represented in a quickly-checkable manner (e. g., equipped with one-sentence summaries), and we're giving LLMs an open-ended task, some results may be useful.

I expect I'd pay $200-$500/month for a working, competently executed tool of this form; even more the more flexible it is. I expect plenty of research mathematicians (not only agent-foundations folks) would, as well. There's a lucrative startup opportunity there.

@johnswentworth [LW · GW], any thoughts?

1. ^{^}

   A more realistic example would concern ansatzes, i. e., various "weird tricks" for working through problems. They likewise bind to representations, such that the idea of using one would only occur to you if you're staring at a specific representation of the problem, and would fail to occur if you're staring at an isomorphic-but-shallowly-different representation.

2. ^{^}

   Or using o3 with a system prompt where you yell at it a lot to produce rigorous Lean code, with a proof-checker that returns errors if it ever uses a placeholder always-passes "sorry" expression. But I don't know whether you can yell at it loudly enough using just the system prompt, and this latest generation of LLMs seems really into Goodharting, so it might straight-up try to exploit bugs in your proof-checker.

I still want something even closer to Givewell but for AI Safety (though it is easier to find where to donate now than before). Hell, I wouldn't mind if LW itself had recommended charities in a prominent place (though I guess LW now mostly asks for Lightcone donations instead).