LessWrong 2.0 Reader

That's true. Would be nice if I could also discuss ideas like this one here in a friendly way. Don't always have time to go to in person stuff. https://www.lesswrong.com/posts/7LFbhL5vfSbTAFLmR/family-line-selection-optimizer [LW · GW]

The intuition is that if we both saw bus 1546, and you guessed that there were 1546 buses and I guessed that there were 1547, you would be a little more likely to be correct but I would almost certainly be closer to the real number.

The Bayesian update isn't generally well-defined because you get a divergent mean. Your implicit prior is  1/n which is an improper prior. This is fine for deriving a posterior median, which in this case happens to be about 3,100 buses, and a posterior distribution, which in this case is a truncated zeta distribution with s=2 and k=1546. But the posterior mean does not exist.

In essence, this is saying that if the pace of progress is the product of two factors (experiment implementation time, and quality of experiment choice), then AI only needs to accelerate one factor in order to achieve an overall speedup. However, AI R&D involves a large number of heterogeneous activities, and overall progress is not simply the product of progress in each activity. Not all bottlenecks will be easily compensated for or worked around.

I agree with this.

I also think that there are some engineering/infrastructure challenges to executing training runs, that one would not necessarily cede to AI, not because it may not be desirable, but because it would involve a level of embodiment that is likely beyond the timeline proposed in the AI 2027 thesis. (I do agree with most of the thesis however).  

I'm not sure there's a research basis (that I could find at least, though I am very open to correction on this point), for embodiment of AI systems (robotic bodies) being able to keep pace with algorithmic improvement. 

While an AI system could likely design a new model architecture, and training architecture, it comes down to very human supply chain and technician speed that enables that physical training to be run at the scales required.  

Further, there are hardware challenges to large training runs of AI systems, which may not be resolvable by an AI system as readily, due to lack of exposure to those kinds of physical issues in their inherent reasoning space. (They have never opened a server during a training run, and resolved an overheat issue for instance). 

Some oft overlooked items involved in training, are based on the fact that the labs tend to not own their own data centers but rather rely on cloud providers. This means they have to contend with: 

• Cluster allocation: Scheduling the time on thousands of GPUs across multiple cloud providers, and reserving time blocks, securing budget, etc. I can easily buy the concept of an AI system recursively self-improving on a baked in infrastructure, but the speed with which its human colleagues may be able to secure additional infrastructure for it may be challenging. I understand that in the article that the model has taken over the 'day to day' operations, but I'm not sure I characterize a significant training run as a 'day to day' activity. This scheduling goes beyond just 'calling some colos', and involves potentially running additional power and fiber to buildings, construction schedules, etc.  
• Topology: Someone has to physically lay out the training network used. This goes beyond the networking per se, but also involves actually moving hardware around, building in redundancies in the data hall (extra PDUs, etc.), running networking cable, and putting mitigations in place for transients, etc. This all requires technicians, parts, hardware, etc. Lead times in some cases for those parts exceed the timeline to ASI proposed.
• Hardware/Firmware Validation: People physically have to check the server infrastructure, the hardware and the firmware, and ensure that all of the cards are up to date, etc. Moving at speed in AI, a lot of 'second hand', or 'relocated' servers and infrastructure tend to be used. It is not a small task to catalogue all of that and place it into a DCIM framework.
• Stress Testing: Running large power loads to check thermal limits, power draw, and inter-GPU comms. Parts fail here routinely, requiring replacement, etc.
• Power: Assuming that in the proposed timeline, compute remains linked to power, we are looking at a generational data center capability issue.
  • The data centers under construction now, set to go on-line in the 2029/2030 timeline, will be the first to use Blackwell GPUs at scale.
  • This then implies that to achieve the 2027 timeline, we'll be able to stretch Hoppers and existing power infrastructure to the point that these improvements emerge out of existing physical hardware.
  • I do tend to agree that if we were unconstrained by power, and physical infrastructure, that algorithmically there is no reason at all to believe that we could not achieve ASI by 2027 - however the infrastructure challenges are absolutely enormous.
  • Land with sufficient water, power (including natural gas lines for onsite power) isn't available. Utilities in the US are currently restricting power access to data centers, by leveraging significant power tariffs and long term take or pay commitments. (The AEP decision in Ohio for instance). This makes life harder for colocation providers in terms of financing and siting large infrastructure.

I believe that an AI system could well be match for the data cleaning and validation, and even the launch and orchestration using Slurm, Kubernetes or similar, but the initial launch phase is also something that I think will be slowed by the need for human hands. 

This phase results in: 

• Out of memory errors on GPUs, which can often only be resolved by 'turning a wrench' on the server.
• Unexpected Hardware failures (GPUs breaking, NVLinks breaking, Network timeouts, cabling issues, fiber optic degradation, power transcients, etc.) All of these require human technicians. 

These errors are also insidious, because the software running the training can't tell the impact of these failures on which parts of the network is being trained, and which isn't. This would make it challenging for an AI director to really understand what was causing issues in the desired training outcome. This makes it unlikely that a runaway situation would take place where a model is just recursively self-improving on a rapid timeline without human input, unless it first cracked the design, and mass manufacture of embodied AI workers that could move and act as quickly as it can. 

A good case study on this is the woes faced by OpenAI in training GPT-4.5, where all of this came to a head, taking a training run scheduled for a month or two, and stretching it over a year. OpenAI spoke very openly about this in a Youtube video they released. 

What's more, at scale, if we are going to be relying on existing data centers for a model of this sophistication, we'd have the model split across multiple clusters, potentially in multiple locations. This causes latency issues, etc. 

That's the part that to me is missing from the near-term timeline. I think the thesis around zones created just to build power and data centers, following ASI, seems very credible, especially with that level of infiltration of government/infrastructure. 

I don't however see a way of getting to a model capable of ASI with current data center infrastructure, prior to the largest new campuses coming online, and power running to Blackwell GPUs. 

While current AI models and tools are demonstrating substantial value in the real world, there is nevertheless a notorious gap between benchmark scores ("Ph.D level" and beyond) and real-world applicability. It strikes me as highly plausible that this reflects one or more as-yet-poorly-characterized chasms that may be difficult to cross.

You probably know this, but for onlookers the magnitude of these chasms are discussed in our timelines forecast, method 2.

Maybe just like hospitals have inpatients and outpatients, prisons can have outprisoners who wear a device which monitors them all the time, and might even restrain them if needed.

It may actually work, but of course, it's just a lil bit too tech dystopian to be politically viable.

The authors address this objection, but the counterargument strikes me as flawed. Here is the key paragraph:

To see why this is conceptually mistaken, consider a theoretical AI with very superhuman experiment selection capabilities but sub-human experiment implementation skills. Even if automation didn’t speed up implementation of AI experiments at all and implementation started as 50% of researchers’ time, if automation led to much better experiments being chosen, a >2x AI R&D progress multiplier could be achieved.

In essence, this is saying that if the pace of progress is the product of two factors (experiment implementation time, and quality of experiment choice), then AI only needs to accelerate one factor in order to achieve an overall speedup. However, AI R&D involves a large number of heterogeneous activities, and overall progress is not simply the product of progress in each activity. Not all bottlenecks will be easily compensated for or worked around.

Also remember that we are talking about very large speedups here. In practice, Amdahl's Law often starts to bite when optimizing a system by factors as small as 2x. Projecting speedups reaching 2000x is "pricing to perfection"; if the ability to route around difficult-to-automate activities is anything short of perfect, progress will fall short of the anticipated speedup curves.

Looking at the quote above, I'll note that "choosing better experiments" is a relatively advanced skill, which will likely not emerge until well after experiment implementation skills. More generally, the high-level decision-making skills needed to maintain accelerating progress in the face of powerful-but-uneven AI capabilities seem like they would not emerge until late in the game.

You're bringing up a more sophisitcated objection than the one I was addressing, which didn't acknowledge things like multiplier effects or being able to shift the task distribution.

Regarding research taste / experiment selection coming well after experiment implementation, I disagree in my median case depending on what you mean by "well after," due to intuitions I describe in my other comment [LW(p) · GW(p)]. Also I'd note that there are some early signs of research taste or management being fairly automatable (take these with pinches of salt, ofc there are limitations!).

Also, we think that research taste is a multiplier which isn't tied to many distinct sub-activities, curious if you could provide examples of several sub-activities without super correlated capabilities.

The model assumes very high speedup factors (25x to 250x), implying very broad and robust capabilities, quite far in advance of ASI.

FWIW, these are heavily informed by surveys of frontier AI researchers, including a more recent higher sample size survey that we haven't made public yet but gave similar results to are previous smaller ones (though the recent one was done quicker so may have more definitional confusion issues).

I feel really weird about this article, because I at least spiritually agree with the conclusion and my lower bound on AGI extinction scenario is "automated industry grows 40%/y and at first everybody becomes filthy rich and in next decade necrosphere devours everything" and this scenario seems to be weirdly underdiscussed relatively to "AGI develops self-sustaining supertech and kills everyone within one month" by both skeptics and doomers. 

Thus said, I disagree with almost everything else. 

You underestimate effect of RnD automation because you seem to ignore that a lot of bottlenecks in RnD and adoption of results are actually human bottlenecks. Like, even if we automate everything in RnD which is not free scientific creativity, we are still bottlenecked by facts that humans can keep in working memory only 10 things and that multitasking basically doesn't exist and that you can perform several hours of hard intellectual work per day maximum and that you need 28 years to grow scientist and that there is only so many people actually capable to produce science and that science progresses one funeral at time, et cetera, et cetera. The same with adoption: even if innovation exists, there is only so many people actually capable to understand it and implement while integrating into existing workflow in maintainable way and they have only so many productive working hours per day.

Even if we dial every other productivity factor into infinity, there is a still hard cap on productivity growth from inherent human limitations and population. When we achieve AI automation, we substitute these limitations with what is achievable inside physical limitations. 

Your assumption "everybody thinks that RnD is going to be automated first because RnD is mostly abstract reasoning" is wrong, because actual reason why RnD is going to be automated relatively early is because it's very valuable, especially if you count opportunity costs. Every time you pay AI researcher money to automate narrow task, you waste money which could be spent on creating an army on 100k artificial researchers who could automate 100k tasks. I think this holds even if you assume pretty long timelines, because w.r.t. of mentioned human bottlenecks everything you can automate before RnD automation is going to be automated rather poorly (and, unlike science, you can't run self-improvement from here).

Yeah, I was a little hand wavy with that, sorry. You're right about Sharpe scaling with (the square root of) number of trades, but that gets dampened a lot by pairwise correlation between trades and correlated leverage/financing costs. I worked at a hedge fund that had a lot of siloed strategies ("pods") and each one targeted a Sharpe of ~2, but then they would get sliced and diced into end products for investors with different risk levels etc. (I never worked on that side of the firm and tbh do not know all the details of how that works).

In practice, you actually care about incremental Sharpe, which you get by subtracting the Sharpe of your strategy minus the covariance-adjusted Sharpe of your existing portfolio over the norm of the covariance matrix, and you want to keep that pro-forma portfolio Sharpe above 2. It would be awesome to have 100 uncorrelated bets each with a 2 Sharpe, but in practice our strategy was so specific that a lot of our trades were still pretty correlated and created incremental transaction and hedging costs. And beyond being directly correlated (if position A goes up position B will also go up), the structure of a pod means you introduce auto-correlation (if our financing rate goes up all of our positions will have reduced Sharpe, or if one position blows out we will have to exit multiple other trades suboptimally) so you end up getting pretty high covariance matrix norms. The cutoff for a standalone position is maybe like 1.2-1.5 Sharpe, if that position is small enough relative to the overall portfolio that you can just think about marginal Sharpe as a first-order effect. So a little lower but still pretty non-trivial for an individual investor picking single stocks to achieve, and in the case of this discussion--a pretty concentrated, highly auto-correlated, long-only equity portfolio--I don't think it's unreasonable to just round off individual thresholds to 2.0 even if it's more like 1.8 or 1.9.

Scott Alexander had a nice piece on this:

When I was young I used to read pseudohistory books; Immanuel Velikovsky’s Ages in Chaos is a good example of the best this genre has to offer. I read it and it seemed so obviously correct, so perfect, that I could barely bring myself to bother to search out rebuttals.

And then I read the rebuttals, and they were so obviously correct, so devastating, that I couldn’t believe I had ever been so dumb as to believe Velikovsky.

And then I read the rebuttals to the rebuttals, and they were so obviously correct that I felt silly for ever doubting.

And so on for several more iterations, until the labyrinth of doubt seemed inescapable. What finally broke me out wasn’t so much the lucidity of the consensus view so much as starting to sample different crackpots. Some were almost as bright and rhetorically gifted as Velikovsky, all presented insurmountable evidence for their theories, and all had mutually exclusive ideas. After all, Noah’s Flood couldn’t have been a cultural memory both of the fall of Atlantis and of a change in the Earth’s orbit, let alone of a lost Ice Age civilization or of megatsunamis from a meteor strike. So given that at least some of those arguments are wrong and all seemed practically proven, I am obviously just gullible in the field of ancient history. Given a total lack of independent intellectual steering power and no desire to spend thirty years building an independent knowledge base of Near Eastern history, I choose to just accept the ideas of the prestigious people with professorships in Archaeology, rather than those of the universally reviled crackpots who write books about Venus being a comet.

You could consider this a form of epistemic learned helplessness, where I know any attempt to evaluate the arguments is just going to be a bad idea so I don’t even try. If you have a good argument that the Early Bronze Age worked completely differently from the way mainstream historians believe, I just don’t want to hear about it. If you insist on telling me anyway, I will nod, say that your argument makes complete sense, and then totally refuse to change my mind or admit even the slightest possibility that you might be right.

(This is the correct Bayesian action: if I know that a false argument sounds just as convincing as a true argument, argument convincingness provides no evidence either way. I should ignore it and stick with my prior.)

Maybe define them to be people who do not want to be a sex worker, even if they take into account the fact they might have no money otherwise.