LessWrong 2.0 Reader

Curious,

 Such extreme speedups are only possible if acceleration is near universal, i.e. only if every tiny detail of the R&D process is amenable to acceleration.

is it possible that AIs can just discover more efficient methods altogether? AlphaFold's real results also seem like a bet against Amdhal's law but it simply side stepped the standard methods for protein folding discovery.

arguing against having probabilistic beliefs about events which are unprecedented

Sorry, I'm definitely not saying this. First, in the linked post (see here [LW · GW]), I argue that our beliefs should still be probabilistic, just imprecisely so. Second, I'm not drawing a sharp line between "precedented" and "unprecedented." My point is: Intuitions are only as reliable as the mechanisms that generate them. And given the sparsity of feedback loops^[1] and unusual complexity here, I don't see why the mechanisms generating AGI/ASI forecasting intuitions would be truth-tracking to a high degree of precision. (Cf. Violet Hour's discussion in Sec. 3 here [EA · GW].)

the level of precedentedness is continous

Right, and that's consistent with my view. I'm saying, roughly, the degree of imprecision (/width of the interval-valued credence) should increase continuously with the depth of unprecedentedness, among other things.

 

forecasters have successfully done OK at predicting increasingly unprecedented events

As I note here [LW · GW], our direct evidence only tells us (at best) that people can successfully forecast up to some degree of precision, in some domains. How we ought to extrapolate from this to the case of AGI/ASI forecasting is very underdetermined.

1. ^{^}

   On the actual information of interest (i.e. information about AGI/ASI), that is, not just proxies like forecasting progress in weaker or narrower AI.

And how exactly are universities a good signifier of that? Note I took an external degree from the university of London, which even if universities were a good signifier of that, this one definitely wasn't, and it did not in any way impact my ability to get a job. Noone cared.

As stated in the OP, I expect this to be the end result of the regulations I suggest. The advantage of this approach is that for now MIT can carry on doing it's thing instead of forcing a hard switch over where you stop it being able to assess is students without yet having an equivalently respected replacement.

We have artificial intelligence trained on decades worth of stories about misaligned, maleficent artificial intelligence that attempts violent takeover and world domination.

Thanks for explaining. I now agree that the current cost of inference isn't a very good anchor for future costs in slowdown timelines.

I'm uncertain, but I still think OpenAI is likely to go bankrupt in slowdown timelines. Here are some related thoughts:

1. OpenAI probably won't pivot to the slowdown in time.
   1. They'd have < 3 years to do before running out of money.
   2. Budgets are set in advance. So they'd have even less time.
   3. All of the improvements you list cost time and money. So they'd need to continue spending on R&D, before that R&D has improved their cost of inference. In practice, they'd need to stop pushing the frontier even earlier, to have more time and money available.
   4. There's not many generations of frontier models left before Altman would need to halt scaling R&D.
   5. Altman is currently be racing to AGI; and I don't think it's possible, on the slowdown hypothetical, for him to get enough evidence to convince him to stop in time.
2. Revenue (prices) may scale down alongside the cost of inference
   1. Under perfect competition, widely-shared improvements in the production of a commodity will result in price decreases rather than profit increases.
   2. There are various ways competition here is imperfect; but I think that imperfect competition benefits Google more than OpenAI. That's really bad, since OpenAI's finances are also much worse.
   3. This still makes the cost of inference/cost of revenue estimates wrong; but OpenAI might not be able to make enough money to cover their debt and what you called "essential R&D". Dunno.
3. Everybody but Anthropic are already locked in to much of their inference (and R&D) spending via capex.
   1. AI 2027 assumes that OpenAI uses different chips specialized in training vs. inference.
   2. The cost of the datacenter and the GPU's it contains is fixed, and I believe it makes up most of the cost of inference today. OpenAI, via Project Stargate, is switching from renting GPU's to building its own datacenters. (This may or may not bring down the cost of inference on its own, depending on what kind of discount OpenAI got from Azure.)
   3. So for inference improvements to be a pure win, OpenAI's usage needs to grow to the same extent. But in this scenario capabilities have stopped improving, so it probably won't. Usage might even shrink if some people adopt AI in preparation for future capability improvements.
   4. How to account for capex seems complicated:
      1. We need to amortize the cost of the GPU's themselves. GPU's in heavy usage break down over time; OpenAI would need to make enough of a profit to repay the loans and to buy replacement GPU's. (If they have lower utilization, their existing stock of GPU's will last longer.) Also, if semiconductor progress continues it will make their existing GPU's obsolete. I'm skeptical that semiconductor progress will continue at the same rate; but if it does, OpenAI will be locked into using obsolete and inefficient GPU's.
      2. Project Stargate is planning on spending 100 billion at first, 50 billion of which would be debt. CoreWeave is paying 11-14% annual interest on its debt; if OpenAI pays 10%, it would be locked into paying ~5 billion/year in interest. This, plus essential R&D, inference electricity, revenue share, and other costs, might or might not be doable; however, they'd almost certainly not have much money left over to buy new GPU's, save up for a balloon payment, or make a profit.

Best of Less Wrong 2023?

Guys, we're well into 2025 now.

I agree that finances are important to consider. I've written my thoughts on them here [LW · GW]; I disagree with you in a few places.

(1) Given Altman's successful ouster of the OpenAI board, his investors currently don't have much drive/desire/will to force him to stop racing. They don't have much time to do so on the current pace of increasing spending before OpenAI runs out of money.

(2) It's not clear what would boost revenue that they're not already doing; the main way to improve profits would just be to slash R&D spending. Much of R&D spending is spent on research compute; OpenAI is intending to own its own datacenters, so it's not clear whether they meaningfully can switch paths quickly.

(3) OpenAI is at a massive structural disadvantage to the rest of the frontier companies: they send 20% of their revenue to Microsoft, and they're taking on tens of billions in debt, which will need to be repaid with interest. So it's unlikely that they'll ever be profitable.

The overwhelming priority for EAs should be to serve as peace emissaries and advocates in the culture war.

War is an ethical inversion: acts that are plainly evil in peacetime become heroic and praiseworthy in times of war. War destroys value at a double rate: the warrior's honor is to accept pain, suffering, and the risk of death in order to inflict those ills on his enemies. Further, in wartime the opposing parties cannot pause their hostilities to make side deals, even if those deals would be mutually beneficial (the Russians and Ukrainians are not going to pause the fighting to ink a trade deal or build a gas pipeline, even if those plans were beneficial to both sides).

If an EA could go back in time to the era of the European wars of religion, it is obvious that the ethically correct strategy would be to attempt to promote peace and understanding between Protestants and Catholics (it would be comically evil and irrational to pick a side and wage war against the other). Our current culture war of Woke vs MAGA is just another instance of this type of conflict and we must draw the same ethical conclusion.

Putting a finite value on both an infinite lifespan of infinite pleasure and an infinite lifespan of torture allows people to avoid difficult decisions in utility maximization such as Pascal's Mugging.

Maybe this is why so many people seem to naively express that they don't actually want to live forever because they would get lonely and all their friends would die and etc. They're actually enacting a smart strategy which provides protection from edge case situations. This strategy also benefits from having a low cost of analysis.