LessWrong 2.0 Reader

Yeah it’s fine to assume that there might be some period of time that (1) the AGIs don’t escape control, (2) the code doesn’t leak or get stolen, (3) nobody else reinvents the same thing, (4) Company A doesn’t have infinite capital (yet) to spend on renting cloud compute (or the contracts haven’t yet been signed or whatever). And it’s fine to be curious about how many AGIs would Company A have available during this period of time.

And then a key question is whether anything happens during that period of time that would change what happens after that period of time. (And if not, then the analysis isn’t too important.) A pivotal act would certainly qualify. I’m kinda cynical in this area; I think the most likely scenario by far is that nothing happens during this period that has an appreciable impact on what happens afterwards. Like, I’m sure that Company A try to get their AGIs to beat benchmarks, do scientific research, make money, etc. I also expect them to have lots of very serious meetings, both internally and with government officials. But I don’t expect that Company A would succeed at making the world resilient to future out-of-control AGIs, because that’s just a crazy hard thing to do even with millions of intent-aligned AGIs at your disposal. I discussed some of the practical challenges at What does it take to defend the world against out-of-control AGIs? [LW · GW].

Well anyway. My comment above was just saying that the OP could be clearer on what they’re trying to estimate, not that they’re wrong to be trying to estimate it.  :)

interesting reading this 3 years later. I occasionally paste a bug report directly into cursor and provided I am right about which file the bug is in, it often one-shots them. i remain confused about why rsi isn't critical by now  

Yes. This is not unusually bad for a medical paper but that's not exactly a defense. 

I'd be fascinated by having a conversation about why 1e14 FLOP/s might be a better estimate.

I think I don’t want to share anything publicly beyond what I wrote in Section 3 here [LW · GW]. ¯\_(ツ)_/¯

For longer term brain processes, you need to take into account fractional shares of relatively-slow-but-high-complexity processes

Yeah I’ve written about that too (here [LW · GW]). :)  I think that’s much more relevant to how hard it is to create AGI rather than how hard it is to run AGI.

But also, I think it’s easy to intuitively mix up “complexity” with “not-knowing-what’s-going-on”. Like, check out this code, part of an AlphaZero-chess clone project. Imagine knowing nothing about chess, and just looking at a minified (or compiled) version of that code. It would feel like an extraordinarily complex, inscrutable, mess. But if you do know how chess works and you’re trying to write that code in the first place, no problem, it’s a few days of work to get it basically up and running. And it would no longer feel very complex to you, because you would have a framework for understanding it.

By analogy, if we don’t know what all the protein cascades etc. are doing in the brain, then they feel like an extraordinarily complex, inscrutable, mess. But if you have a framework for understanding them, and you’re writing code that does the same thing (e.g. sets certain types of long-term memory traces in certain conditions, or increments a counter variable, or whatever) in your AGI, then that code-writing task might feel pretty straightforward.

Waterloo, Ontario
December 14th, 6:00pm.
Event link: https://www.lesswrong.com/events/LBZGbJRnsuqGP7Mnh/waterloo-solstice-2024 [? · GW]

one data-point: i have been generating songs with lyrics I like and it’s most of my music consumption now.

the song you generated has a slop vibe i am not a fan of - but we are all wireheaded in different ways. however, if I generate hundreds of songs I usually get what I want. focusing on simple lyrics helps a lot and “no autotune” in the prompt helps too.

>As a community we produce more way more breastmilk than we can use!
This doesn't really seem right to me; or at least it relies on mothers' volunteer work to pump, sterilize, and store their milk. If you actually need to get rid of extra milk, pumping and dumping is way easier than keeping the milk clean and cold. And if you have an oversupply, pumping a lot is how to continue having an oversupply.

This is sort of like claims that we could produce lots of vegetables if everyone turned their front yard into a miniature farm and spent their spare time doing subsistence agriculture; technically true but not how most people want to spend their time.

Let’s say Company A can make AGIs that are drop-in replacements for highly-skilled humans at any existing remote job (including e.g. “company founder”), and no other company can. And Company C is a cloud provider. Then Company A will be able to outbid every other company for Company C’s cloud compute, since Company A is able to turn cloud compute directly into massive revenue. It can just buy more and more cloud compute from C and every other company, funding itself with rapid exponential growth, until the whole world is saturated.

I think this is outside the timeline under consideration. Transforming compute into massive revenue is still gated by the ability of non-AGI enabled customers to decide to spend with Company A; regardless of price the ability of Company C to make more compute available to sell depends quite a bit on the timelines of their contracts with other companies, etc. The ability to outbid the whole rest of the world for commercial compute already crosses the transformational threshold, I claim. This remains true regardless of whether it is a single dominant bidder or several.

I think the timeline we are looking at is from initial launch through the first round of compute-buy. This still leaves all normal customers of compute as bidders, so I would expect the amount of additional compute going to AGI to be a small fraction of the total.

Though let the record reflect based on the other details in the estimate this could still be an enormous increase in the population.

Other health claims: breastfeeding slightly reduces risk of breast cancer in the mother and increases chance of colorectal cancer and breast cancer in the child.

I think I'm also learning that people are way more interested in this detail than I expected! 

I debated changing it to "203X" when posting to avoid this becoming the focus of the discussion but figured, "eh, keep it as I actually wrote it in the workshop" for good epistemic hygiene.