LessWrong 2.0 Reader

Good work! A few questions:

1. Where do the edges you draw come from? IIUC, this method should result in a collection of features but not say what the edges between them are.
2. IIUC, the binary masking technique here is the same as the subnetwork probing baseline from the ACDC paper, where it seemed to work about as well as ACDC (which in turn works a bit worse than attribution patching). Do you know why you're finding something different here? Some ideas:
   1. The SP vs.  ACDC comparison from the ACDC paper wasn't really apples-to-apples because ACDC pruned edges whereas SP pruned nodes (and kept all edges betwen non-pruned nodes IIUC). If Syed et al. had compared attribution patching on nodes vs. subnetwork probing, they would have found that subnetwork probing was better.
   2. There's something special about SAE features which changes which subnetwork discovery technique works best.
      1. I'd be a bit interested in seeing your experiments repeated for finding subnetworks of neurons (instead of subnetworks of SAE features); does the comparison between attribution patching/integrated gradients and training a binary mask still hold in that case?

The $25-40bn figure is an estimate for about 1 GW worth of GB200s. SemiAnalysis expects 1 GW training systems for Google in 2025 and something comparable for Microsoft/OpenAI. This is discussed by Dylan Patel publicly on Dwarkesh Podcast, claiming that there is a 300K B200s cluster and 500K-700K B200s worth of compute in total currently being constructed, possibly networked into a single training system. So if planned Microsoft capex was $60bn, that would've been surprising, too little for this project without cutting something else, but $80bn fits this story, that's my takeaway.

With Stargate, $100bn is still too much for the training systems of 2024-2025, so it's either not about what's being built in 2024-2025 at all, or a larger project that has current activities as part (which wouldn't fit building a big training system using a specific generation of hardware). Musk's 100K H100s Colossus tells me that building a training system in a year is feasible, even though it normally takes longer. The preliminary steps (land, power, permits, buildings) are much cheaper, but securing power and permits can require starting years in advance. So talking about a $100bn Stargate in 2024 is consistent with building it in late 2026, once there is a plot with 3-5 GW of power and datacenter permits, most of the expense will then be in 2026 (Nvidia Rubin probably).

I mean humans with strong AGIs under their control might function as if they don't need sleep, might become immortal, will probably build up superhuman protections from assasination, etc

Different example - I said "instead"

If you look back, you'll see I was specifically responding to the hypothetical scenario about public admission in that comment. For your points about private shame, you might want to check my other comment replying to you [LW(p) · GW(p)] where I addressed how internal shame and self-image maintenance connect to social dynamics.

I notice you're attributing positions to me that I haven't taken and expressing confusion about points I've already addressed in detail. It would be helpful if you could engage more carefully with what I've carefully written.

so if the musician openly admits and apologize for only being average they are ashamed because they are afraid of the reaction of the fan who clearly loved their performance (not their failure to abstain from what they believe is the cause of their average performance?)

You're introducing new elements that weren't in your original scenario. But more importantly: you described the show as "a hit" where "everyone loves them." Calling this performance "only average" isn't revealing accurate adverse information - it's a lie.

but if they don't mention it to anyone (therefore are committing neither a dominance nor submission gesture) they are also ashamed?

In my other reply to you [LW(p) · GW(p)], I explained how private shame often involves maintaining conflicting mental models - one that enables confident performance and another that tracks specific flaws for improvement. Even when no one would directly know or care about staying up late drinking, the performer may feel shame because they've invested in an identity as a "professional musician" or "disciplined performer" - an identity that others care about and grant certain privileges to. The shame comes from violating the requirements of this identity, which serves as a proxy for social approval and professional opportunities. This creates internal pressure toward shame even without a specific idea of someone else who would directly condemn the behavior or trait in question.

Are you telling me there is no conceivable circumstance where any human being feels shame for something which is totally alone, none at all?

What I'm suggesting is that shame inherently involves at least a tacit social component - some imagined perspective by which we are condemned. This is consistent with Smith's and Hume's moral sentiments theory, where moral judgments fundamentally involve taking up imagined perspectives of others. This doesn't mean the shame isn't genuinely felt or that any specific others would actually condemn us. But in my experience people can frequently unravel particular cases of such shame by honestly examining what specific others would actually think if they knew, which is some experimental validation for this view.

Is anyone else on this website making YouTube videos? Less Wrong is great, but if you want to broadcast to a larger audience, video seems like the place to be. I know Rational Animations [LW · GW] makes videos. Is there anyone else? Are you, personally, making any?

Stargate was reported in 2024, and that reporting specified that the Stargate $100b phase hadn't started yet because MS was still building the previous phase, with "in excess of $115b" for all the phases, implying a large ramp up. And since Stargate was intended for OA, while MS of course has its own knitting to tend to, that implies much larger datacenter capex total. Given how vague the reporting is and how large the numbers are, but that the sooner the better, $80b in FY 2025 doesn't clearly tell me that there must be some mystery $25-40bn training system which is a big surprise. You don't build a Stargate overnight, and if it is to be finished and fully operational "as soon as 2028", you're going to need to be spending a lot of money 3 years beforehand.

I've answered a question on this discussion, and my short answer is that I basically agree with the post, mostly because I think computationalism is closest to accurate as a model of identity in the general case, with physicalism being a special case of the general case (with caveats) but I definitely think you were pretty epistemically terrible during your interactions, and I don't blame @andeslodes [LW · GW] and @sunwillrise [LW · GW] for disagreeing with the post, and the way you handled disagreements here does not make me confident that LW thought leaders will reliably go in truth-tracking directions.

Answer is below:

https://www.lesswrong.com/posts/yoAhc7ZhQZfGqrzif/what-are-the-actual-arguments-in-favor-of-computationalism#KTWgPbomupmwE2TFb [LW(p) · GW(p)]

General comments on consciousness:

https://www.lesswrong.com/posts/TkahaFu3kb6NhZRue/quick-general-thoughts-on-suffering-and-consciousness#FaMEMcpa6mXTybarG [LW(p) · GW(p)]

https://www.lesswrong.com/posts/TkahaFu3kb6NhZRue/quick-general-thoughts-on-suffering-and-consciousness#WEmbycP2ppDjuHAH2 [LW(p) · GW(p)]

Perhaps, depends how it is. I think we could do worse than just have Anthropic have a 2 year lead etc. I don't think they would need to prioritize profit as they would be so powerful anyway - the staff would be more interested in getting it right and wouldn't have financial pressure. WBE is a bit difficult, there needs to be clear expectations, i.e. leave weaker people alone and make your own world
https://www.lesswrong.com/posts/o8QDYuNNGwmg29h2e/vision-of-a-positive-singularity [LW · GW]
There is no reason why super AI would need to exploit normies. Whatever we decide, we need some kind of clear expectations and values regarding what WBE are before they become common, Are they benevolent super-elders, AI gods banished to "just" the rest of the galaxy, the natural life progression of first world humans now?

I agree that it's plausible just from priors that ASI could find a way to eat the sun.  The matter is there, and while it's strongly gravitationally bound in a way that's inconvenient, there's nothing physically impossible about getting it out of that arrangement into one that's more convenient to using fusion reactors or something.

But an analysis of how plausible the scenario is would certainly have made the post more valuable.  There are plausible proposals for how to get the fuel present in the sun out such that it could be used more efficiently, and while it may be possible that an ASI might come up with a more elegant or efficient plan, there are some fundamental physical limits on exactly how efficient the process could be made.

Wikipedia has some discussion of possible methods: https://en.m.wikipedia.org/wiki/Star_lifting

That article says: "This energy could be collected by a Dyson sphere; using 10% of the Sun's total power output would allow 5.9 x10^21 kilograms of matter to be lifted per year (0.0000003% of the Sun's total mass)", but this doesn't take account of the possibility of using the collected mass to fuel fusion reactions that are then used to power the mass collection.  What are the constraints on that process (my first thought is you have to worry about heat if you try to get the total power too high).

10,000 years sounds like enough time if you can get an exponential process going that uses the fuel harvested from the sun to collect more fuel.  But any process will have some constraints, such as max temperature at which the various parts of your system can function, or the specific materials which your system is made of (do you have to build your fusion reactors out of materials harvested from metal rich bodies? can you use carbon converted into diamondoid nanomachines? can you get enough of those materials out of the fusion of hydrogen to keep the process going once it's started?).  Even if your fuel harvesters and fusion reactors can stand up to the high temperatures necessary to eat the sun in that time frame, what about everything else in the solar system?  Does this process sterilize the earth of biological life?

Once I consider that there will be some sort of physical contraints on the process and also remember the fact that the sun is really big, it's not obvious that even an exponential process of fuel harvesting from the sun will be completed in a 10,000 year time frame.

Except frequently I think people who are ashamed don't expect this.

That’s why I distinguished explicitly between shame and depravity in the OP.