LessWrong 2.0 Reader

Stargate was reported in 2024, and that reporting specified that the Stargate $100b phase hadn't started yet because MS was still building the previous phase, with "in excess of $115b" for all the phases, implying a large ramp up. And since Stargate was intended for OA, while MS of course has its own knitting to tend to, that implies much larger datacenter capex total. Given how vague the reporting is and how large the numbers are, but that the sooner the better, $80b in FY 2025 doesn't clearly tell me that there must be some mystery $25-40bn training system which is a big surprise. You don't build a Stargate overnight, and if it is to be finished and fully operational "as soon as 2028", you're going to need to be spending a lot of money 3 years beforehand.

I've answered a question on this discussion, and my short answer is that I basically agree with the post, mostly because I think computationalism is closest to accurate as a model of identity in the general case, with physicalism being a special case of the general case (with caveats) but I definitely think you were pretty epistemically terrible during your interactions, and I don't blame @andeslodes [LW · GW] and @sunwillrise [LW · GW] for disagreeing with the post, and the way you handled disagreements here does not make me confident that LW thought leaders will reliably go in truth-tracking directions.

Answer is below:

https://www.lesswrong.com/posts/yoAhc7ZhQZfGqrzif/what-are-the-actual-arguments-in-favor-of-computationalism#KTWgPbomupmwE2TFb [LW(p) · GW(p)]

General comments on consciousness:

https://www.lesswrong.com/posts/TkahaFu3kb6NhZRue/quick-general-thoughts-on-suffering-and-consciousness#FaMEMcpa6mXTybarG [LW(p) · GW(p)]

https://www.lesswrong.com/posts/TkahaFu3kb6NhZRue/quick-general-thoughts-on-suffering-and-consciousness#WEmbycP2ppDjuHAH2 [LW(p) · GW(p)]

Different example - I said "instead" - so if the musician openly admits and apologize for only being average they are ashamed because they are afraid of the reaction of the fan who clearly loved their performance (not their failure to abstain from what they believe is the cause of their average performance?), but if they don't mention it to anyone (therefore are committing neither a dominance nor submission gesture) they are also ashamed? Or are they not ashamed in both circumstances? I'm just saying I'm really confused.

Are you telling me there is no conceivable circumstance where any human being feels shame for something which is totally alone, none at all? Because at the risk of assuming I have privileged knowledge of myself - I assure you I've felt shame for things which no one would care about.

Perhaps, depends how it is. I think we could do worse than just have Anthropic have a 2 year lead etc. I don't think they would need to prioritize profit as they would be so powerful anyway - the staff would be more interested in getting it right and wouldn't have financial pressure. WBE is a bit difficult, there needs to be clear expectations, i.e. leave weaker people alone and make your own world
https://www.lesswrong.com/posts/o8QDYuNNGwmg29h2e/vision-of-a-positive-singularity [LW · GW]
There is no reason why super AI would need to exploit normies. Whatever we decide, we need some kind of clear expectations and values regarding what WBE are before they become common, Are they benevolent super-elders, AI gods banished to "just" the rest of the galaxy, the natural life progression of first world humans now?

I agree that it's plausible just from priors that ASI could find a way to eat the sun.  The matter is there, and while it's strongly gravitationally bound in a way that's inconvenient, there's nothing physically impossible about getting it out of that arrangement into one that's more convenient to using fusion reactors or something.

But an analysis of how plausible the scenario is would certainly have made the post more valuable.  There are plausible proposals for how to get the fuel present in the sun out such that it could be used more efficiently, and while it may be possible that an ASI might come up with a more elegant or efficient plan, there are some fundamental physical limits on exactly how efficient the process could be made.

Wikipedia has some discussion of possible methods: https://en.m.wikipedia.org/wiki/Star_lifting

That article says: "This energy could be collected by a Dyson sphere; using 10% of the Sun's total power output would allow 5.9 x10^21 kilograms of matter to be lifted per year (0.0000003% of the Sun's total mass)", but this doesn't take account of the possibility of using the collected mass to fuel fusion reactions that are then used to power the mass collection.  What are the constraints on that process (my first thought is you have to worry about heat if you try to get the total power too high).

10,000 years sounds like enough time if you can get an exponential process going that uses the fuel harvested from the sun to collect more fuel.  But any process will have some constraints, such as max temperature at which the various parts of your system can function, or the specific materials which your system is made of (do you have to build your fusion reactors out of materials harvested from metal rich bodies? can you use carbon converted into diamondoid nanomachines? can you get enough of those materials out of the fusion of hydrogen to keep the process going once it's started?).  Even if your fuel harvesters and fusion reactors can stand up to the high temperatures necessary to eat the sun in that time frame, what about everything else in the solar system?  Does this process sterilize the earth of biological life?

Once I consider that there will be some sort of physical contraints on the process and also remember the fact that the sun is really big, it's not obvious that even an exponential process of fuel harvesting from the sun will be completed in a 10,000 year time frame.

Except frequently I think people who are ashamed don't expect this.

That’s why I distinguished explicitly between shame and depravity in the OP.

In this example?

Except frequently I think people who are ashamed don't expect this. Imagine that instead of concealing they openly admit and apologize for being only average: then what? Aren't they still ashamed?

Um, most of those don't sound very "aligned". I think perhaps a very idiosyncratic definition of that word is in play here...

Yeah, any small group of humans seizing unprecedented control over the entire world seems like a bad gamble to take, even if they start off seeming like decent people.

I'm currently hoping we can figure some kind of new governance solution for managing decentralized power while achieving adequate safety inspections.

https://www.lesswrong.com/posts/FEcw6JQ8surwxvRfr/human-takeover-might-be-worse-than-ai-takeover?commentId=uSPR9svtuBaSCoJ5P [LW(p) · GW(p)]

This is also my view of the situation, as well, and is a big portion of the reason why solving AI alignment, which reduces existential risk a lot, is non-trivially likely without further political reforms I don't expect to lead to dystopian worlds (from my values).