LessWrong 2.0 Reader

Twenty years seems indeed probably too short, though it’s hard to say how post-singularity technology will affect things like public deliberation timelines. 

My best guess is 200 years will very likely be enough.

I agree with you that there exist some very small minority of people who will have a specific attachment to the sun, but most people just want to live good and fulfilling lives, and don’t have strong preferences about whether the sun in the sky is exactly 1 AU away and feels exactly like the sun of 3 generations past. 

To be clear, for me it’s a crux whether not dismantling the sun is basically committing everyone who doesn’t want to be uploaded to relative cosmic poverty. It would really suck if all remaining biological humans would be unable to take advantage of the vast majority of the energy in the solar system. 

I am not at present compelled that the marginal galaxies are worth destroying the sun and earth for (though I am also not confident it isn’t, I feel confused about it, and also don’t know where most people would end up after having been made available post-singularity intelligence enhancing drugs and deliberation technologies, which to be clear not everyone would use, but most people probably would). 

The 2019 LW post [LW · GW] discusses a podcast which talks a lot about gears-y models and proposed mechanisms; as I understand it, the high level "zinc ions inhibit viral replication" model is fairly well accepted, but some of the details around which brands are best aren't as well-attested elsewhere in the literature. For instance, many of these studies don't use zinc acetate, which this podcast would suggest is best. (To its credit, the 2013 meta-analysis does find that acetate is (nonsignificantly) better than gluconate, though not radically so.)

I thought it'd be useful for others to link to your longer writings on this: 

• Consider granting AIs freedom [EA · GW]
• The moral argument for giving AIs autonomy [EA · GW] 

I expect non-positional material goods to be basically saturated for Earth people in a good post-Singularity world, so I don't think you can promise them to become twice as rich. And also, people dislike drastic change and new things they don't understand. 20% of the US population refused the potentially life-saving covid vaccine out of distrust of new things they don't understand. Do you think they would happily move to a new planet with artificial sky maintained by supposedly benevolent robots? Maybe you could buy off some percentage of the population if material goods weren't saturated, but surely not more than you could convince to get the vaccine? Also, don't some religions (Islam?) have specific laws about what to do at sunrise and sunset and so on? Do you think all the imams would go along with moving to the new artificial Earth? I really think you are out of touch with the average person on this one, but we can go out to the streets and interview some people on the matter, though Berkeley is maybe not the most representative place for this. 

(Again, if you are talking about cultural drift over millennia, that's more plausible, though I'm below 50% they would dismantle the Sun. But I'm primarily arguing against dismantling the Sun within twenty years of the Singularity.)

Seems right, I used the language of the thread, but edited it since I agree. 

(It is not implied in the song, to be clear, that part seems made up as far as I can tell, or you have a reading of the lyrics I do not understand. 

The song talks about there being a singularity in ~20 years, and separately that the sun is wasteful, but I don’t see any reference to the sun being dismantled in 20 years. For reference, lyrics are here: https://luminousalicorn.tumblr.com/post/175855775830/a-filk-of-big-rock-candy-mountain-one-evening-as) 

Propaganda-masquerading-as-paper: the paper is mostly valuable as propaganda for the political agenda of AI safety. Scary demos are a central example. There can legitimately be valuable here.

Maybe you disagree, but it's worth noting that imo the main reason to do model organisms work like Sleeper Agents and Alignment Faking is not for the demo value but for the value of having concrete examples of the important failure modes for us to then study scientifically, e.g. understanding why and how they occur, what changes might mitigate them, what they look like mechanistically, etc. We call this the "Scientific Case" in our Model Organisms of Misalignment [LW · GW] post. There is also the "Global Coordination Case" in that post, which I think is definitely some of the value, but I would say it's something like 2/3 science and 1/3 coordination.

Side-note: Just registering that I personally aspire to always taboo 'normal people' and refer to specific populations. I think it tends to sneak in a lot of assumptions to call people 'normal' – I've seen it used to mean "most people on Twitter" or "most people in developed countries" or "most working class people" or "most people alive today" – the latter of which is not at all normal by historical standards!

Someone will live on old earth in your scenario. Unless those people are selected for extreme levels of attachment to specific celestial bodies, as opposed to the function and benefit of those celestial bodies, I don’t see why those people would decide to not replace the sun with a better sun, and also get orders of magnitude richer by doing so.

It seems to me that the majority of those inhabitants of old earth would simply be people who don’t want to be uploaded (which is a much more common preference I expect than maintaining the literal sun in the sky) and so have much more limited ability to travel to other solar systems. I don’t see why I would want to condemn most people who don’t want be uploaded to relative cosmic poverty just because a very small minority of people want to keep burning away most of the usable energy in the solar system for historical reasons. 

Are you arguing that if technologically possible, the Sun should be dismantled in the first few decades after the Singularity, as it is implied in the Great Transhumanist Future song, the main thing I'm complaining about here? In that case, I don't know of any remotely just and reasonable (democratic, market-based or other) governance structure that would allow that to happen given how the majority of people feel.

If you are talking about population dynamics, ownership and voting shifting over millennia to the point that they decide to dismantle the Sun, then sure, that's possible, though that's not what I expect to happen, see my other comment on market trades and my reply to Habryka on population dynamics.