LessWrong 2.0 Reader

I think it is an argument that they should not happen, but a very weak one, especially for things like this where the current public really hasn’t thought much about. 

Also, I think David is just wildly wrong here about what realistically would happen in 10,000 years for a society that could actually start using all of the suns energy. This would involve hundreds of generations of people each deciding to not grow, to not expand into the rest of the solar system, to pass up on enormous opportunities for joy and greatness and creation, out of sentimentality for a specific kind of attachment for the specific arrangement of our planet and solar system at this moment in time. This attachment is very much real, and worth something, but IMO obviously will not remotely outweigh the preferences and actions of the trillions of people who will all want to do more things and have more things (and since we are talking about gradual expansion, very much present in the conversation). 

it shows up in Solstice songs as a thing we want to do in the Great Transhumanist Future twenty years from now

Is this true?! (Do you have a link or something?)

Most decisions are not made democratically, and pointing out that a majoritarian vote is against a decision is no argument that they will not happen nor should not happen. This is true of the vast majority of resource allocation decisions such as how to divvy up physical materials.

You are putting words in people's mouths to accuse lots of people of wanting to round up the Amish and hauling them to extermination camps, and I am disappointed that you would resort to such accusations.

I agree that I don't viscerally feel the loss of the 200 galaxies, and maybe that's a deficiency. But I still find this position crazy. I feel this is a decent parallel dialogue:
Other person: "Here is a something I thought of that would increase health outcomes in the world by 0.00000004%."
Me: "But surely you realize that this measure is horrendously unpopular, and the only way to implement it is through a dictatorial world government."
Other person: "Well yes, I agree it's a hard dilemma, but on absolute terms, 0.00000004% of the world population is 3 people, so my intervention would save 3 lives. Think about how horrible tragedy every death is, I really feel there is a missing mood here when you don't even consider the upside of my proposal."

I do feel sad when the democratic governments of the world screw up policy in a big way according to my beliefs and values (as they often do). I still believe that democracy is the least bad form of government, but yes, I feel the temptation of how much better it would be if we were instead governed by perfect philosopher king who happens to be aligned with my values. But 0.00000004% is just a drop in the ocean.

Similarly, I believe that we should try to maintain a relatively democratic form of government at least in the early AI age (and then the people can slowly figure out if they find something better than democracy). And yes, I expect that democracies will do a lot of incredibly wasteful, stupid, and sometimes evil things by my lights, and I will sometimes wish if somehow I could have become the philosopher king. That's just how things always are. But leaving Earth alone really is chump change, and won't be among the top thousand things I disagree with democracies on. 

(Also, again, I think there will also be a lot of value in conservatism and caution, especially that we probably won't be able to fully trust our AIs in the most complicated issues. And I also think there is something intrinsically wrong about destroying Earth, I think that if you cut out the world's oldest tree to increase health outcomes by 0.00000004%, you are doing something wrong, and people have a good reason to distrust you.)

This point doesn't make sense to me. It sounds similar to saying "Most people don't like it when companies develop more dense housing in cities, therefore a good democracy will not have it" or "Most people don't like it when their horse-drawn carriages are replaced by cars, therefore a good democracy will not have it". 

The cost-benefit calculations on these things work out and it's good if most uninformed people who haven't spent much time on it are not able to get in the way of companies that are building goods and services in this regard. 

There are many many examples (e.g. GMOs, nuclear power, coal power, privatized toll roads, fracking, etc), and I expect if I researched for a few hours I would find even clearer examples for which it is currently consensus that it is a good idea, but at the time the majority disliked it.

More generally:

• People's mass preferences are sometimes dumb, and sometimes quite reasonable, and you should have a decision rule that distinguishes between the two, and when things are orders of magnitude more cost effective than other things, this is a good argument against arguments based on simple preference / aesthetics, and this comment does nothing to show that this isn't stupidity rather than wisdom.
• Just because a lot of people in a democracy disapproves of things does not mean that market forces shouldn't be able to disagree with them and be correct about that. Analogous to the Luddites who had little concept of how technological and economic progress lifts everyone out of poverty, most people today do not appreciate that future computational-life forms will be just as meaningful as the meat-based ones today, and should not sacrifice orders of magnitudes more life-years than will be lived on Earth^[1] for the difference between a big ball of plasma and something else that recreates the same quality of light.
• Majoritarian vote on everything is a terrible way to make decisions; most decisions should be given to as small a group as possible (ideally an individual) who is held accountable for the outcome being good, and is given the resources to make the decision well. We do it for political leaders due to the low levels trust and high levels of adversarial action, but this should not be extended to whether to take the sun apart for parts.

1. ^{^}

   I'd quickly guess that the energy difference supports ${10}^n$ life-years where $n$ is somewhere between $10$ and $30$.

Yes, I wanted to argue something like this. 

"Rogue celestial body" should also include more commonly encountered interstellar objects, such as Oumuamua. Attempting to detect these objects (and to deflect potential impactors) does not seem obviously futile.

(I think a lot of unlearning research is bullshit, but besides that, is anyone deploying large models doing unlearning?)

Why do you think this? Is there specific research you have in mind? Some kind of reference would be nice. In the general case, it seems to me that unlearning matters because knowing how to effectively remove something from a model is just the flip-side of understanding how to instill values. Although not the primary goal of unlearning, work into how to 'remove' should also equally benefit attempts to 'instill' robust values into the model. If fine-tuning for value alignment just patches over 'bad facts' with 'good facts' any 'aligned' model will be less robust than one with harmful knowledge properly removed. If the alignment faking paper and peripheral alignment research are important at a meta level, then perhaps unlearning will be important because it can tell us something about 'how deep' our value installation really is. Lack of current practical use isn't really important, we should be able to develop theory that will tell us something important about model internals. I think there is a lot of very interesting mech-interp of unlearning work waiting to be done that can help us here.

Thanks for the comment - the reason I focus on cosmic unfairness here is because I addressed local unfairness in a previous post [LW · GW] in the sequence - apologies this wasn't clear, I've now added a hyperlink to clarify. 

I don't agree that the challenges of Dostoevsky etc are only about local unfairness though: as I say I think it's typically a mixture of local and cosmic unfairness that are not clearly distinguished. 

The 'problem from the lack of intervention' that you mention is much discussed by people in this context, so presumably they think it is relevant to the challenges they are considering, even if there is no easy solution.