LessWrong 2.0 Reader

Button is very intimidating, indeed.

Yeah I'm on board with deontological-injunction shaped constraints. See here [LW · GW] for example.

Perhaps instead of "attempting to achieve the goal at any cost" it would be better to say "being willing to disregard conventions and costs imposed on uninvolved parties, if considering those things would get in the way of the pursuit of the goal".

Nah I still disagree. I think part of why I'm interpreting the words differently is because I've seen them used in a bunch of places e.g. the lightcone handbook to describe the lightcone team. And to describe the culture of some startups (in a positively valenced way).

Being willing to be creative and unconventional -- sure, but this is just part of being capable and solving previously unsolved problems. But disregarding conventions that are important for cooperation that you need to achieve your goals? That's ridiculous. 

Being willing to impose costs on uninvolved parties can't be what is implied by 'going hard' because that depends on the goals. An agent that cares a lot about uninvolved parties can still go hard at achieving its goals.

I suspect we may be talking past each other here.

Unfortunately we are not. I appreciate the effort you put into writing that out, but that is the pattern that I understood you were talking about, I just didn't have time to write out why I disagreed.

I expect this to continue to be true in the future

This is the main point where I disagree. The reason I don't buy the extrapolation is that there are some (imo fairly obvious) differences between current tech and human-level researcher intelligence, and those differences appear like they should strongly interfere with naive extrapolation from current tech. Tbh I thought things like o1 or alphaproof might cause the people who naively extrapolate from LLMs to notice some of these, because I thought they were simply overanchoring on current SoTA, and since the SoTA has changed I thought they would update fast. But it doesn't seem to have happened much yet. I am a little confused by this.

What observations lead you to suspect that this is a likely failure mode?

I didn't say likely, it's more an example of an issue that comes up so far when I try to design ways to solve other problems. Maybe see here [LW · GW] for instabilities in trained systems, or here for more about that particular problem.

I'm going to drop out of this conversation now, but it's been good, thanks! I think there are answers to a bunch of your claims in my misalignment and catastrophe [LW · GW] post.

I'm thinking that the most ethical (morally least risky) way to "insure" against a scenario in which AI takes off and property/wealth still matters is to buy long-dated far out of the money S&P 500 calls. (The longest dated and farthest out of the money seems to be Dec 2029 10000-strike SPX calls. Spending $78 today on one of these gives a return of $10000 if SPX goes to 20000 by Dec 2029, for example.)

My reasoning here is that I don't want to provide capital to AI industries or suppliers because that seems wrong given what I judge to be high x-risk their activities are causing (otherwise I'd directly invest in them), but I also want to have resources in a post-AGI future in case that turns out to be important for realizing my/moral values. Suggestions welcome for better/alternative ways to do this.

I contend that P(H2) is very close to P(H1), and certainly in the same order of magnitude, since (conditional on H1) a simulation that does not test for H2 is basically useless.

As for priors I’d refuse to update down – well, the ASI is smarter than either of us!

For the principle of indifference to apply, you’d need infinitely many purposes as plausible as this one, or at least similarly plausible. I can’t imagine how this could hold. Can you think of three?

But this assumes a model should aim to fit all data, which is a waste of effort.

Being pets also means human agency would no longer be a relevant input to the trajectory of human lives.

You aren't supposed to use metaethics to settle ethical arguments, the point of metaethics is to get people to stop discussing metaethics.

I greatly enjoyed this book back in the day, but the whole scenario was wild enough to summon the moral immune system. Past a certain point, for me it’s a safe default to put up mental barriers and actively try not to learn moral lessons from horror fiction. Worm, Gideon the 9th, anything by Stephen King- great, but I don’t quite expect to learn great lessons.

While rejecting them as sources of wisdom now, I can remember these books and return to them if I suddenly need to make moral choices in a world where people can grow wiser by being tortured for months, or stronger by

So the responsible thing to do is to refrain, right? Cause if everybody did nobody would have the ability to destroy the site, and what would you do with such an ability anyway except not use it?

The problem is it feels like a game/exercise, and a game/exercise is something I want to opt in to, even if just to challenge myself.

Other than that, I think twice before pressing any big red button, and as of yet I haven't pressed this button.