LessWrong 2.0 Reader

I woke up this morning thinking 'would be nice to have a concise source for the whole zinc/colds thing'. This is amazing.

I help run an EA coliving space, so I started doing some napkin math on how many sick days you'll be saving our community over the next year. Then vaguely extrapolated to the broader lesswrong audience who'll read your post and be convinced/reminded to take zinc (and given decent guidance for how to use it effectively).

I'd guess at minimum you've saved dozens of days over the next year by writing this post. That's pretty cool. Thankyou <3

Or, worse, if most directions are net negative and you have to try quite hard to find one which is positive, almost everyone optimizing for magnitude will end up doing harm proportional to how much they optimize magnitude.

This is an extremely important unsolved question IMO, because a multipolar scenario appears to be where were heading if we can adequately solve alignment in time.

See if we solve alignment do we die anyway [LW · GW] and the discussion and edited conclusion. Even after all of that, I notice I'm still confused.

The best I've come up with is: Don't be in a multipolar scenario any more than you absolutely have to. Nonproliferation, lime with nukes seems like the only answer. The best solution to a multipolar scenario is to not let it become any more multipolar than it is, and ultimately make it less multipolar.

The problems you mention seem very bad and it gets worse when you consider that very advanced technology is probably able to save a few of the genocidal AI controllers favorite people, or maybe the mind states of a lot of people, even while wiping out humanity and rival AGIs to provide some control of the future for whatever ideology.

Another possibility I should add is that rival AGIs may resort to mutually assured destruction. Having a dead man switch to crack the earth crust or send the sun nova if you're not around to stop it would be an extreme measure that could be applied. Sending a copy of yourself off to a nearby star with a stealthy departure would seem like good insurance against a genocidal takeover.

Universql surveillance of earth and the solar system might suffice to prevent hostile exponential military improvements. That might even be done by a neutral AGI that keeps everyone's secrets as long as they're not violating a treaty about developing the capacity to kill everyone else.

Makes sense and totally agree, thanks!

The vNM axioms constrain the shape of an agent's preferences, they say nothing about how to make decisions

Suppose your decision in a particular situation comes down to choosing between some number of lotteries (with specific estimated probabilities over their outcomes) and there's no complexity/nuance/tricks on top of that. In that case, vNM says that you should choose the one with the highest expected utility as this is the one you prefer the most.

At least assuming that choice is the right operationalization of preferences but if it isn't, then the Dutch book / money-pump arguments don't follow.

ETA: I guess I could just say:

What are your preferences if not your idealized evaluations of decision-worthiness of options (modulo "being a corrupted piece of software running on corrupted hardware")?

while, at the same time, the reach of his actions and opinions have also maximized

I disagree. Companies will become complacent and stagnant very fast if leadership is not going hard against it. I'm not sure I'd like to work for him but my impression is that he really pushes his teams to go for more faster and results have shown up. I'm not sure "dumb"(at first look) ideas like Mechazilla would exist without him being willing to try cool things.

I'm on the fence for X, there have been good changes and the site feels a bit less like an echo chamber but there have also been negatives and I'm concerned by his recent implications in politics because I don't think he can bring much value, especially compared to his companies. 

Neuralink could also be very important. 

Overall, he would benefit from talking less but you can't have everything. I still think very much net positive.

The answer surely depends mostly on what his impact will be on AI developments, both through his influence on the policy of the new administration and what he does with xAI. While I understand that his political actions might be mind-killing (to say the least) to many of his former fans, I would much prefer a scenario where Elon has infuriating politics but a positive impact on solving alignment over one with the opposite outcome.

Oops, you're of course right. I'll change my comment.

P(doom) can be approximately measured. 
If reality fluid describes the territory well, we should be able to see close worlds that already died off.

For nuclear war we have some examples.
We can estimate the odds that the Cuban missile crisis and Petrov's decision went badly. If we accept that luck was a huge factor in us surviving those events (or not encountering events like it), we can see how unlikely our current world is to still live.

A high P(doom) implies that we are about to (or already did) encounter some very unlikely events that worked out suspiciously well for our survival. I don't know how public a registry of events like this should be, but it should exist.

Our self-reporting murderers or murder-witnesses should be extraordinarily well protected from leaks however, which in part seems like a software question.

Yes, this seems unlikely to happen, but again if your P(doom) is high, then we are only to survive in unlikely worlds. Working on this, to me, seems dignified: a way to make those unlikely worlds a bit less unlikely.

It's impossible to know until he is done/defeated, because the things he experiences due to his actions now could cause huge swings in his impacts on the future.