LessWrong 2.0 Reader

Sounds like a lot of political alliances! (And "these two political actors are aligned" is arguably an even weaker condition than "these two political actors are allies".)

At the end of the day, of course, all of these analogies are going to be flawed. AI is genuinely a different beast.

Be faithless to your cause and betray it to a stronger enemy.

This is such an incredible quote. I feel like I could write a whole essay on this.

Google fires 28 employees working on cloud and AI services for doing a ten hour sit in where they occupied their boss’s office until the police were eventually involved. And yes, if what you do at work is spend your time blockading your boss’s office until your policy demands are met, it seems like you are going to get fired?

In a company other than Google, I would say: yes, obviously.

But remember, when James Damore wrote his document, and as a reaction other people stopped doing their work in protest, it was he who was fired, not them. How were they supposed to know that this time it will be different?

you should substitute more specific language in place of “rational”: “The self-benefiting thing to do is to run away, but I hope I would at least try to drag the child off the railroad tracks,”

Wouldn't it be correct to say that it would be 'instrumentally rational' to run away in this case? It sounds rational to me, as far as you 'winning' means you 'surviving'.

Yeah, I was thinking it's hard to beat dried salted meat, hard cheese, and oil or butter. 

You also don't have to assume that all the food travels the whole way. If (hypothetically) you want to send 1 soldier's worth of food and water 7 days away, and each person can only carry 3 days worth at a time, then you can try to have 3 days worth deposited 6 days out, and then have a porter make a 2 day round trip carrying 1 day's worth to leave for that soldier to pickup up on day 7. Then someone needs to have carried that 3 days worth to 6 days out, which you can do by having more porters make 1 day round trips from 5 days out, etc. Basically it you need exponentially more people and supplies the farther out your supply chains stretch. I think I first read about this in the context of the Incas, because potatoes are less calorie dense per pound than dried grains so it's an even bigger problem? Being able to get water along the way, and ideally to pillage the enemy's supplies, are also a very big deal.

Aren't there a lot of clearer words for this? "Well-intentioned", "nice", "benevolent", etc.

Fair enough. I guess it just seems somewhat incongruous to say. "Oh yes, the AI is aligned. Of course it might desperately crave murdering all of us in its heart (we certainly haven't ruled this out with our current approach), but it is aligned because we've made it so that it wouldn't get away with it if it tried [LW · GW]."

I think at that point the limiting factors become the logistics of food, waste, water, and waste heat. In Age of Em Robin Hanson spends time talking about fractal plumbing systems and the like, for this kind of reason.

All good points, many I agree with. If nothing else, I think that humanity should pre-commit to following this strategy whenever we find ourselves in the strong position. It's the right choice ethically, and may also be protective against some potentially hostile outside forces.

However, I don't think the acausal trade case is strong enough that I would expect all sufficiently powerful civilizations to have adopted it. If I imagine two powerful civilizations with roughly identical starting points, one of which expanded while being willing to pay costs to accommodate weaker allies while the other did not and instead seized whatever they could, then it is not clear to me who wins when they meet. If I imagine a process by which a civilization becomes strong enough to travel the stars and destroy humanity, it's not clear to me that this requires it to have the kinds of minds that will deeply accept this reasoning. 

It might even be that the Fermi paradox makes the case stronger - if sapient life is rare, then the costs paid by the strong to cooperate are low, and it's easier to hold to such a strategy/ideal.

I struggle a bit to remember what ASI is but I'm gonna assume it's Artificial Super Intelligence. 

Let's say that that's markedly cleverer than 1 person. So it's capable of running very successful trading strategies or programming extremely well. It's not clear to me that such a being:

• Has been driven towards being agentic, when its creators will prefer something more docile
• Can cooperate well enough with itself to manage some massive secret takeover
• Is competent enough to recursively self improve (and solve the alignment problems that creates)
• Can beat everyone else combined

Feels like what such a being/system might do is just run some terrifically successful trading strategies and gather a lot of resources while frantically avoiding notice/trying to claim it won't take over anything else. Huge public outcry, continuing regulation but maybe after a year it settles to some kind of equilibrium. 

Chance of increasing capabilities and then some later jump, but seems plausible to me that that wouldn't happen in one go. 

Hi everyone! Skylar (ACX meetup czar) has asked me to circulate the following short survey about the meetup: https://docs.google.com/forms/d/e/1FAIpQLSeTUw3pHQK0oGEKZwHjPFar3PyWkEry-X5M6hD6cGrJpgplBQ/viewform 

If you have a moment to fill it out, that would be awesome. Thanks!