LessWrong 2.0 Reader

I was scared that by clicking on the red button I was going to launch nukes... so I did not hit it. Now I know that next year I will click.

Well, maybe.

Assuming this is the important distinction, I like something like “isolated”/“integrated” better than either of those.

I think a problem with all the proposed terms is that they are all binaries, and one bit of information is far too little to characterize takeoff: 

• One person's "slow" is >10 years, another's is >6 months. 
• The beginning and end points are super unclear; some people might want to put the end point near the limits of intelligence, some people might want to put the beginning points at >2x AI R&D speed, some at 10, etc. 
• In general, a good description of takeoff should characterize capabilities at each point on the curve.  

So I don't really think that any of the binaries are all that useful for thinking or communicating about takeoff. I don't have a great ontology for thinking about takeoff myself to suggest instead, but I generally try to in communication just define a start and end point and then say quantitatively how long this might take. One of the central ones I really care about is the time between wakeup and takeover capable AIs. 

wakeup = "the first period in time when AIs are sufficiently capable that senior government people wake up to incoming AGI and ASI" 

takeover capable AIs = "the first time there is a set of AI systems that are coordinating together and could take over the world if they wanted to" 

The reason to think about this period is that (kind of by construction) it's the time where unprecedented government actions that matter could happen. And so when planning for that sort of thing this length of time really matters. 

Of course, the start and end times I think about are both fairly vague. They also aren't purely a function of AI capabilities, and they care about stuff like "who is in government" and "how capable our institutions are at fighting a rogue AGI".  Also, many people believe that we never will get takeover capable AIs even at superintelligence.

None of it would look like it was precipitated by an AI taking over.

But, to be clear, in this scenario it would in fact be precipitated by an AI taking over? Because otherwise it's an answer to "humans went extinct, and also AI took over, how did it happen?" or "AI failed to prevent human extinction, how did it happen?"

I was finding it a bit challenging to unpack what you're saying here. I think, after a reread, that you're using ‘slow’ and ‘fast’ in the way I would use ‘soon’ and ‘far away’ (aka. referring to the time it will occur from the present). Is this read about correct?

Not an American but support Trump from afar. Genuine curiosity here - if you were to steelman the rational Trump supporter, what would you say? (happy for pushback in the ensuing discussion).

As a Petrov, it was quite engaging and at times, very stressful. I feel very lucky and grateful that I could take part. I was also located in a different timezone and operating on only a few hours sleep which added a lot to the experience!

"I later found out that, during this window, one of the Petrovs messaged one of the mods saying to report nukes if the number reported was over a certain threshold. From looking through the array of numbers that the code would randomly select from, this policy had a ~40% chance of causing a "Nukes Incoming" report (!). Unaware of this, Ray and I made the decision not to count that period."

I don't mind outing myself and saying that I was the Petrov who made the conditional report while the site was down. This occurred during the opening hours of the game and it was unclear to me if generals could unilaterally launch nukes without their team being aware. I'm happy to take a weighted karma penalty for it, particularly as the other Petrov did not take a similar action when faced with (presumably) the same information I had.^[1] 

Once it was established that unilateral first strike by a general still informed their teammates of their action and people staked their reputation on honest reporting, the game was essentially over. From that point, my decisions to report "All Clear" were independent of the number of detected missiles. 

I recorded my timestamped thoughts and decision making process throughout the day, particularly in the hour before making the conditional report. I intend on posting a summary^[2] of it, but have other commitments in the next few days:

How much would people value seeing a summary of my hour by hour decisions in the next few days over seeing a more digestible summary posted later?

1. ^{^}

   Prior to the game I outlined what I thought my hypothetical decision making process was going to be, and this decision was also in conflict with that.

2. ^{^}

   Missile counts, and a few other details, would of course be hidden to preserve the experience for future Petrovs. Please feel free to specify other things you believe should be hidden. 

That much I knew already at the time I wrote the comment. I know that e.g. human brains are basically just scaled up chimp brains etc. I definitely am closer to the blank slate end of the spectrum than the 'evolution gave us tons of instincts' end of the spectrum. But it's not a total blank slate. What do you think would happen if we did a bunch of mad science to make super-dogs that had much bigger brains? And then took one of those dogs and tried to raise it like a human child? My guess is that it would end up moving some significant fraction of the distance from dog to human, but not crossing the gap entirely, and if it did end up fitting in to human society, getting a job, etc. it would have very unusual fetishes at the very least and probably all sorts of other weird desires and traits.

Fast/Slow takeoff

Yeah I think the similarity of takeoff and timelines is maybe the real problem.

Like if Takeoff wasn’t two syllables that starts with T I might be happy with ‘short/long’ being the prefix for both.