LessWrong 2.0 Reader

The load bearing assumption here seems to be that we won't make unaligned superintelligent systems given current methods soon enough to think it matters.

This seems false, and at the very least should be argued explicitly.

asking it to think long term instead could change the fate of our species

If it's superintelligent, it has already thought more deeply about the long term than any human ever has. 

I think society is very inconsistent about AI risk because the "Schelling point" is that people feel free to believe in a sizable probability of extinction from AI without looking crazy, but nobody dares argue for the massive sacrifices (spending or regulation or diplomacy) which actually fit those probabilities.

The best guess by basically every group of people, is that with 2%-12%, AI will cause catastrophe [LW · GW] (kill 10% of people). At these probabilities, AI safety should be an equal priority to the military!

Yet at the same time, nobody is doing anything about it. Because they all observe everyone else doing nothing about it. Each person thinks the reason that "everyone else" is doing nothing, is that they figured out good reasons to ignore AI risk. But the truth is that "everyone else" is doing nothing for the same reason that they are doing nothing. Everyone else is just following everyone else.

This "everyone following everyone else" inertia is very slowly changing, as governments start giving a bit of lip-service and small amounts of funding to organizations which are half working on AI Notkilleveryoneism. But this kind of change is slow and tends to take decades. Meanwhile many AGI timelines are less than one decade.

It’s the backfire supposition that is baseless

There is a potential instinctive association of protests with violence, riots, vandalism, obstruction of public property, crackpots/conspiracy theorists, et cetera. I don't think it's baseless to worry whether this association is strong enough, in a median person's mind, for any protest towards an unknown cause to be instinctively associated with said negative things, with this first impression then lingering.

Anti-technology protests, in particular, might have an association with Unabomber-style terrorism, and certainly the AGI labs will be eager to reinforce this association. Protests therefore make your cause/movement uniquely vulnerable to this type of attack (via corresponding biased newspaper coverage). The marginal increase in visibility does not necessarily offset it.

It doesn't seem obvious to me whether the net effects are positive or negative. Do you have theoretical or empirical support for the effects being positive?

Small protests are the only way to get to big protests

I don't think so, and I'm not even saying small protests bad. I'm saying small protests might be bad without the appropriate groundwork.

Thanks for catching that! I hadn't heard. I will probably have to rewrite that section of the post.

What's your impression about the general finding about many autoimmune variants increasing protection against ancient plauges?

It's totally possible that I'm seeing faces in the clouds but there seems to be a non-trivial relationship between these two glitch token and what they make the model say.

• Хронологија -> chronologija -> chronology, i.e. time-related, like February
• "kwiet" is similar to "kwiecień" which means "April" in Polish (also "kviten'" in Ukrainian)

No, the problem really is technical right now.

There may be additional societal and political problems afterwards. But none of those problems actually matter unless the technology works.

Obviously we are going to do it in animals first. We have in fact DONE gene editing in animals many times (especially mice, but also some minor stuff in cows and other livestock). But you're correct that we need to test massive multiplex editing. My hope is we can have good data on this in cows in the next 1-3 years.

I don't understand your question

An other problem with authors calculs of potential to improve intelligence: let's suppose, there is a problem in the human brain that reduces IQ by 10 points, and it can be solved by Gene1 or Gene2. Let's suppose that 99% of humans do not have either Gene1 or Gene2. In this case, the author's method would show that if we added both Gene1 and Gene2 to the same person, their IQ would increase by 20 points.

Curious about the claim regarding bistable perception as the brain "settling" differently on two distinct but roughly equally plausible generative model parameters behind an observation. In standard statistical terms, should I think of it as: two parameters having similarly high Bayesian posterior probability, but the brain not explicitly representing this posterior, instead using something like local hill climbing to find a local MAP solution—bistable perception corresponding to the two different solutions this process converges to?

If correct, to what extent should I interpret the brain as finding a single solution (MLE/MAP) versus representing a superposition or distribution over multiple solutions (fully Bayesian)? Specifically, in which context should I interpret the phrase "the brain settling on two different generative models"?