LessWrong 2.0 Reader

Well said. Though IMO this analogy breaks down since Bill Gates (or Bernard Arnault) giving a random person $77.18 of their wealth is a very different scenario from an ASI taking a solar system from its local inhabitants.

A more direct analogy would be the chances of Bernauld Arnault stealing $77.18 from a random person. That doesn't seem very likely!

Of course many of the other arguments ring true - I'm just dubious of this particular comparison

Meta: OP and some replies occasionally misspell the example billionaire’s surname as “Arnalt”; it’s actually “Arnault”, with a ‘u’.

Re different algorithms, I actually agree with both you and Daniel Murfet in that conditional on non-reversible computers, there is at most 1-3 algorithms to achieve intelligence that can scale arbitrarily large, and I'm closer to 1 than 3 here.

But once reversible computers/superconducting wires are allowed, all bets are off on how many algorithms are allowed, because you can have far, far more computation with far, far less waste heat leaving, and a lot of the design of computers is due to heat requirements.

I agree but I'm not very optimistic about anything changing. Eliezer is often this caustic when correcting what he perceives as basic errors, and criticism in LW comments is why he stopped writing Sequences posts.

Bernald Arnalt has given eight-figure amounts to charity. Someone who reasoned, "Arnalt is so rich, surely he'll spare a little for the less fortunate" would in fact end up making a correct prediction about Bernald Arnalt's behavior!

Just for the sake of concreteness, since having numbers here seems useful, it seems like Bernald Anault has given around ~$100M to charity, which is around 0.1% of his net worth (spreading this contribution equally to everyone on earth would be around one cent per person, which I am just leaving it here for illustrative purposes, it's not like he could give any actually substantial amount to everyone if he really wanted). 

I wish the title of this made it clear that the post is arguing that ASIs won't spare humanity because of trade, and isn't saying anything about whether ASIs will want to spare humanity for some other reason. This is confusing because lots of people around here (e.g. me and many other commenters on this post) think that ASIs are likely to not kill all humans for some other reason.

(I think the arguments in this post are an okay defense of "ASI wouldn't spare humanity because of trade" and "ASIs are pretty likely to be scope-sensitively-maximizing enough that it's a big problem for us".)

<controversial statement>

This statement had two parts. Part 1:

What if objectionists had a correct thermodynamics-style heuristic that implied superintelligence/RSI is impossible, but which could not answer the question of where exactly it failed? Then the failure of objectionists doesn't mean they were wrong.

We have to be willing to investigate the new evidence as it arrives, perform root cause analysis on why A but not B happened, and use this to update our models.

And the evidence I've gotten since then suggests something like "it is impossible to do something without assistance from a higher power"/"greater things can cause lesser things but not vice versa", as a sort of generalization of the laws of thermodynamics.

If appropriate thought had been applied by a knowledgeable person back in 2004, maybe they could have taken this model and realized that nanotech violates this ordering constraint while AlphaProteo does not. Either way, we have the relevant info now.

And part 2:

The particular way the objectionists failed was in that they didn't give a concrete prediction that matched the way stuff played out.

Part 2 is what Eliezer said was false, but it's not really central to my point (hence why I didn't write much about it in the original thread), and so it is self-sabotaging of Eliezer to zoom into this rather than the actually informative point.

To be clear, I mean "your communication in this particular thread".

Pattern:

<controversial statement>

<this statement is false>

<controversial statement>

<this statement is false>

<mix of "this is trivially true because" and "here is my blogpost with esoteric terminology">

The following responses from EY are more in genre "I ain't reading this", because he is more using you as example for other readers than talking directly to you, with following block. 

On the topic of bistable perception, this is one of my favorite examples:

(Animated version)

I suspect this won't get published until November at the earliest, but I am already delightfully pleased with this bit:

Canada geese fly overhead, honking. Your inner northeast Ohioan notices that you are confused; it’s the wrong season for them to migrate this far south, and they’re flying westwards, anyways.

A quick Google discovers that some Canada geese have now established themselves non-migratorily in the Bay Area:

"The Migratory Bird Treaty Act of 1918 banned hunting or the taking of eggs without a permit. These protections, combined with an increase in desirable real estate—parks, golf course and the like—spurred a dramatic turnaround for the species. Canada geese began breeding in the Bay Area—the southern end of their range – in the late 1950s."

You nod, approvingly; this clearly is another part of the East Bay’s well-known, long-term philanthropic commitment to mitigating Acher-Risks.