LessWrong 2.0 Reader

i think this would not happen for the same fundamental reason that an aligned superintelligence can foresee whatever you can, and prevent / not cause them if it agrees they'd be worse than other possibilities. (more generally, "an aligned superintelligence would cause some bad-to-it thing" is contradictory, usually^[1].)

(i wonder if you're using the term 'superintelligence' in a different way though, e.g. to mean "merely super-human"? to be clear i definitionally mean it in the sense of optimal)

(tangentially: the 'nations' framing confuses me)^[2]

I think the main point is that what's worse than other possibilities partially depends on your value system at the start, and there is no non-circular way of resolving deep enough values conflicts such that you can always prevent conflict, so with differing enough values, you can generate conflict on it's own.

(Nitpick that doesn't matter, but when I focus on superintelligence, I don't focus on the AI literally doing optimal actions, because that leads to likely being wrong about what AIs can actually do.)

On the nations point, my point here is that people will program their superintelligences with quite different values, and the superintelligences will disagree about what counts as optimal from their lights, and if the disagreements are severe enough (which I predict is plausible if AI development cannot be controlled at all), conflict can definitely happen between the superintelligences, even if humans no longer are the main players.

Also, it's worth it to read these posts and comments, because I perceive some mistakes that are common amongst rationalists:
 

https://www.lesswrong.com/posts/895Qmhyud2PjDhte6/responses-to-apparent-rationalist-confusions-about-game [LW · GW]

https://www.lesswrong.com/posts/HFYivcm6WS4fuqtsc/dath-ilan-vs-sid-meier-s-alpha-centauri-pareto-improvements#jpCmhofRBXAW55jZv [LW(p) · GW(p)]

i think i wrote before that i agree (trivially) that not all possible values can be maximally satisfied; still, you can have the best possible world, which i think on this axis would look like "there being very many possible environments suited to different beings preferences (as long as those preferences are not to cause suffering to others)" instead of "beings with different preferences going to war with each other" (note there is no coordination problem which must be solved for that to happen. a benevolent superintelligence would itself not allow war (and on that, i'll also hedge that if there is some tragedy which would be worth the cost of war to stop, an aligned superintelligence would just stop it directly instead.))

 

I agree you can have a best possible world (though that gets very tricky in infinite realms due to utility theory breaking at that point), but my point here is that the best possible world is relative to a given value set, and also quite unconstrained, and your vision definitely requires other real-life value sets to lose out on a lot, here.

Are you assuming that superintelligences will have common enough values for some reason? To be clear, I think this can happen, assuming AI is controlled by a specific group that has enough of a monopoly on violence to prevent others from making their own AI, but I don't have nearly the confidence that you do that conflict is always avoidable by ASIs by default.

I'm thinking of cases like Eliezer's Politics is the Mind-Killer [LW · GW], which makes the relatively narrow claim that politically loaded examples are bad examples for illustrating principles of rationality in the context of learning and teaching those principles, so they should be avoided when a less politicized alternative is available. I think this falsely assumes that it's feasible under current circumstances for some facts to be apolitical in the absence of an active, political defense of the possibility of apolitical speech. But that's a basically reasonable and sane mistake to make. Then I see LessWrongers proceed as though Politics is the Mind-Killer established canonically that it is bad to mention when someone is saying or doing something politically loaded, which interferes with the sort of defense that Politics is the Mind-Killer implicitly assumed was a solved problem.

Or how Eliezer both explicitly wrote at length against treating intellectual authorities as specially entitled to opinions AND played with themes of being an incomprehensibly powerful optimization process, but the LessWrong community ended up crystallizing around an exaggerated version of the latter while mostly ignoring his explicit warnings against authority-based reasoning. Eliezer's personally commented on this [LW(p) · GW(p)] (higher-context link [LW(p) · GW(p)] that may take longer to load):

"How dare you think that you're better at meta-rationality than Eliezer Yudkowsky, do you think you're special" - is somebody trolling? Have they never read anything I've written in my entire life? Do they have no sense, even, of irony? Yeah, sure, it's harder to be better at some things than me, sure, somebody might be skeptical about that, but then you ask for evidence or say "Good luck proving that to us all eventually!" You don't be like, "Do you think you're special?" What kind of bystander-killing argumentative superweapon is that? What else would it prove?

I really don't know how I could make this any clearer. I wrote a small book whose second half was about not doing exactly this. I am left with a sense that I really went to some lengths to prevent this, I did what society demands of a person plus over 10,000% (most people never write any extended arguments against bad epistemology at all, and society doesn't hold that against them), I was not subtle. At some point I have to acknowledge that other human beings are their own people and I cannot control everything they do - and I hope that others will also acknowledge that I cannot avert all the wrong thoughts that other people think, even if I try, because I sure did try. A lot. Over many years. Aimed at that specific exact way of thinking. People have their own wills, they are not my puppets, they are still not my puppets even if they have read some blog posts of mine or heard summaries from somebody else who once did; I have put in at least one hundred times the amount of effort that would be required, if any effort were required at all, to wash my hands of this way of thinking.

Or how Eliezer wrote about how modern knowledge work has become harmfully disembodied and dissociated from physical reality - going into detail about how running from a tiger engages your whole sensorimotor system in a way that staring at a computer screen doesn't [LW · GW] - but lots of Lesswrongers seem to endorse and even celebrate this very dissociation from physical reality in practice.

When personal life expectancy of these same people alive today is something like 1e34 years, billions of years is very little.

How long would it even take to reach any of these places? Billions of years, right?

I think the 20 years somewhat unambiguously refers to timelines until AGI is built. 

Separately, “the sun is a battery” I think also doesn’t really imply anything about the sun getting dismantled, if anything it seems to me imply explicitly that the sun is still intact (and probably surrounded by a Dyson swarm or sphere). 

Cool stuff!

I'm a little confused what it means to mean-ablate each node...

Oh, wait. ctrl-f shows me the Non-Templatic data appendix. I see, so you're tracking the average of each feature, at each point in the template. So you can learn a different mask at each token in the template and also learn a different mean (and hopefully your data distribution is balanced / high-entropy). I'm curious - what happens to your performance with zero-ablation (or global mean ablation, maybe)?

Excited to see what you come up with for non-templatic tasks. Presumably on datasets of similar questions, similar attention-control patterns will be used, and maybe it would just work to (somehow) find which tokens are getting similar attention, and assign them the same mask.

It would also be interesting to see how this handles more MLP-heavy tasks like knowledge questions. maybe someone clever can find a template for questions about the elements, or the bibliographies of various authors, etc.

Yeah, the main takeaway is that you must have all decision procedures as look-up tables, in the worst case.

I agree.

When applied to object-level behavior like stealing cookies, this kind of norm internalization is ethically neutral. But when applied to protocols and coordination mechanisms, this becomes part of how shame-based coordination infiltrates and subverts communities doing something more interesting - people who recognize and try to leave bad communities end up recreating those same dysfunctional behaviors in the better communities they seek out.

In my reply to CstineSublime on pecking orders [LW(p) · GW(p)] I explored how this works through specific social mechanisms like using self-deprecation to derail accountability.

I think it might have been important to the process for this post to exist (so you can tell Elizabeth did the work), but for there still to be a shorter post that just gets the point across. 

But we already knew that some people think AGI is near and others think it's farther away!

And what do you conclude based on that?

I would say that as those early benchmarks ("can beat anyone at chess", etc.) are achieved without producing what "feels like" AGI, people are forced to make their intuitions concrete, or anyway reckon with their old bad operationalizations of AGI.

The relation between the real world and our intuition is an interesting topic. When people’s intuitions are violated (e.g., the Turing test is passed but it doesn’t “feel like” AGI), there’s a temptation to try to make the real world fit the intuition, when it is more productive to accept that the intuition is wrong. That is, maybe achieving AGI doesn’t feel like you expect. But that can be a fine line to walk. In any case, privileging an intuitive map above the actual territory is about as close as you can get to a “cardinal sin” for someone who claims to be rational. (To be clear, I’m not saying you are doing that.)

what do you think about Robin Hanson's culture and value drift?

https://www.overcomingbias.com/p/how-fix-cultural-drift