Open Thread, Aug. 22 - 28, 2016

Agreed.

At some moment, there was a period when there were debates about NR on LW, simply because those were the times when people on LW were okay with discussing almost anything. And NR happened to be one of the many interesting topics to discuss.

The problem was, everyone else on this planet was completely ignoring NR at that moment. Starved for attention, NRs decided to focus their recruitment attempts on LW audience, and started associating themselves with LW on their personal blogs. After repeating this lie enough times, it started getting quoted by other people as a fact. (Such as on Breitbart now.)

The LW debates of NR were interesting at the beginning, but they soon became repetitive (and it was kinda impossible to decipher what specifically Moldbug is saying in his long texts, other than that "Cthulhu is always swimming left"). The few local NR fans decided to start their own collective blog.

The largest long-term impact is that one hardcore NR fan decided to stay on LW, despite repeated bans, and created an army of automated sockpuppets, downvoting comments of people he perceives hostile to the NR idea, plus any comments about himself or his actions here. (I expect this comment to have -10 karma soon, but whatever.)

Probably long before NR existed, LW had the "politics is the mindkiller" approach to politics. This didn't prevent us from having relatively friendly discussions of political topics once in a while. But the automated downvoting of every perceived enemy of NR had a chilling effect on such debates.

I don't know whether you've heard of it, but someone wrote an ebook called "Neoreaction a Basilisk" that claims Eliezer Yudkowsky was an important influence on Mencius Moldbug and Nick Land. There was a lot of talk about it on the tumblr LW diaspora a few months back.

I don't think any of the thought leaders of either group were significantly influential to the other.

Yvain did say that he was influenced by Moldbug.

Touche.

I seem to recall a Yudkowsky anti-NRx comment on Facebook a year or two ago, but does anyone recall / have a link to an earlier disagreement on Yudkowsky's part?

NRx is systematized hatred.

Am NRx, this assertion is false.

Because it's a link to a godawful clickbait piece-of-crap writing that has no relationship to LW, a sprinkling of related terms notwithstanding. Just because Gleb has an account here doesn't mean this is a good place to post about his excursions into the slime pools of 'net pseudo-journalism.

There was a lengthy and informative discussion of why many EA/LW/diaspora folks don't like Gleb's work on Facebook last week. I found both Owen Cotton-Barratt's mention of the unilateralist's curse, and Oliver Habryka's statement that people dislike what Gleb is doing largely because of how much he's done to associate himself with rationality and EA, to be informative and tactful.

Agreed, to me it also makes no sense to do cash transfers to people with above average income. I see basic income as mainly about a social safety net.

Why DNA? For most of taxonomy's existence, DNA "didn't exist". Just because genotyping changed the game doesn't mean there was no game before.

There we go. No wonder I couldn't find it, it wasn't on LessWrong, and also a lot older than three years. Thanks!

This is a good point. Pretty much all the things we're optimizing for which aren't our values are due to coordination problems. (There's also Akrasia/addiction sorts of things, but that's optimizing for values which we don't endorse upon reflection, and so arguably isn't as bad as optimizing for a random part of value-space.)

So, Moloch might optimize for things like GDP instead of Gross National Happiness, and individuals might throw a thousand starving orphans under the bus for a slightly bigger yacht or whatever, but neither is fully detached from human values. Even if U(orphans)>>U(yacht), at least there’s an awesome yacht to counterbalance the mountain of suck.

I guess the question is precisely how diverse human values are in the grand scheme of things, and what the odds are of hitting a human value when picking a random or semi-random subset of value-space. If we get FAI slightly wrong, precisely how wrong does it have to be before it leaves our little island of value-space? Tiling the universe with smiley faces is obviously out, but what about hedonium, or wire heading everyone? Faced with an unwinnable AI arms race and no time for true FAI, I’d probably consider those better than nothing.

That's a really, really tiny sliver of my values though, so I'm not sure I'd even endorse such a strategy if the odds were 100:1 against FAI. If that's the best we could do by compromising, I'd still rate the expected utility of MIRI's current approach higher, and hold out hope for FAI.

I don't that's a good description of the orthogonality thesis.

Probably not, but it highlights the relevant (or at least related) portion. I suppose I could have been more precise by specifying terminal values, since things like paperclips are obviously instrumental values, at least for us.

Human's don't effectively self modify

Agreed, except in the trivial case where we can condition ourselves to have different emotional responses. That's substantially less dangerous, though.

Partial optimization for human values is easier than total optimization.

Why do you believe that?

I'm not sure I do, in the sense that I wouldn't assign the proposition >50% probability. However, I might put the odds at around 25% for a Reduced Impact AI architecture providing a useful amount of shortcuts.

That seems like decent odds of significantly boosting expected utility. If such an AI would be faster to develop by even just a couple years, that could make the difference between winning and loosing an AI arms race. Sure, it'd be at the cost of a utopia, but if it boosted the odds of success enough it'd still have enough expected utility to compensate.

I'm pretty sure I also mostly disagree with claim 6. (See my other reply below.)

The only specific concrete change that comes to mind is that it may be easier to take one person's CEV than aggregate everyone's CEV. However, this is likely to be trivially true, if the aggregation method is something like averaging.

If that's 1 or 2 more lines of code, then obviously it doesn't really make sense to try and put those lines in last to get FAI 10 seconds sooner, except in a sort of spherical cow in a vacuum sort of sense. However, if "solving the aggregation problem" is a couple years worth of work, maybe it does make sense to prioritize other things first in order to get FAI a little sooner. This is especially true in the event of an AI arms race.

I’m especially curious whether anyone else can come up with scenarios where a maxipok strategy might actually be useful. For instance, is there any work being done on CEV which is purely on the extrapolation procedure or procedures for determining coherence? It seems like if only half our values can easily be made coherent, and we can load them into an AI, that might generate an okay outcome.

AI gets it wrong then well that's it.

Not necessarily, depends on your AI and how god-like it is.

In the XIX century you could probably make the same argument about corporations: once one corporation rises above the rest, it will use its power to squash competition and install itself as the undisputed economic ruler forever and ever. The reality turned out to be rather different and not for the lack of trying.

That was pretty much my take. I get the feeling that "okay" outcomes are a vanishingly small portion of probability space. This suggests to me that the additional marginal effort to stipulate "okay" outcomes instead of perfect CEV is extremely small, if not negative. (By negative, I mean that it would actually take additional effort to program an AI to maximize for "okay" outcomes instead of CEV.)

However, I didn't want to ask a leading question, so I left it in the present form. It's perhaps academically interesting that the desirability of outcomes as a function of “similarity to CEV” is a continuous curve rather than a binary good/bad step function. However, I couldn't really see any way of taking advantage of this. I posted mainly to see if others might spot potential low hanging fruit.

I guess the interesting follow up questions are these: Is there any chance that humans are sufficiently adaptable that human values are more than just an infinitesimally small sliver of the set of all possible values? If so, is there any chance this enables an easier alternative version of the control problem? It would be nice to have a plan B.

Done using inverted text

You say blackmail, I say altruistic punishment.