LessWrong 2.0 Reader

In software, network effects are strong. A solution people are already familiar with has an advantage. A solution integrated with other solutions you already use has an advantage (and it is easier to do the integration when both solutions are made by you). You can further lock the users in by e.g. creating a marketplace where people can sell plugins to your solution. Compared to all of this, things like "nice to use" remain merely wishes.

I've added a link to listen on Apple Podcasts.

Sorry - YouTube's taking an abnormally long time to process the video.

Orthogonality Thesis [? · GW]

The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.

It basically says that intelligence and goals are independent

Images from A caveat to the Orthogonality Thesis [LW · GW].

While I claim that all intelligence that is capable to understand "I don't know what I don't know" can only seek power (alignment is impossible).

the ability of an AGI to have arbitrary utility functions is orthogonal (pun intended) to what behaviors are likely to result from those utility functions.

As I understand you say that there are Goals on one axis and Behaviors on other axis. I don't think Orthogonality Thesis is about that.

Yes, that would be cool [LW(p) · GW(p)].

Next to the author name of a post orcomment, there's a post-date/time element that looks like "1h 🔗". That is a copyable/bookmarkable link.

Another example of Mental Masturbation I decided to exclude from the main text:

• While playing a piano piece I already know fairly well but am lacking in one specific spot, my brain will say, “Hey, rather than painstakingly drill the 1-2 measures that are tricky, you should play the entire song through to get a balanced perspective of the piece!” To fix this, I put a sticky note on my music stand that says: “Stop Mental Masturbation and practice the hard part!”

Capabilities are likely to cascade once you get to Einstein-level intelligence, not just because an AI will likely be able to form a good understanding of how it works and use this to optimize itself to become smarter^[4][5], but also because it empirically seems to be the case that when you’re slightly better than all other humans at stuff like seeing deep connections between phenomena, this can enable you to solve hard tasks like particular research problems much much faster (as the example of Einstein suggests).

1. Aka: Around Einstein-level, relatively small changes in intelligence can lead to large changes in what one is capable to accomplish.

OK but if that were true then there would have been many more Einstein like breakthroughs since then. More likely is that such low hanging fruit have been plucked and a similar intellect is well into diminishing returns. That is given our current technological society and >50 year history of smart people trying to work on everything if there are such breakthroughs to be made, then the IQ required is now higher than in Einsteins day.

I'm arguing exactly the opposite; experts want to make comparisons carefully, and those trying to transmit the case to the general public should, at this point, stop using these rhetorical shortcuts that imply wrong and misleading things.

On net, the analogies being used to try to explain are bad and misleading.

I agree that I could have tried to convey a different message, but I don't think it's the right one [LW · GW]. Anyone who wants to dig in can decide for themselves, but you're arguing that ideal reasoners won't conflate different things and can disentangle the similarities and differences, and I agree, but I'm noting that people aren't doing that, and others seem to agree [LW · GW].

I wish I could bookmark comments/shortform posts.