LessWrong 2.0 Reader

At market open, the instability I was seeing appears to have passed. That did not last long. I'll keep checking it.

Hmmm, this is a good point -- but here's a counter that just now occurred to me:

Let's disambiguate "intelligence" into a bunch of different things. Reasoning, imitation, memory, data-efficient learning, ... the list goes on. Maybe the complete bundle has only evolved once, in humans, but almost every piece of the bundle has evolved separately many times.

In particular, the number 1 thing people point to as a candidate X for "X is necessary for TAI and we don't know how to make AIs with X yet and it's going to be really hard to figure it out soon" is data-efficient learning.

But data-efficient learning has evolved separately many times; AlphaStar may need thousands of years of Starcraft to learn how to play, but dolphins can learn new games in minutes. Games with human trainers, who are obviously way out of distribution as far as Dolphin's ancestral environment is concerned.

The number 2 thing I hear people point to is "reasoning" and maybe "causal reasoning" in particular. I venture to guess that this has evolved a bunch of times too, based on how various animals can solve clever puzzles to get pieces of food.

(See also: https://www.lesswrong.com/posts/GMqZ2ofMnxwhoa7fD/the-octopus-the-dolphin-and-us-a-great-filter-tale [LW · GW] )

Your paragraph that outlines your position mixes multiple different things into the concept of reason.

There's the intelligence of individual scientists or engineers, there are conceptual issues and there's the quality of institutions.

An organizations that's a heavily disfunctional immoral maze is going to innovate less new technology then an organization with access to the same resources but with a better organizational setup. 

When it comes to raw intelligence that a lot of the productive engineers have an IQ that far exceeds that of the average population.

Conceptual insights like the idea of running controlled trials heavily influence the medical technology that can be developed in our society. We might have had concepts that would have allowed us to produce a lot more vaccines against COVID-19 much earlier

I suppose by 'the universe' I meant what you would call the inflationary multiverse, that is including distant regions we are now out of contact with. I personally tend not to call regions separated by mere distance separate universes.

"and the only impact of our actions with infinite values is the number of black holes we create."

Yes, that would be the infinite impact I had in mind, doubling the number would double the number of infinite branching trees of descendant universes.

Re simulations, yes, there is indeed a possibility of influencing other levels, although we would be more clueless, and it is a way for us to be in a causally connected patch with infinite future.

I believe this echos out my thoughts perfectly, I might quote it in full if I ever do get around to reviving that draft :))

The bit about "perfect" as not giving slack for development, I think, could be used even in the single individual scenario if you assume any given "ideal" action as lower chance of discovering something potential useful than a "mistake". I.e. adding:

• Actions have unintended and unknown consequences that reveal an unknown landscape of possibilities
• Actions have some % chance of being "optimal", but one can never be 100% certain they are so, just weigh them as having a higher or lower chance of being so
• "optimal" is a shifting goal-post and every action will potentially shift it

I think the "tinkerer" example is interesting, but even that assumes "optimal" to be goals dictates by natural-selection, and as per the sperm bank example, nobody care about those goals per-say, their subconscious machinery and the society they live in "cares". So maybe a complementary individual-focused example would be a world in which inactivity is equivalent to the happiest state possible (i.e. optimal for the individual) or at least some form of action that does not lend to itself or something similar to it's host being propagated through time.

Can you go into more detail for #4?

This, I assume, you'd base on a "hasn't happened before, no other animal or thing similar to us is doing it as far as we know, so it's improbable we will be able to do it" type assumption? Or something different?

I don't necessarily think you have to take the "AI" example for the point to make sense though.

I think "reasoning your way to a distant inference", as a human, is probably a far less controversial example that could be used here. In that most people here seem to assume there are ways to make distant inferences (e.g. about the capabilities of computers in the far off future), which historically seems fairly far fetched, it almost never happens when it does it is celebrated, but the success rate seems fairly small and there doesn't seem to be a clear formula for it that works.

This example by Tom Scott is excellent imo

It is a feature of the AI system but it's very important to first discover proper alignment before discovering AGI. If you randomly go about making discoveries it's more likely that you end up discovering AGI and ending humanity before discovering proper alignment.