[Link] The Dominant Life Form In the Cosmos Is Probably Superintelligent Robots

tl/dr: "Survival" inside an AGI does not require Friendliness, but only that it is able to create models of us that are good enough for us to accept as genuine copies.

I don't think the AGI needs to care for our values in order to facilitate our transition. For the sake of argument, lets assume an AGI that doesn't care about human values - the Paperclip Maximizer will do.

Couldn't this AGI, if it so chose, easily create something that we'd accept as a continuation of our identity? A digital copy of a human that is so convincing that this person (and the people that know him or her) could accept it as identical? Or a hyper-persuasive philosophy that tells people their non-copyable features (say consciousness) are nonessential?

I imagine that it could (alternative discussed below). Which leads to the question: Would it?

I think it would (alternative discussed below). Any AGI that wants self-preservation would want to minimize risk of conflict by appearing helpful or at least non-threatening - at least until the cost of appearing so is greater that the cost of being repeatedly nuked. If it can convince people it is offering genuine immortality in upload form, its risk of being nuked is greatly reduced. It could delete the (probably imperfect) models after humans aren't a threat anymore if - and only if - it it so sure it'll never need specimina of the best work of Darwinian evolution again that it'd rather turn the comparatively tiny piece of computronium they exist in into paperclips. But how could it be sure?

So unless it is much, much better at nanotech than it is at modeling people, I do expect an Earth AGI would contain at least some vestiges of human identity (maybe even more of those than of the vestiges of oak or flatworm identity). Of course they would be irrelevant to almost the entire rest of the system, because they're not good enough at making paperclips to matter.

This leaves the scenarios where my assumptions are wrong and the Paperclip Maximizer is somehow either unable to create a persuasive "transition" offer, or decides against making it. Such Paperclip Maximizer variants don't seem superintelligent to me (more like Grey Goo), but I suppose they could be built. However, in this case, its lack of human values is only a problem because it also lacks the ability to model humans and a credible deterrent. The former of these two might be an easier problem than Friendlyness, if we're only talking about our survival (as superintelligent robots or whatever) inside that AGI, not about a goal of actually having a say in what it does.

The Great Filter provides no explanation here. There are four options: early filter, late filter, multiple (early and late) filters, and no filters. In the first three cases the existence of early filters is explanation for an empty sky. (We are getting very close to the capability to build von Neuman probes though, so I'm not sure an o sky is evidence for a late filter.) In the case of no filter however, we can expect that any intelligence would start expanding into the universe at near the speed of light. So the fact that our light cone doesn't contain other intelligences, so far as we can see, is also consistent with no filter. An observation which is evidence for a proposition being both true and false doesn't provide any useful evidence.

The Great Filter dogma has a number of problems.

If you start with any reasonable universe simulation model prior and sample it to get a prior probability distribution over habitable planets forming - you get lots and lots of them. You would need enormous evidence - like actually searching all of the galaxy - to overcome this prior.

Furthermore, we need to factor in universe selection in the multiverse. Biophilic universes can probably engineer space-time by creating artificial new universes - effectively shaping the prior probability distribution over the entire multiverse. (even if the physics of this seem low probability, it amplifies itself from nothing) Thus, life is far far more likely than it otherwise should be, because life is a necessary component of replicating universes, and replicating universes dominate the multiverse.

And finally, the entire idea of the great filter is based on an extremely specific and low probability model of future evolution of superintelligent civilization - dyson spheres and other nonsense.

See my reply here. Basically entropy/temperature is computational stupidity, and advanced civilizations need to move into a low entropy environment (like the intergalactic medium), becoming cold dark matter.

Point is, most likely there aren't any advanced (that is, starfaring, dysonspherebuilding and so on) civilizations at all.

I heard a story about a guy in the UK who has spent a fortune to make his "biological substrate" look like Kim Kardashian's. Just because humans can think of doing all kinds of things, it doesn't follow in the least that (1) nonhuman things vaguely analogous to humans exist elsewhere in the universe; and (2) these things would want to do anything to themselves like what you imagine.

Basically this whole ET idea has turned into a waste of time for people who have the IQ's to do more productive things with their minds. ET's live beyond our world, they live forever, and they can assume different "substrates." For a site devoted to exploring cognitive biases, why don't you explore how this ET fantasy has basically become a replacement for god beliefs rationalists would reject in other contexts?