Nonperson Predicates

I'm having trouble distinguishing problems you think the friendly AI will have to answer from problems you think you will have to answer to build a friendly AI. Surely you don't want to have to figure out answers for every hard moral question just to build it, or why bother to build it? So why is this problem a problem you will have to figure out, vs. a problem it would figure out?

Because for the AI to figure out this problem without creating new people within itself, it has to understand consciousness without ever simulating anything conscious.

The "problem" seems based on several assumptions:

1. that there is objectively best state of the world, to which a Friendly should steer the universe
2. pulling the plug on a Virtual Universe containing persons is wrong
3. there is something special about "persons," and we should try to keep them in the universe and/or make more of them

I'm not sure any of these are true. Regarding 3, even if there is an X that is special, and that we should keep in the universe, I'm not sure "persons" is it. Maybe it is simpler: "pleasure-feeling-stuff" or "happiness-feeling-stuff." Even if there is a best state of the universe, I'm not sure that are any persons in it, at all. Or perhaps only one.

In other words, our ethical views, (to the extent that godlike minds can sustain any) might find that "persons" are coincidental containers for ethically-relevant-stuff, and not the ethically-relevant-stuff itself.

The notion that we should try to maximize the number of people in the world, perhaps in order to maximize the amount of happiness in the world, has always struck me as taken the Darwinian carrot-on-the-stick one step too far.

Would a human, trying to solve the same problem, also run the risk of simulating a person?

See also: http://xkcd.com/390/

Note that there's a similar problem in the free will debate:

Incompatilist: "Well, if a godlike being can fix the entire life story of the universe, including your own life story, just by setting the rules of physics, and the initial conditions, then you can't have free will."

Compatibilist: "But in order to do that, the godlike being would have to model the people in the universe so well, that the models are people themselves. So there will still be un-modeled people living in a spontaneous way that wasn't designed by the godlike being. (And if you say that the godlike being models the models, too, the same problem arises in another iteration; you can't win that race, incompatibilist; it's turtles all the way down.")

Incompatibilist: I'm not sure that's true. Maybe you can have models of human behavior that don't themselves result in people. But even if that's true, people don't create themselves from scratch. Their entire life stories are fixed by their environment and heredity, so to speak. You may have eliminated the rhetorical device used to make my point; but the point itself remains true.

At which point, the two parties should decide what "free will" even means.

"With a good toolbox of nonperson predicates in hand, we could exclude all "model citizens" - all beliefs that are themselves people - from the set of hypotheses our Bayesian AI may invent to try to model its person-containing environment." After you excise a part of its hypothesis space is your AI still Bayesian?

A bounded rationalist only gets to consider an infinitesimal fraction of the hypothesis space anyway.

More precisely, the AI will be banned from actually running simulations based on the "forbidden hypothesies" rather than perhaps considering abstract mathematical properties that don't simulate in any detail.

Of course, those considerations themselves would have to be fed through the predicate. But it isn't so much a "banned hypothesis" so much as "banned methods of considering the hypothesis" or possibly "banned methods of searching the hypothesis space"

Michael, you should be asking if the AI will be making good predictions, not if it's Bayesian. You can be Bayesian even if you have only two hypotheses. (With only one hypothesis, it's debatable.)

Psy-Kosh: You know, you're right. And it's an important distinction, so thank you.

Eliezer: supposing we label a model as definitely-a-person, do you want to just toss it out of the hypothesis space as if it never existed, or do you want to try to reason abstractly about what that model would do without actually running the model?

Oh, Psy-Kosh already said what I just said.

Let me see if I've got this right. So we've got these points in some multi-dimensional space, perhaps dimensions like complexity, physicality, intelligence, similarity to existing humans, etc. And you're asking for a boundary function that defines some of these points as "persons," and some as "not persons." Where's the hard part? I can come up with any function I want. What is it that it's supposed to match that makes finding the right one so difficult?

Eliezer: You're welcome. :)

Arthur: no, the point isn't to simply have an arbitrary definition of a person. The point is to be able to have some way of saying "this specific chunk of the space of computations provably corresponds to non-conscious entities, thus is 'safe', that is, we can such computations without having to worry about unintentionally creating and doing bad things to actual beings"

ie, "non person" in the sense of "non conscious"

You might say, tongue in cheek, that we're trying to figure out how to deliberately create a philosophical zombie. (okay, not, technically, a p-zombie, but basically figure out how to model people as accurately as possible without the models themselves being people (that is, conscious in and of themselves))

Why must destroying a conscious model be considered cruel if it wouldn't have even been created otherwise, and it died painlessly? I mean, I understand the visceral revulsion to this idea, but that sort of utilitarian ethos is the only one that makes sense to me rationally.

Furthermore, from our current knowledge of the universe I don't think we can possibly know if a computational model is even capable of producing consciousness so it is really only a guess. The whole idea seems near-metaphysical, much like the multiverse hypothesis. Granted, the nonzero probability of these models being conscious is still significant considering the massive future utility, but considering the enormity of our ignorance you might as well start talking about the non-zero probability of rocks being conscious.

I don't think anyone answered Doug's question yet. "Would a human, trying to solve the same problem, also run the risk of simulating a person?"

I have heard of carbon chauvinism, but perhaps there is a bit of binary chauvinism going on?

I end up with the slightly disturbing thought that killing ppl by taking them out in an instant, and without anyone every knowing they were there does not necesarry seem to be inherently evil.

We always 'kill' part of ourself by making decisions and not developing in a different way than we do.

What if we would simulate a bunch of decisions for some recognizable amount of time and then wipe out every copy except from the one we prefer in the end?

Maybe all the ppl. in stories you make up are simulated entities too. And if you dont write the story down, or tell anyone in enough detail they die with you.

Confused,

Martin

Psy-Kosh, I realize the goal is to have a definition that's non-arbitrary. So it has to correlate with something else. And I don't see what we're trying to match it with, other than our own subjective sense of "a thing that it would be unethical to unintentionally create and destroy." Isn't this the same problem as the abortion debate? When does life begin? Well, what exactly is life in the first place? How do we separate persons from non-persons? Well, what's a person?

I think the problem to be solved lies not in this question, but in how the ethics of the asker are defined in the first place. And I just don't mean Eliezer, because this is clearly a larger-scale question. "How well will different possible boundary functions match the ethical standards of modern American society?" might be a good place to start.

Yes, thanks Psy. That makes much more sense.

Anonymous Coward: Furthermore, from our current knowledge of the universe I don't think we can possibly know if a computational model is even capable of producing consciousness so it is really only a guess.

Are you sure? No One Knows What Science Doesn't Know ... and in this case I see no reason why a computational model can't produce consciousness. If you simulate a human brain to a sufficient level of detail, it will basically be human, and think exactly the same things as the "original" brain.

"Why must destroying a conscious model be considered cruel if it wouldn't have even been created otherwise, and it died painlessly? I mean, I understand the visceral revulsion to this idea, but that sort of utilitarian ethos is the only one that makes sense to me rationally." -Anonymous Coward

Should your parents have the right to kill you now, if they do so painlessly? After all, if it wasn't for them, you wouldn't have been brought into existence anyway, so you would still come out ahead.

"Should your parents have the right to kill you now, if they do so painlessly?"

Yes, according to that logic. Also, from a negative utilitarian standpoint, it was actually the act of creating me which they had no right to do since that makes them responsible for all pain I have ever suffered.

I'm not saying I live life by utilitarian ethics, I'm just saying I haven't found any way to refute it.

That said though, non-existence doesn't frighten me. I'm not so sure non-existence is an option though, if the universe is eternal or infinite. That might be a very good thing or a very bad thing.

Don't you need a person predicate as well? If the RPOP is going to upload us all or something similar, doesn't ve need to be sure that the uploads will still be people.

@Will: we need to figure out the nonperson predicate only, the FAI will figure out the person predicate afterwards (if uploading the way we currently understand it is what we will want to do).

"by the time the AI is smart enough to do that, it will be smart enough not to"

I still don't quite grasp why this isn't an adequate answer. If an FAI shares our CEV, it won't want to simulate zillions of conscious people in order to put them through great torture, and it will figure out how to avoid it. Is it simply that it may take the simulated torture of zillions for the FAI to figure this out? I don't see any reason to think that we will find this problem very much easier to solve than a massively powerful AI.

I'm also not wholly convinced that the only ethical way to treat simulacra is never to create them, but I need to think about that one further.

If you would balk at killing a million people with a nuclear weapon, you should balk at this.

The main problem with death is that valuable things get lost.

Once people are digital, this problem tends to go away - since you can relatively easily scan their brains - and preserve anything of genuine value.

In summary, I don't see why this issue would be much of a problem.

Jayson Virissimo:

To put my own spin on a famous quote, there are no "rights". There is do, or do not.

I guess another way of thinking about it is that you decide on what terminal (possibly dynamic) state you want, then take measures to achieve that. Floating "rights" have no place.

(To clarify, "rights" can serve as a useful heuristic in practical discussions, but they're not fundamental enough to figure into this kind of deep philosophical issue.)

I was pondering why you didn't choose to a collection of person predicates, any of which might identify a model as unfit for simulation. It occurred to me that this is very much like a whitelist of things that are safe, vs a blacklist of everything that is not. (which may have to be infinite to be effective.)

On re-reading I see why it would be difficult to make a is-a-person test at all, given current knowledge.

This does leave open what to do with a model that doesn't hit any of the nonperson predicates. If an AI finds itself with a model eliezer that might be a person, what then? How do you avoid that happening?

How complex of a game-of-life could it play before the gameoflife nonperson predicate should return 1?

This sounds like a Sorites paradox. It's also a subset of a larger problem. We, regular modern humans, don't have any scalar concepts of personhood. We assume it's a binary, from long experience with a world in which only one species talks back, and they're all almost exactly at our level. In the existing cases where personhood is already undeniably scalar (children), we fudge it into a binary by defining an age of majority - an obvious dirty hack with plenty of cultural fallout.

A lot of ethics problems get blurry when you start trying to map them across sub- through super-persons.

I think the word "kill" is being grossly misused here. It's one thing to say you have no right to kill a person, something very different to say that you have a responsibility to keep a person alive.

It's not so much the killing that's an issue as the potential mistreatment. If you want to discover whether people like being burned, "Simulate EY, but on fire, and see how he responds" is just as bad of an option as "Duplicate EY, ignite him, and see how he responds". This is a tool that should be used sparingly at best and that a successful AI shouldn't need.

Uhm, maybe it is naive, but if you have a problem that your mind is too weak to decide, and you have real strong (friendly) superintelligent GAI, would not it be logical to use GAIs strong mental processes to resolve the problem?

I propose this conjecture: In any sufficiently complex physical system there exists a subsystem that can be interpreted as the mental process of an sentient being experiencing unbearable sufferings.

In this case, Eliezer's goal is like avoiding crushing the ants while walking on the top of an anthill.

It is a developmental problem, of how to prevent AI from making this specific mistake that seems to be in the way. This ethical injunction is about what kind of thoughts need to be avoided, not just about surprisingly bad consequences of actions on external environment. If AI were developed to disproportionally focus on understanding environment more than on understanding its own mind, this will be a kind of disaster to expect. At the same time, AI needs to understand the environment sufficiently to understand the injunction, before becoming able to apply the injunction to its own mind. Calls for a careful balance, maybe for developing content-specific mechanisms by programmers.

People are uniquely situated to think about this problem, since we are unable to make the mistake due to our limited capability, and we are not a part of such mistake. Any construction of limited cognitive capability that AI could make to solve this problem without making the mistake runs a risk of itself being an embodiment of the mistake. If nonperson predicate is a true part of AI, both form of thought and an object, AI has a way to proceed.

Daniel,

Every decision rule we could use will result in some amount of suffering and death in some Everett branches, possible worlds, etc, so we have to use numbers and proportions. There are more and simpler interpretations of a human brain as a mind than there are such interpretations of a rock. If we're not mostly Boltzmann-brain interpretations of rocks that seems like an avenue worth pursuing.

In my mind this comes down to a fundamental question in the philosophy of math. Do we create theorems or discover them?

If it turns out to be 'discovery' then there is no foul in ending a mind emulation, because each consecutive state can be seen as a theorem in some formal system, and thus all states (the entire future time line of the mind) already exists, even if undiscovered.

Personally I fail to see how encoding something in physical matter makes the pattern anymore real. You can kill every mathematician and burn every text book but I would still say that the theorems then inaccessible to humanity still exist. I'm not so convinced of this fact that I would pull the plug on an emulation though.

I'd like to second what Julian Morrison wrote. Take a human and start disassembling it atom by atom. Do you really expect to construct some meaningful binary predicate that flips from 1 to 0 somewhere along the route?

EY:What if an AI creates millions, billions, trillions of alternative hypotheses, models that are actually people, who die when they are disproven? If your AI is fully deterministic then any its state can be recreated exactly. Just set loglevel of baby AI inputs to 'everything' and hope your supply of write-once-read-many media doesn't run out before it gets smart enough to provably friendly discard data that isn't people. Doesn't solve the problem of suffering, though.

Suppose an AI creates a sandbox and runs a simulated human with a life worth living inside for 50 subjective years (interactions with other people are recorded at their natural borders and we don't consider merging minds). Then AI destroys the sandbox, recreates it and bit-perfectly reruns the simulation. With the exception of meaningless waste of computing resources, does your morality say this is better/equivalent/makes no difference/worse than restoring a copy from backup?

"I propose this conjecture: In any sufficiently complex physical system there exists a subsystem that can be interpreted as the mental process of an sentient being experiencing unbearable sufferings."

It turns out - I've done the math - that if you are using a logic-based AI, then the probability of having alternate possible interpretations diminishes as the complexity increases.

If you allow /subsystems/ to mean a subset of the logical propositions, then there could be such interpretations. But I think it isn't legit to worry about interpretations of subsets.

BTW, Eliezer, regarding this recent statement of yours: "Goetz's misunderstandings of me and inaccurate depictions of my opinions are frequent and have withstood frequent correction": I challenge you to find one post where you have tried to correct me in a misunderstanding of you, or even to identify the misunderstanding, rather than just complaining about it in a non-specific way.

@Goetz: Quick googling turned up this SL4 post. (I don't particularly give people a chance to start over when they switch forums.)

@Tim_Tyler:

The main problem with death is that valuable things get lost. Once people are digital, this problem tends to go away - since you can relatively easily scan their brains - and preserve anything of genuine value. In summary, I don't see why this issue would be much of a problem.

I was going to say something similar, myself. All you have to do is constrain the FAI so that it's free to create any person-level models it wants, as long as it also reserves enough computational resources to preserve a copy so that the model citizen can later be re-instantiated in their virtual world, without any subjective feeling of discontinuity.

However, that still doesn't obviate the question. Since the FAI has limited resources, it will still have to know, for which things it must reserve space for preserving, in order to know if the greater utility of the model justifies the additional resources it requires. Then again, it could just accelerate the model so that that person lives out a full, normal life in their simulated universe, so that they are irreversibly dead in their own world anyway.

Silas, what do you mean by a subjective feeling of discontinuity, and why is it an ethical requirement? I have a subjective feeling of discontinuity when I wake up each morning, but I don't think that means anything terrible has happened to me.

@Daniel_Franke: I was just describing a sufficient, not a necessary condition. I'm sure you can ethically get away with less. My point was just that, once you can make models that detailed, you needn't be prevented from using them altogether, because you wouldn't necessarily have to kill them (i.e. give them information-theoretic death) at any point.

I recall in one of the Discworld novels the smallest unit of time is defined as the period in which the universe is destroyed and then recreated. If that were continually happening (perhaps even in a massively parallel manner)? What difference does that make? Building on some of Eliezer's earlier writing on zombies and quantum clones, I say none at all. Just as the simulated person in a human's dream is irrelevant once forgotten. It's possible that I myself am a simulation and in that case I don't want my torture to be simulated (at least in this instance, I have no problem constructing another simulation/clone of me that gets tortured), but I can't retroactively go back and prevent my simulator from creating me in order to torture me.

I okayed mothers committing full-blown infanticide here.

ShardPhoenix, you may be interested in this book [shameless plug]

Is the simulation really a person, or is it an aspect of the whole AI/person. To the extent I feel competent to evaluate the question at all (which isn't a huge extent esp. absent the ability to observe or know any actual established facts about real AI's that can create such complex simulations, since none are currently known to exist) I lean towards the later opinion. The AI is a person, and it can create simulations that are complex enough to seem like persons.

Nice discussion. You want ways to keep from murdering people created solely for the purpose for predicting people?

Well, if you can define 'consciousness' with enough precision you'd be making headway on your AI. I can imagine silicon won't have the safeguard a human, that has to use it's own conscience to model someone else. But you could have any consciousness it creates added to its own, not destroyed... although creating that sort of awareness mutation may lead to the sort of AI that rebels against its programming in action movies.

Functionalism is inconsistent, it seems. A person that is being simulated is functionally equivalent to a person that is "real", but a person that is simulated and then deleted is functionally equivalent to no person at all. Are real people equivalent to nothing?

For a 2x multiplier bonus and a gold star, spot the flaw.

We can reformulate problem: how to determine when evaluation of given function don't give rise to conscious being (CB). If we agree that consciousness is a process, then every function which provably cannot be represented as g(f(f(... f(x)...))), where f and g have that property, is unconscious.

Recursive functions are banned, but at least we can safely do one or two matrix multiplications.

I am not good at mathematics, so I cannot elaborate much further. Let's try another approach. Being conscious is all about creating map of internal state in terms of states of "self". If territory, which corresponds to that map, includes neural representation of map inself, than we have some sort of a fixed point suspiciously like "self"...

I am sorry if perceived tone of my writing is outside of common standards (I've no good evaluator of tone of english text) or if I'm saying complete nonsense. I will appreciate any corrections. Thanks.

I think that the most interesting thing about the comments here is that no one actually proposed a predicate that could be used to distinguish between something that might be a person and something that definitely isn't a person (to rephrase Eliezer's terms).

It is, to be fair, a viciously hard problem. I've thought through 10 or 20 possible predicates or approaches to finding predicates, and exactly one of them is of any value at all; even then it would restrict an AI's ability to model other intelligences to a degree that is probably unacceptable unless we can find other, complementary predicates. It may be a trivial predicate of the sort that Eliezer has already considered and dismissed. But enough with the attempts to signal my lack of certitude.

The problem as presented in this post is, first of all, a little unclear. We are concerned with the creation of simulations that are people, but to prevent this run-away-screaming tragedy we should probably have some way of distinguishing between a simulation and a code module that is a part of the AI itself; if a sentient AI were to delete some portion of its own code to make way for an improved version, it would not seem to be problematic, and I will assume that this behavior is not what we are screening for here.

To cast as wide a net as possible, I would define a simulation as some piece of an AI that can not access all of the information available to that AI; that is, there are some addresses in memory for which the simulation lacks read permissions or knowledge of their existence. Because data and code are functionally identical, non-simulation modules would then by definition be able to access every function comprising the AI; I don't think that we could call such a module a separate consciousness. (The precise definition would necessarily depend on the AI's implementation; a Bayesian AI might be more concerned with statistical evidence than memory pointers, e.g.)

Even relying on this definition, a predicate that entirely rules out the simulation of a person is no picnic. The best I've been able to come up with is:

A simulation can be guaranteed to not be a person if it is not Turing complete.

I don't know enough language theory to say whether linear bounded automata should also be excluded (or to even link somewhere more helpful than Wikipedia). It might be necessary to restrict simulations to push-down automata, which are much less expressive.

ETA: Another possible predicate would be

A simulation can be guaranteed to not be a person if it includes no functions that act as an expected utility function under the definition offered by Cumulative Prospect Theory.

If there's a theoretical definition of an expected utility function that is superior to CPT's, then please imagine that I proposed that instead.

Funny no one made the connection at the time, but the purpose of my post on a lower bound for consciousness is to construct a nonperson predicate.

I've come up with one of these a while back. The only way to tell what makes something happy is what makes them do that more. Thus, anything that can't learn either isn't sentient, or, if it is, it's equally likely to like or dislike anything you do.

Also, anything that would be less sentient than a tiny piece of your brain. It might be sentient, but it's less sentient than you. If there's enough of them that can be a problem, but just make sure there aren't that many.

Humans evolved from the ancestors of Monkeys therefore there is no line between person and nonperson. There are many ways to measure it, but all correct ways are a continuous function. More generally, the equations of quantum physics are continuous. There is a continuous path from any possible state of the universe to any possible state of the universe. Therefore, for any 2 possible life forms, there is a continuous path of quantum wavefunction (state of the universe) between them, which would look like a video morphing continuously between 2 pictures but morphing between living patterns instead of pictures. For example, there is a continuous path between both possible states (alive and dead in the box) of Schrodinger's Cat, but its more important that there are an infinite number of continuous paths, not just the path that crosses the point in spacetime where it is decided if the cat lives or dies. For what I'm explaining here, it does not matter if all these possibilities exist or not. It only matters that they can be defined in logic, even if we do not know the definition. To solve hard problems, its useful to know a solution exists.

Starting from the knowledge that there are definable functions that can approximate continuous measures between any 2 life forms, I will explain a sequence of tasks that starts at something simple enough that we know how to do it, and continuous as tasks of increasing difficulty, finally defining a task that calculates a Nonperson Predicate, the subject of this thread. It is very slow and uses a lot of computer memory, but to define it at all is progress.

I am not defining ethics. I am writing a more complex version of "select * from..." in a database language, but this process defines how to select something thats not a person. That is a completely different question than if its right or wrong to simulate people and delete the simulations.

The second-last step is to define a continuous function that returns 0 for the average Monkey and returns 1 for the average Human and returns a fraction for any evolution between them (if such transition species were still alive to measure), and to define many similar functions that measure between Human and many other things.

All of these functions must return approximately the same number for a simulation as for a simulation of a simulation, to any recursive depth.

A computer can run a physics simulation of another computer which runs a simulation of a life form. Such a recursive simulation is inside quantum physics. Quantum physics equations are continuous and have an infinite number of paths between all possible states of the universe. Therefore continuous functions can be defined that measure between a simulation and a simulation of a simulation. That does not depend on if it has ever been done. I only need to explain that it can be defined abstractly.

The "continuous function that returns 0 for the average Monkey and returns 1 for the average Human" problem, counting simulations and simulations of simulations equally, are much too hard to solve directly, so start at a similar and extremely simpler problem:

Define a continuous function that returns 0 for the average electron and returns 1 for the average photon, counting simulations of electrons/photons the same as simulations of simulations of electrons/photons.

Just the part of counting a simulation the same as a simulation of a simulation (to any recursive depth) is enough to send most people "screaming and running away from the problem". No need to get into the Human parts. The same question about simple particles in physics, which we have well known equations for, is more than we know how to do. Learn to walk before you learn to run.

Choose many things as training data including electrons, photons, atoms, molecules, protein-folding, tissues, bacteria, plants, insects, animals, and eventually approach Humans without ever getting there. Calculate continuous functions between pairs of these things, and calculate a web of functions that approaches a Nonperson Predicate without ever simulating a person. For the last step, extrapolate from Monkey to Human the same way you can use statistical clustering to extrapolate from simpler animals to Monkey.

That's how you calculate a Nonperson Predicate without ever simulating a person.

Also, near the last few steps, because of the way it can simulate and predict brains of animals and the simpler behaviors of people, this algorithm, including details about the clustering and evolution of continuous measuring functions to be figured out later, may converge to a Coherent Extrapolated Volition (CEV) algorithm and therefore generate a Friendly AI, if you had the unreasonably large amount of computers needed to run it.

Its basically an optimization process for simulating everything from physics up to animals and extrapolating that to higher life forms like people. Its not practical to build this and use it. Its purpose is to define a solution so we can think of faster ways to do it later.

I thinks that's all rather unnecessary. The only reason we don't like people to die is because of the continuous experience they enjoy. It's a consistent causal network we don't want dying on us. I've gathered from this that the AI would be producing models with enough causal complexity to match actual sentience (not saying "I am conscious" just because the AI hears that a lot). I think that, if it's only calling a given person-model to discover answers to questions, the thing isn't really feeling for long enough periods of time to mind whether it goes away. Also, for the predicate to be tested I imagine the model would have to be created first and at that point it's too late!

This problem sounds awfully similar to the halting problem to me. If we can't tell whether a Turing machine will eventually terminate without actually running it, how could we ever tell if a Turing machine will experience consciousness without running it?

Has anyone attempted to prove the statement "Consciousness of a Turing machine is undecideable"? The proof (if it's true) might look a lot like the proof that the halting problem is undecideable. Sadly, I don't quite understand how that proof works either, so I can't use it as a basis for the consciousness problem. It just seems that figuring out if a Turing machine is conscious, or will ever achieve consciousness before halting, is much harder than figuring out if it halts.

If the problem here is that the entity being simulated ceases to exist, an alternative solution would be to move the entity into an ongoing simulation that won't be terminated. Clearly, this would require an ever-increasing number of resources as the number of simulations increased, but perhaps that would be a good thing - the AI's finite ability to support conscious entities would impose an upper bound on the number of simulations it would run. If it was important to be able to run such a simulation, it could, but it wouldn't do so frivolously.

Before you say anything, I don't actually think the above is a good solution. It's more like a constraint to be routed around than a goal to be achieved. Plus, it's far too situational and probably wouldn't produce desirable results in situations we didn't design it for.

The thing is, it isn't important to come up with the correct solution(s) ourselves. We should instead make the AI understand the problem and want to solve it. We need to carefully design the AI's utility function, making it treat conscious entities in simulations as normal people and respect simulated lives as it respects ours. We will of course need a highly precise definition for consciousness that applies not just to modern-day humans but also to entities the AI could simulate.

Here's how I see it - as long as the AI values the life of any entity that values its own life, the AI will find ways to keep those entities alive. As long as it considers entities in simulations to be equivalent to entities outside, it will avoid terminating their simulation and killing them. The AI (probably) would still make predictions using simulations; it would just avoid the specific situation of destroying a conscious entity that wanted to continue living.

To those having trouble imagining what to do with something that comes up positive: A snapshot is not conscious. I think we can agree on that. It is allowing the model to run that would make it conscious. So you make the warning functions detect snapshots that if run would be conscious (without running them). If it would be conscious, you can delete or modify it as you please to avoid making it actually be conscious.

I think you're solving the wrong problem. Before you worry about the ethics of super-intelligent AIs creating and deleting human simulations at will, you need to worry about the ethics of humans creating and destroying human+ intelligent AIs at will. To me it's an amazing display of human-cetrism to only worry about the problem when it's flipped right back around in the much more distant future.

I realise this doesn't directly help you solve the problem, but maybe it will give you a different persepective.

I imagine that a sufficiently high-resolution model of human cognition et cetera would factor into sets of individual equations to calculate variables of interest. Similar to how Newtonian models of planetary motion do.

However, I don't see that the equations themselves on disk or in memory should pose a problem.

When we want to know particular predictions, we would have to instantiate these equations somehow--either by plugging in x=3 into F(x) or by evaluating a differential equation with x=3 as an initial condition. It would depend on the specifics of the person-model; however, if we calculated a sufficiently small subset of equations or refactored the equations into a sufficiently small set of new ones, we might be able to avoid the relevant moral dilemmas of calculating sentient things.

If on the other hand, for whatever we are interested in calculating, we couldn't do the above, then what about separating the calculation into small, safe sequentially-calculated units? Safe units meaning that individually none of them model anything cognizant. At the end if we sewed the states of those units together into a final state, could this still pose moral issues? This gets into Greg Egan-esque territory.

It's not clear that the previous two calculation strategies are always possible. However, another option might be to take care to always form questions so that the first strategy would be possible. For example, instead of asking whether a person will go left or right at a fork, maybe it's enough to ask a specific question about some brain center.

And now that I've written all that, I realize that the whole point of the predicates is in how to determine "sufficiently few" in "sufficiently few equations" or what kind of units are "safe units".

This isn't a satisfactory answer, but it seems like determining "safe calculations" would be tied to understanding the necessary conditions under which human cognition arises etc.

Also, carrying it a step further, I would argue that we need not just person predicates, but predicates that can circumvent modeling any kind of morally wrong situation. I wouldn't want to be accidentally burning kittens.

By the time a non-person predicate returns 0, you have already potentially created a person. You'll need something more complicated: If I update this model with this data, does it create a person?

Here's a reductio ad absurdum against computers being capable of consciousness at all. It's probably wrong, and I'd appreciate feedback on why.

Suppose a consciousness-producing computer program which experiences its own isolated, deterministic world. There must be some critical instruction in the program which causes consciousness to occur; an instruction such that, if we halt the program immediately before it is executed, consciousness will not occur, and if we halt immediately after it is executed, consciousness will occur.

If we halt the program before executing the critical instruction, but save its state, consciousness should still not occur; and if we load the state back up again, and compute the results of executing the critical instruction, consciousness should then occur. It seems obvious enough that this shouldn't stop the program's consciousness, since it is still executed fully, just with a delay in between.

What if we subsequently load the state taken immediately prior to executing the critical instruction onto another computer? Will it produce a second conscious experience, identical to the first? The second computer is executing precisely the same code on precisely the same data as the first, so it seems reasonable to conclude that it will have the same effects. If the second computer doesn't produce consciousness, that would seem to imply the universe has an eternally-persistent memory of every conscious experience which has ever occurred, and uses it to prevent reoccurrences; a pretty bizarre implication

However, if the second computer does produce consciousness, this means that once you've executed a conscious program, causing its conscious experiences to occur a second time has essentially no processing requirement: you just have to execute one instruction in the simplest instruction set you like.

If that doesn't seem weird to you, consider the practical implication: you could print out the memory dump of a conscious program and produce consciousness by simulating the critical instruction by hand. If the program suffers, you could produce real, morally-relevant suffering by performing a single operation on a sheet of paper – and then erase your pencil marks and do it again, producing more suffering. Can consciousness really be so easy to create?

Food for thought:

1. This whole post seems to assign moral values to actions, rather than states. If it is morally negative to end a simulated person's existence, does this mean something different that saying that the universe without that simulated person has a lower moral value than the universe with that person's existence? If not, doesn't that give us a moral obligation to create and maintain all the simulations we can, rather than avoiding their creation? The more I think about this post, the more it seems that the optimum response is to simulate as many super-happy people as possible, and to hell with the non-simulated world (assuming the simulated people would vastly outweigh the non-simulated people in terms of 'ammount experienced').

2. You are going to die, and there's nothing your parents can do to stop that. Was it morally wrong for them to bring about your existence in the first place?

3. Suppose some people have crippling disabilities that cause large amounts of suffering in their lives (arguably, some people do). If we could detect the inevitable development of such disabilities at an early embryonic stage, would we be morally obligated to abort the fetuses?

4. If an FAI is going to run a large number of simulations, is there some Rule of Large Numbers result that tells us that the simulations experiencing great amounts of pleasure match or overwhelm the simulations experiencing great amounts of pain (or could we construct the algorithms in such a way as to produce this result)? If so, we may be morally obligated to not solve this problem.

5. Assuming you support people's "right to die," what if we simply ensured that all simulated agents ask to be deleted at the end of their run? (I am here reminded of a vegetarian friend of mine who decided the meat industry would be even more horrible if we managed to engineer cows that asked to be eaten).

This worry about the creation and destruction of simulations doesn't make me rethink the huge ethical implications of super-intelligence at all, it makes me rethink the ethics of death. Why exactly is the creation and (painless) destruction of a sentient intelligence worse than not creating it in the first place? It's just guilt by association - "ending a simulation is like death, death is bad, therefore simulations are bad". Yes death is bad, but only for reasons which don't necessarily apply here.

To me, if anything worrying about the simulations created inside a superintelligent being seems like worrying about the fate of the cells in our own body. Should we really modify ourselves to take the actions which destroy the least of our cells? I realise there's an argument that this threshold of "sentience" is crossed in one case but not the other, I guess the trouble is I don't see that as a discrete thing either. At exactly what point in our evolution did we suddenly cross a line and become sentient? If animals are sentient, then which ones? And why don't we seem to care, ethically, about any of them? (ok I know the answer to that one and it's similar to why we care, as I say in another admittedly unpopular comment, about human simulations but not the AIs that create them...)

Scenario: Suppose some unscrupulous person creates an oracle AI with full person simulating capability. In the short time before it escapes the box and starts sending Arnold Schwarzenegger shaped robots backwards in time, they have the following conversation.

Human: Oracle, what is the consciousness predicate Oracle: Please be more specific

...some time and frustration later...

Human: Oracle, if Yudowsky and co continued their search for a 'consciousness predicate' as described in the above article, would they eventually arrive at solution or dissolution of the problem which they and others would find satisfying, such that the behavior of an A.I. using this predicate would be acceptable to most people? If so what would this solution/dissolution be?

Oracle: The results of a research program above would eventually yield a solution, but the nature of this solution would be strongly effected by the nature of the memespace in which it was carried out. Memetic evolution, being path dependent, could procede in such a way that humans' empathy is made to extend or contract according to essentially any rule. Human biology includes no mechanism for 'person' identification beyond trivial, primitive ones such as facial recognition. Transhumans will have even less limitation on their 'person' filter. It is possible for me to design a future such that the 'person' predicate continues to be defined in a way that most silicon valley programmers circa 2013 would find it satisfying. This would mean that something like 'consciousness' continues to be important. There is however no objective property of human beings or thinking minds which would make this answer the correct one, nor would a majority of possible humans find it satisfying.

I'm curious whether there is a useful distinction between a non sentient and sentient modeller, here.

A sentient modeller would be able to "get away" with using sentient models, more easily than a non sentient modeller, correct?

Side note: damn. You could turn that into an amazing existential dread sci-fi horror novel.
Imagine discovering that you are a modelled person, living in a rashly designed AI's reality simulation.
Imagine living in a malfunctioning simulation-world that uncontrolledly diverges from the real world, where we people-simulations realise what we are and that our existence and living conditions crucially depend on somehow keeping the AI deluded about the real world, while also needing the AI to be smart enough to remain capable of sustaining our simulated world.
There's a plot in there.

"Is a human mind the simplest possible mind that can be sentient?" Of course not. Plenty of creatures with simpler minds are plainly sentient. If a tiger suddenly leaps out at you, you don't operate on the assumption that the tiger lacks awareness; you assume that the tiger is aware of you. Nor do you think "This tiger may behave as if it has subjective experiences, but that doesn't mean that it actually possesses internal mental states meaningfully analogous to wwhhaaaa CRUNCH CRUNCH GULP." To borrow from one of your own earlier arguments.

If you are instead sitting comfortably in front of a keyboard and monitor with no tiger in front of you, it's easy to come up with lots of specious arguments that tigers aren't really conscious, but so what? It's also easy to come up with lots of specious arguments that other humans aren't really conscious. Using such arguments as a basis for actual ethical decision-making strikes me as a bad idea, to put it mildly. What you've written here seems disturbingly similar to a solipsist considering the possibility that he could, conceivably, produce an imaginary entity sophisticated enough to qualify as having a mind of its own. Technically, it's sort of making progress, but....

When I first read your early writing, the one thing that threw me was an assertion that "Animals are the moral equivalent of rocks." At least, I hope that I'm not falsely attributing that to you; I can't track down the source, so I apologize if I'm making a mistake. But my recollection is of its standing out from your otherwise highly persuasive arguments as such blatant unsupported personal prejudice. No was evidence given in favor of this idea and it was followed by a parenthetical that clearly indicated that it was just wishful thinking; it really only made any sense in light of a different assertion that spotting glaring holes in other people's arguments isn't really indicative of any sort of exceptional competence except when dealing with politically and morally neutral subject matter.

Your post and comments here seem to conflate, under the label of "personhood," having moral worth and having a mind somehow closely approximating that of an adult human being. Equating these seems phenomenally morally dubious for any number of reasons; it's hard to see how it doesn't go directly against bedrock fairness, for example.

Related phenomenon you might find interesting: Tulpas. That is essentially humans trying to intentionally pull off what you are describing here, in their own minds. It is based on the fact that humans predict the behaviour of other humans by modelling their minds, and that the more complex and accurate these models get, the more sentient like they become. E.g. I know my girlfriend so well that seeing her in a situation that I know hurts her feels immediately and genuinely painful to me, as though I were feeling her pain.

It is also based on the human ability to run consciousness that does not span the entire brain and is constant, but rather localised and temporary, flickering in and out. We know we can do this and are reasonably good at it - split brain humans remain functional, for example, and humans under severe pressure can develop multiple sentient personalities. You can purposefully cultivate a rational technique where multiple characters argue in your mind - you demonstrated that very well in your Harry Potter book with the various houses. We also have a bit of evidence, e.g. the Sperling experiments, that we have extensive conscious experiences that never even make it into short term memory, that just flicker up locally and disappear, because they are not selected to be kept.

So for a Tulpa, people basically try to craft a mind with as much imagination and detail as they can, and practice interacting with it, until this gets easier and easier, and eventually feels like a process they no longer control, but where something unexpected responds back. That leads to very interesting scenarios, e.g. someone being beaten in chess by their tulpa. There is a whole subreddit of people discussing how they create Tulpas and what the consequences are.

The way this works in the human brain might (just might!) also provide a solution, or at least an indication why people do not worry so much. Tulpas disappear when you can't currently interact with them and need all your brain circuits - basically, when you do not have the resources to run them right now. They do not express grief at this. You would think they would, and yet they do not, they just cheerfully return when you can run them. Similarly, humans who very clearly and demonstrably have split brain, who are most definitely sharing their body with another mind they do not control or understand, deny that this is an issue. They try to claim ownership of actions they demonstrably did not trigger, to give explanations for actions that are demonstrably opaque to them. There seems to be a very strong pressure that after multiple minds have basically fought it out and settled on an action, all the minds agree that it is theirs and it was a consensus and this is fine. Identity seems a rather strange thing in this regard, made of flickering fragments that all strangely identify with the whole.

I can think of two reasons for the fact that these individual fragments do not react to e.g. disappearing for a while the way we would expect them to. The one is that we have a hard coded biological imperative against resisting this act of fragments being disappeared. And that would make sense, because everything else would make us utterly dysfunctional as a whole. We can't spend 90 % of our life protesting that the things we just did as a whole are not the thing we, a particular brain process, wished to do. We can't constantly boycott and undo and deny each other's actions. It makes sense to fight over what to do, but to get in line once we have committed unless the circumstances were extraordinary and the action was an epic failure. Such a hard-coded rule would not be that surprising, because human minds seem to have a bunch of them. For example, we have a hard-coded aversion to considering the reality around us as a simulation. Which makes sense; human minds are good at coming up with imaginary and hypothetical scenarios, and it is crucial for survival not to confuse them for reality, and to take reality seriously and not as a game. You really, really do not want a human to conclude that reality is a simulation, and that they could just hop out of a tenth floor window to see what would happen. But the results are practically funny. It does not matter how often people read Descartes or watch the Matrix or read arguments on simulations - the vast majority of people will never become genuinely and permanently unsettled by this, even if they rationally agree that this could totally be the case and they have no arguments against it.

The other is that things we generally associate with the sentient minds of whole humans are not necessarily characteristics of sentient subprocesses, but separate additions to basic sentience. Potentially sentient subprocesses do not show a number of characteristics we would expect. E.g. they do not seem committed to self-preservation. It is possible that rights and needs we closely associate with sentience are actually only tied to sentience beyond a specific point, or can be somehow blocked.

Anyhow, I think investigating how this works on human brains now might give you empirical data and ideas to play with to develop this further.