Ontological Crisis in Humans

To fully confront the ontological crisis that we face, we would have to upgrade our world model to be based on actual physics, and simultaneously translate our utility functions so that their domain is the set of possible states of the new model. We currently have little idea how to accomplish this, and instead what we do in practice is, as far as I can tell, keep our ontologies intact and utility functions unchanged, but just add some new heuristics that in certain limited circumstances call out to new physics formulas to better update/extrapolate our models. This is actually rather clever, because it lets us make use of updated understandings of physics without ever having to, for instance, decide exactly what patterns of particle movements constitute pain or pleasure, or what patterns constitute oneself.

This seems true and important.

should we be very sanguine that for humans everything must "add up to moral normality"?

No. People do go out of their minds on nihilism now and then.

And I've already seen two LWers who have discovered such compassion for the suffering of animals that they want to exterminate them all except for a few coddled pets, one of whom isn't sure that humans should exist at all.

Less dramatically, any number of Buddhists have persuaded themselves that they don't exist, although I'm not sure how many just believe that they believe that.

"If nothing really exists and it's all just emptiness and even the emptiness is empty of existence, then how could I have killed all those nonexistent people with my nonexistent hands?

"Tell it to the nonexistent prison walls, buddy."

An ontology consisting of Gods, Self, Other People, and Dumb Matter just isn't very different from one consisting of Self, Other People, and Dumb Matter

Benja just posted a neat proof of why, if your preferences don't satisfy the axiom of continuity in von Neumann-Morgenstern utility theory, your rational behavior would be almost everywhere identical to the behavior of someone whose continuity-satisfying-preferences simply ignore the "lower priority" aspects of yours. E.g. if you prefer "X torture plus N dust specks" over "Y torture plus M dust specks" for any X < Y but also for any X==Y, N < M, then you might as well ignore the existence of dust specks because in practical questions there's always going to be some epsilon of probability between X and Y.

But now what if instead of "torture" and "dust specks" we have a lexical preference ordering on "displeasure of God(s)" and "everything else bad", and then we remove the former from the picture? Suddenly the parts of probability space that you were previously ignoring (except indirectly insofar as you tried to reflect the preferences of God(s) regarding everything else) are now the only thing you should care about!

From other frameworks the problem looks even worse: If your previous answer to the is-ought problem was to derive every ethical proposition from the single "ought" axiom "We ought to do what God wants regarding X", and now you're down to zero "ought" axioms, that makes a huge difference, no?

To fully confront the ontological crisis that we face

What ontological crisis is that? The rest of the article is written in general terms, but this phrase suggests you have a specific one in mind, but without actually being specific. Is there some ontological crisis that you are facing, that moved this article?

Personally, learning that apples are made of atoms doesn't give me any difficulty in eating them.

To fully confront the ontological crisis that we face, we would have to upgrade our world model to be based on actual physics, and simultaneously translate our utility functions so that their domain is the set of possible states of the new model. We currently have little idea how to accomplish this, and instead what we do in practice is, as far as I can tell, keep our ontologies intact and utility functions unchanged, but just add some new heuristics that in certain limited circumstances call out to new physics formulas to better update/extrapolate our models.

Here's a hypothesis (warning for armchair evpsych)...

Define "preferences" to refer broadly to a set that includes an individual's preferences, values, goals, and morals. During an individual's childhood and early adulthood, their ontology and preferences co-evolve. Evolution seeks to maximize fitness, so the preference acquisition process is biased in such a way that the preferences we pick up maximize our ability to survive and have surviving offspring. For example, if hunting is considered high-status in our tribe and we display a talent for hunting, we'll probably pick up a preference for being a hunter. Our circle of altruism gets calibrated to cover those who are considered part of our tribe, and so on. This has the ordinary caveat that the preference acquisition process should be expected to be optimized for the EEA, not the modern world.

There is an exploration/exploitation tradeoff here, and the environment in the EEA probably didn't change that radically, so as time goes by this process slows down and major changes to our preferences become less and less likely. Because our preferences were acquired via a process aiming to maximize our fit to our particular environment, they are intimately tied together with our ontology. As our neurology shifts closer towards the exploitation phase and our preferences become less amenable to change, we become more likely to add new heuristics to our utility functions rather than to properly revise them when our ontology changes.

This is part of the reason for generational conflict, because as children are raised in a different environment and taught a different view of their world, their preferences become grounded in a new kind of ontology that's different from the one the preferences of their parents came from. It also suggests that the preferences of any humans still alive today might to some extent be simply impossible to reconcile with those of a sufficiently advanced future - though the children (or robots) born and raised within that future will have no such problem. Which, of course, is just as it has always been.

This seems to be one of the biggest problems for FAI... keeping an utility function constant in a self-modifying agent is hard enough, but keeping it the same over a different domain... well, that's real hard.

Actually, there might be three outcomes:

• we can extrapolate so that it all adds up to normality in when mapped back to the original ontology (unlikely)
• we can extrapolate in various ways that is consistent with the original onthology & original human brain design, but not unique (which doesn't seem to be a "fail" scenario... we just might need new values in addition to the old ones)
• our current utility function turns out to be outright contradictory (I can imagine an AI, after a few turns of self-modification, looking at the instructions that turn out to be "the goal of your existence is making X blue" and "avoid making X blue at all costs" after converting them to a better ontology...)

CEV, for example, seems to assume 1 (or 2?). Do we have any indication that it's not 3 that is the case?

To fully confront the ontological crisis that we face, we would have to upgrade our world model to be based on actual physics, and simultaneously translate our utility functions so that their domain is the set of possible states of the new model.

Once upon a time, developmental psychology claimed that human babies learned object permanence as they aged. I don't know if that's still the dominant opinion, but it seems at least possible to me, a way that the world could be, if not the way it is. What would that mean, for a baby to go from not having a sense of objects persisting in locations to having it?

First, let's unpack what an object might be. If there's a region of silver color in a baby's visual field, and rather than the region breaking apart in different directions over time, if it stays together, if the blob is roughly invariant to translations, that's the beginning of an object concept. A lump that stays together. Then the baby notices that the variances are also predictable, that the flowery texture bits are sometimes showing, and the curved bits are sometimes showing, but not usually together, or neither shows when the silver blob looks especially small, relative to its position. From coactivation of features, and maybe higher order statistics, the baby eventually learns a representation of spoon which predicts which features will be visible as a function of some rotation parameters. That's what a primitive spoon object means in my mind. There are of course other things to incorporate into the spoon model, like force dynamics (its weight, malleability, permeability and sharpness, et cetera), lighting invariances, and haptic textural information.

Permanence would be something like being able to make predictions of the object's position even when the object is occluded (having a representation of the face behind the hands, which doesn't compress the present visual scene, but compresses visual scenes across time). Old experiments showed that babies' object tracking gazes for occluded objects increased with age, which was supposed to be support for a theory of learned object permanence.

Now, if that's at all how human macroscopic object perception starts out, I think it's fair to call that "ruled by a hodgepodge of heuristics and prediction algorithms". However, it seems psychologically implausible that babies undergo a utility function change throughout this process, in the way you seem to mean. If we think of a world model as supplying predictions (note, this is something of an abuse of terminology, since it probably includes both structured, quickly updateable predictions from "model-based" brain regions like hippocampal place cells, as well as slowly revised model-free habit-type predictions) - if we think of a world model as supplying predictions and utility functions as supplying valuations over predicted worlds, then the domain of the utility function is still some kind of predicted state, before and after learning object permanence. Intuitively, worlds without object permanence are a very different hypothesis space, and thus a very different space of appreciable hypothetical realities, than "our" models which "divide cleanly into these 3 parts", but I think both types fall into a broader category which reward circuitry functions can take as argument. Indeed, if developmental psychology was right about learning object permanence, humans probably spend a few weeks with world models which have graded persistence.

Do we have any examples of humans successfully navigating an ontological crisis?

How do you define "successfully"?

For example, all the disagreement over "free will" seems to be because many humans have a sense of morality which presumes some sense of an extraphysical free will. Confronted with evidence that we are physical systems, some people resort to claiming that we aren't actually physical systems, others modify their conception of free will to be compatible with us being physical systems, and some declare that since we are physical systems, we have no free will and the concept was confused from the start. Those are three different outcomes of running into an ontological crisis - which one counts as being successful? (Of course, not everyone thinks that morality requires an extraphysical free will in the first place.)

Similarly, there is currently controversy over animal rights, which for some is influenced by the ontological question of whether animals suffer in a manner similar to us; historically, similar ontological considerations influenced the question of whether to treat black people or women the same as white men. The theory of evolution probably caused some sort of ontological crises, and so on. In each case, there are various ways of dealing with the issue, but it's not clear which one of them counts as a "success". (Aside for the fact that we'd like to call the reactions which align with our way of thought as successes, of course. Needless to say, this comment does not endorse the ill-treatment of black people, women, nor for that matter animals.) Society just generally splits into different opposing camps, they debate each other for a while, and then generally some of the camps just happens to win out for whatever sociological reason.

When your preferences operate over high-level things in your map, the problem is not that they don't talk about the real world. Because there is a specific way in which the map gets determined by the world, in a certain sense they already do talk about the real world, you just didn't originally know how to interpret them in this way. You can compose the process that takes the world and produces your map with the process that takes your map and the high-level things in it and produces a value judgement, obtaining a process that takes the world and produces a value judgment. So it's not a problem of judgments being defined for only the map, it's an issue arising from gaining the ability to examine new interpretations of the same implementation of preferences, that are closer to being in terms of the world than the original intuitive perception that only described them in terms of vague high level concepts.

The problem is that when you examine your preference in terms of judging the world, you notice that you don't like some properties of what it does, you want to modify it, but you are not sure how. There are more possibilities when you are working with a more detailed model. But gaining access to this new information doesn't in any way invalidate the original judgment procedure, except to the extent that you are now able to make improvements. The ability to notice flaws and to make improvements means in particular that your preference includes judgments that refer to originally unknown details, as is normal for sufficiently complicated mathematical definitions.

I recently realized that a couple of problems that I've been thinking over (the nature of selfishness and the nature of pain/pleasure/suffering/happiness) can be considered instances of ontological crises in humans (although I'm not so sure we necessarily have the cognitive algorithms to solve them).

Uploading also seems like it's going to spawn a whole lot of ontological crises in humans. Suppose you value equality and want to weigh everyone's views equally, either via an ordinary democratic process or something more exotic like CEV. So what do you do when somebody can make a huge number of copies of themselves, does each of get them equal weight? How do you weigh the values of two minds that have merged together? Or even without uploading, if you happen to think that "alters" in dissociative identity disorder were genuine entities, should their desires be counted separately?

A lot of modern-day values seem to be premised on the assumption that human minds are indivisible, uncopyable and unmergeable - if you discard those assumptions, translating our values into something meaningful becomes really hard. To say nothing about what "death" or "personal identity" means when minds can be copied and deleted...

Given that we do not actually have at hand a solution to ontological crises in general or to the specific crisis that we face, what's wrong with saying that the solution set may just be null?

My answer is that this seems like a severely underdetermned problem, not an overdetermined one. If we start from the requirement that the change-gathering robot still gathers change in the ancestral environment, we're at least starting with lots of indeterminacy, and I don't think there's some tipping point as we add desiderata. Unless we take the tragic step of requiring our translated morality to be the only possible translation :D

Do we have any examples of humans successfully navigating an ontological crisis?

I bet a number of LW regulars have done so. I commented earlier (and rather too often) of how my ontology shifted from realism to instrumentalism after the standard definitions of terms "reality" and "existence" proved unsatisfactory for me. Certainly being exposed to the stuff discussed on this forum ought to have had a profound impact at least on some others, as well.

A formal look at ways of thinking about agency

Actually, dealing with a component of your ontology not being real seems like a far harder problem than the problem of such a component not being fundamental.

According to the Great Reductionist Thesis everything real can be reduced to a mix of physical reference and logical reference. In which case, if every component of your ontology is real, you can obtain a formulation of your utility function in terms of fundamental things.

The case where some components of your ontology can't be reduced because they're not real and where your utility function refer explicitly to such entity seem considerably harder, but that is exactly the problem that someone who realize God doesn't actually exist is confronted with, and we do manage that kind of ontology crisis.

So are you saying that the GRT is wrong or that none of the things that we value are actually real or that we can't program a computer to perform reduction (which seems absurd given that we have managed to perform some reductions already) or what? Because I don't see what you're trying to get at here.

A ontological crisis would indicate a serious problem. I don't see one.

There may be weird boundary cases where our base ontologies have a problem, but I don't find applying morality a daily struggle.

Do you have concrete examples of serious problems with applying morality in the real world today?

By the way, I think nihilism often gets short changed around here. Given that we do not actually have at hand a solution to ontological crises in general or to the specific crisis that we face, what's wrong with saying that the solution set may just be null? Given that evolution doesn't constitute a particularly benevolent and farsighted designer, perhaps we may not be able to do much better than that poor spare-change collecting robot? If Eliezer is worried that actual AIs facing actual ontological crises could do worse than just crash, should we be very sanguine that for humans everything must "add up to moral normality"?

I really like the robot metaphor, and I fully agree with the kind of nihilism you describe here. Let me note, though, that nihilism is a technically precise but potentially misleading name for this world view. I am an old-fashioned secular humanist when it comes to 2012 humans. I am a moral nihilist only when I have to consider the plethora of paradoxes that come with the crazy singularity stuff we like to discuss here (most significantly, substrate-independence). Carbon-based 2012 humans already face some uncomfortable edge cases (e.g. euthanasia, abortion, animal rights), but with some introspection and bargaining we can and should agree on some ground rules. I am a big fan of such ground rules, that's why I call myself an old-fashioned humanist. On the other hand, I think our morality simply does not survive the collision with your "ontological crisis". After the ontological crisis forces itself on us, it is a brand new world, and it becomes meaningless to ask what we ought to do in this new world. I am aware that this is an aesthetically deeply unsatisfying philosophical position, so I wouldn't accept it if I had some more promising alternatives available.