I suspect that it hides more assumptions about the nature of intelligence than we can necessarily make at this time.

At the present moment, we are the only general intelligences around, and we don't seem to have terminal goals as such. As biological bodies, we are constrained by evolutionary processes, and there are many ways in which human behavior actually is reducible to offspring maximization (social status games, etc.). But it doesn't appear to be a 'utility function', so much as a series of strong tendencies in the face of specific stimuli. Using novel approaches like superstimuli, it's just as easy to make an impulse's reproductive utility drop sharply. So we have habits constrained by evolutionary forces, but not algorithmic utility in the paper clipper sense.

There is no such thing as a general intelligence with a 'goal' (as Bostrom defines it). There may be at some point, but it's not real yet. And we do have non-general intelligences with goals, that's an easy weekend coding project. But before we declare that a GI could accept any goal regardless of its strength, we should at least check to make sure that a GI can have a goal at all.

Yes, with some technicalities.

If your resources are limited, you cannot follow certain goals. If your goal is to compute at least 1000 digits of Chaitin's constant, sucks to be computable. I think no agent with a polynomial amount of memory can follow a utility function vulnerable to Pascal's Mugging.

Other than those sorts of technicalities, the thesis seems obvious. Actually, it seems so obvious that I worry that I don't understand the counterarguments.

It seems to me that at least the set of possible goals is correlated with intelligence -- the higher it is, the larger the set. This is easier to see looking down rather than up: humans are more intelligent than, say, cows, and humans can have goals which a cow cannot even conceive of. In the same way a superintelligence is likely to have goals which we cannot fathom.

From certain points of view, we are "simple agents". I have doubts that goals of a superintelligence are predictable by us.

I'm glad it was discussed in the book because I'd never come across it before. So far though I find it one of the least convincing parts of the book, although I am skeptical that I am appropriately evaluating it. Would anyone be able to clarify some things for me?

How generally accepted is the orthogonality thesis? Bostrom presents it as very well accepted.

Danaher's Motivating Belief Objection is similar to an objection I had while reading about the orthogonality thesis. Mine was not as strict though. It just seemed to me that as intelligence increases new beliefs about what should be done are likely to be discovered. I don't see that these beliefs need to be "true beliefs" although as intelligence increases I guess they approach true. I also don't see that they need to be "necessarily motivating", but rather they should have some non-zero probability of being motivating. I mean, to disprove the orthogonality thesis we just have to say that as intelligence increases there's a chance that final goals change right?

Good question. Some of these seem to me like a change in instrumental goals only. If you meant to include such things, then there are very many examples - e.g. if I learn I am out of milk then my instrumental goal of opening the fridge is undermined.

Among many other things, and most relevantly... We don't know what we want. We have to hack ourselves in order to approximate having a utility function. This is fairly predictable from the operation of evolution. Consistency in complex systems is something evolution is very bad at producing.

An artificial agent would most likely be built to know what it wants, and could easily have a utility function.

The consequences of this one difference are profound.

Evolved agents would be in rough equality to other agents. So, their game-theoretic considerations would be different from an artificial agent. The artificial agent could have a design very different from all other agents and also could far surpass other agents. Neither of these is possible in evolution.

In fact, because of the similarity between evolved agents in any given ecosystem, these game-theoretic considerations include not only the possibility of reciprocity or reciprocal altruism, but also the sort of acausal reciprocal morality explored by Drescher and MIRI -- "you are like me, so my niceness is correlated with yours, so I'd better ask nicely."

The form of moral realism I prefer is that the word 'morality' just means something like utilitarianism, and therefore for moral knowledge you need to be able to figure out which things can have preferences/welfare, assess what their preferences/welfare are, and somehow aggregate them. I think there are also plausible versions where a moral system is something like a set of social norms that ensure that everyone gets along together, in which case you need to be in a society of rough equals, figure out which norms promote cooperation or defer to someone else who has figured it out, and be able to apply those norms to various situations.

That would depend on the flavour of moral realism in question..Platonic , Kantuan or whatever.

One way intelligence and goals might be related is that the ontology an agent uses (e.g. whether it thinks of the world it deals with in terms of atoms or agents or objects) as well as the mental systems it has (e.g. whether it has true/false beliefs, or probabilistic beliefs) might change how capable it is...

This is totally right as well. We live inside our ontologies. I think one of the most distinctive, and important, features of acting, successfully aware minds (I won't call them 'intelligences" because of what I am going to say further down, in this message) is this capacity to mint new ontologies as needed, and to do it well, and successfully.

Successfully means the ontological additions are useful, somewhat durable constructs, "cognitively penetrable" to our kind of mind, help us flourish, and give a viable foundation for action that "works" ... as well as not backing us into a local maximum or minimum.... By that I mean this: "successfull" minting of ontological entities enables us to mint additional ones that also "work".

Ontologies create us as much as we create them, and this creative process is I think a key feature of "successful" viable minds.

Indeed, I think this capacity to mint new ontologies and do it well, is largely orthogonal to the other two that Bostrom mentions, i.e. 1) means-end reasoning (what Bostrom might otherwise call intelligence) 2) final or teleological selection of goals from the goal space, and to my way of thinking... 3) minting of ontological entities "successfully" and well.

In fact, in a sense, I would put my third one in position one, ahead of means-end reasoning, if I were to give them a relative dependence. Even though orthogonal -- in that they vary independently -- you have to have the ability to mint ontologies, before means-end reasoning has anything to work on. And in that sense, Katja's suggestion that ontologies can confer more power and growth potential (for more successful sentience to come), is something I think is quite right.

But I think all three are pretty self-evidentally largely orthogonal, with some qualifications that have been mentioned for Bostrom's original two.

One way intelligence and goals might be related is that the ontology an agent uses (e.g. whether it thinks of the world it deals with in terms of atoms or agents or objects) as well as the mental systems it has (e.g. whether it has true/false beliefs, or probabilistic beliefs) might change how capable it is, as well as which values it can comprehend.

I think the remarks about goals being ontologically-associated, are absolutely spot on. Goals, and any “values” distinguishing among the possible future goals in the agent's goal space, are built around that agent's perceived (actually, inhabited is a better word) ontology.

For example, the professional ontology of a wall street financial analyst includes the objects that he or she interacts with (options, stocks, futures, dividends, and the laws and infrastructure associated with the conceptual “deductive closure” of that ontology.)

Clearly, “final” -- teleological and moral – principles involving approach and avoidance judgments … say, involving insider trading (and the negative consequences at a practical level, if not the pure anethicality, of running afoul of the laws and rules of governance for trading those objects) , are only defined within an ontological universe of discourse, which contains those financial objects and the network of laws and valuations that define – and are defined by -- those objects.

Smarter beings, or even ourselves, as our culture evolves, generation after generation becoming more complex, acquire new ontologies and gradually retire others. Identity theft mediated by surreptitious seeding of laptops in Starbucks with keystroke-logging viruses, is “theft” and is unethical. But trivially in 1510 BCE, the ontological stage on which this is optionally played out did not exist, and thus, the ethical valence would have been undefined, even unintelligible.

That is why, if we can solve the friendlieness problem, it will have to be by some means that gives new minds the capacity to develop robust ethical meta-intuition, that can be recruited creatively, on the fly, as these beings encounter new situations that call upon them to make new ethical judgements.

I happen to be a version of meta -ethical realist, like I am something of a mathematical platonist, but in my position, this is crossed also with a type of constructivist metaethics, apparently like that subscribed-to by John Danaher in his blog (after I followed the link and read it.)
At least, his position sounds like it is similar to mine, although constructivist part of my theory is supplemented with a “weak” quasi-platonist thread, that I am trying to derive from some more fundamental meta-ontological principles (work in progress on that.)

1. Perhaps in resolving internal inconsistencies in the value system.
2. An increased intelligence might end up min-maxing. In other words, if the utility function contains two terms in some sort of weighted balance, the agent might find that it can ignore one term to boost another, and that the weighting still produces much higher utility as that first term is sacrificed. This would not strictly be a change in values, but could lead to some results that certainly look like that.

I think in a world with multiple superintelligent agents that have read access to each others' code, I expect that agents 'change their own goals' for the social signalling/bargaining reasons that Bostrom mentions. Although it's unclear whether this would look more like spawning a new successor system with different values and architecture.

I expect a system to face a trade off between self improvement and goal stability.

http://johncarlosbaez.wordpress.com/2013/12/26/logic-probability-and-reflection/

the Chinese way back during their great days as philisophers discovered that it is us humans that input values onto the world at large, objects that it is us who give meaning to something that is meaningless in itself [Kant's thing-in-itself] so that a system's values is there as long as it delivers. luckily humans move on [boredom helps] so that values should never be enshrined: otherwise we may go the way of the Neaderthals. So does a system change with its intelligence? The problem here is that AI's potential intelligence is a redefinition of itself because intelligence [per se] is innate within us: it is a resonance- a mind-field-wave-state [on a quantum level] that self manifests sort of. No AI will ever have that unless symbiosis as interphasing. So the answer to date is: No.

My main problem with this chapter is that Bostrom assumes that an AI has a single utility function, and that it is expressed as a function on possible states of the universe. Theoretically, you could design a program that is given a domain (in the form of, say, a probabilistic program) and outputs a good action within this domain. In principle, you could have asked it to optimize a function over the universe, but if you don't, then it won't. So I think that this program can determine near-optimal behavior across a variety of domains without actually having a utility function (especially not one that is over states of the universe). Of course, this would still be dangerous because anyone could ask it to optimize a function over the universe, but I do think that the assumption that AIs have utility functions over universe states should be more clearly stated and discussed.

It should also be noted that we currently do know how to solve domain-specific optimization problems (such as partially observable Markov decision processes) given enough computation power, but we do not know how to optimize a function over universe states in a way that is agnostic about how to model the universe; this is related to the ontological crisis problem.