Intelligence vs. Wisdom

Whether AI cares about "long-term" effects is not related to complexity of its goals. A paperclip maximizer variant might want to maximize the maximum time that a single paperclip continues to exist, pushing all available resources to get as close to the physical limitations as it can manage.

Well, I'm not sure how far that advances things, but a possible failure mode -- or is it? -- of a Friendly AI occurs to me. In fact, I foresee opinions being divided about whether this would be a failure or a success.

Someone makes an AI, and intends it to be Friendly, but the following happens when it takes off.

It decides to create as many humans as it can, all living excellent lives, far better than what even the most fortunate existing human has. And these will be real lives, no tricks with simulations, no mere tickling of pleasure centres out of a mistaken idea of real utility. It's the paradise we wanted. The only catch is, we won't be in it. None of these people will be descendants or copies of us. We, it decides, just aren't good enough at being the humans we want to be. It's going to build a new race from scratch. We can hang around if we like, it's not going to disassemble us for raw material, but we won't be able to participate in the paradise it will build. We're just not up to it, any more than a chimp can be a human.

It could transform us little by little into fully functional members of the new civilisation, maintaining continuity of identity. However, it assures us, and our proof of Friendliness assures us that we can believe it, the people that we would then be would not credit our present selves as having made any significant contribution to their identity.

Is this a good outcome, or a failure?

Oh. I see why the post is being downvoted as well

I think we're seeing different reasons. I think you're being downvoted because people think you're wrong, and you think you're being downvoted because people think you're right.

I'm being forced to address multiple audiences with different requirements by a nearly universal inclination to look for anything to justify criticism or downvoting -- particularly since I'm rocking the boat or perceived as a newbie.

That hypothesis fails to account for all of the dataset: a lot of top-level posts don't get downvoted to oblivion, even when they're "rocking the boat" more than you are: see this and this.

I don't perceive you as "rocking the boat"; I don't understand enough of what you're trying to say to tell whether I agree or not. I don't think you're more confused or less clear than the average lesswronger, however your top-levels posts on ethics and Friendly AI come off as more confused/confusing than the average top-level post on ethics and Friendly AI, most of which were written by Eliezer.

I don't know if the perceived confusion comes from the fact that your own thinking is confused, that your thinking is clear but your writing is unclear, or that I myself am confused or biased in some way. There is a lot of writing that falls into that category (Foucault and Derrida come to mind), and I don't consider it a worthwhile use of my time to try to figure it out, as there is also a large supply of clear writing available.

Specified goals are exactly what you would expect - a fixed number of goals that are specified in the design (or random creation) of the entity. These are assumed goals: inflexible and, in the best case, having and needing no interpretation other than what is exactly specified.

Maximal goals are maximal in both number and diversity (so please don't conjure the strawman of 100 copies of the same goal ;-).

Still way too vague. How for example to multiple goals interact? If an entity has two specific goals how does it decide to prioritize the two. If it has some priority system how is that not just one slightly more complicated goal? How is "interpretation" relevant to the specific goals? And if a goal is flexible how can it be a goal?

The existence of goals fundamentally make sense across the broad swath of mindspace. I had assumed that this was a given at this site. Classifying regions of mindspace by their relationship to goals and what the implications of that might be should be an obvious question.

Missing the point. This is probably true. But that there's a useful distinction between maximal and specific goals in a broad section of mindspace has not been demonstrated.

Given your last comment, I think you'd be very surprised by how much I read and how carefully I do consider what I post (assuming that this isn't the standard reflexive dig that is very prominent around here to protect old ideas and put down upstart newbies ;-) I normally have a couple of people provide feedback on all my writing before I let it out ion public -- and I have to say that this is the only community that has any problem with lack of clarity on most of my terms (maybe that's because it's always EXPECTING everything to have an odd nonstandard definition?).

I have no way of judging how much you read which isn't at all terribly relevant to my comment. I don't think that there's anything here about expecting non-standard definitions. But definitions in the world around us are often highly imprecise. It is probably true that LW expects more precision in definitions and more careful thinking than most of the web. There are comments I'd make at Reddit or Slashdot that I'd never say here simply because I'd never get away with making such imprecise claims (this in fact worries me that I'm making such statements because it suggests that the level of rationality is not necessarily rubbing off on me. I'll need to think about that.).

As far as I can tell, this is contradictory advice. First, you want me to tell you why you should care about the distinction that I am drawing (which basically requires an overview of where I am going with a huge topic) then you hit me going the other way when I try to give an overview of where I'm going.

OK, I agree it might be somewhat contradictory.

I think there are two problems:

• You're covering a large and abstract scope which lends itself to glossing over important details and prerequesites (such as clarifying what you mean by the various kinds of goals)

• You don't give us many reasons for paying attention to your approach in particular - will it provide us with new insight? Why is that way of dividing "ways to think about goals" better than another? Is this post supposed to be the basic details on which you'll build later, or an overview the details of which you'll fill in later?

On Consciousness: Richard said it better than me; if you just said "an agent's ability to achieve personal goals in a wide range of environments.", I don't think people would translate that in their minds as "consciousness". Contrast your definition with those given on wikipedia.

Why? What is the value of removing a distinction that might just give you a handle on avoiding the most dangerous class of intelligences?

If an "intelligence" focuses on short term goals to the detriment of long term goals, then that would appear to be not very intelligent at all.

But this is arguing over semantics, so I'm going to stop here. I suppose you can define words any way you like, but I question the utility of those definitions. I suppose I'll have to wait to see what you're drawing the distinction for.