Link: why training a.i. isn’t like training your pets

That's not what an artificial neural net actually is. ...

Thank you for the unnecessary tutorial. But actually, what I said is that a super-human AI might be something like a very large neural net. Clearly, a neural net by itself doesn't act autonomously - to get anything approaching 'intelligence' you will need to at least add some feedback loops beyond simple backpropagation.

Again, those other people you mention seem to think that a lot more is going on in an AI system than is actually going on.

More will go on in a future superhuman AI than goes on in any present-day toy AI. Well, yes, those other people I mention do seem to think that. But they are not indulging in any kind of mysticism. Only in the kinds of conceptual extrapolation which took place, for example, in going from simple combinational logic circuitry to the instruction fetch-execute cycle of a von Neumann computer architecture.

... the words intelligent and intelligence in this context and simply refer to a computer capable of doing at least everything a regular person can do.

But we already have things capable of doing everything a regular person can do. We call them regular people. Are we trying to build another person in digital format here, and if so, why? Just because we want to see if can? Or because we have some big plans for it?

I don't think this is a good argument. Just because you cannot define something doesn't mean it's not a real phenomena or that you cannot reason about it at all.

If you have no working definition for what you're trying to discuss, you're more than likely to be barking up the wrong tree about it. We didn't understand fire completely, but we knew that it was hot, you couldn't touch it, and you made it by rubbing dry sticks together really, really fast, or by making a spark with rocks and have it land on dry straw.

Also, where did I say that until I get a definition of intelligence all discussion about the concept is meaningless? I just want to know what criteria an AI must meet to be considered human and match them with what we have so far so I can see how far we might be from those benchmarks. I think it's a perfectly reasonable way to go about this kind of discussion.

So if you develop a new defintion or conjecture and can state why and how you did it, then develop a proof, you've shown thought. Your attempt to suddenly create a new definition or theorem just because you wanted to and were curious rather than just tasked to do it would be initiative.

Hmm, so would a grad student who is thinking about a thesis problem because their advisor said to think about it be showing initiative? Is a professional mathematician showing initiative? They keep thinking about math because that's what gives them positive feedback (e.g. salary, tenure, positive remarks from their peers).

No, you see, a bug is when a computer does something it's not supposed to do and handles its data incorrectly

Is "incorrectly" a normative or descriptive term? .How is it different than "this program didn't do what I expected it to do" other than that you label it a bug when the program deviates more from what you wanted to accomplish? Keep in mind that what a human wants isn't a notion that cleaves reality at the joints.

Birth control is actually another approach to reproduction most of the time, delaying progeny until we feel ready to raise them. Those who don't have children have put their evolutionary desire to provide for themselves above the drive to reproduce and counter that urge with protected sex. So it's not really a bug as it is a solution to some of the problems posed by reproduction.

Ok. So when someone (and I know quite a few people in this category) deliberately uses birth control because they want the pleasure of sex but don't want to ever have kids, is that a bug in your view?

If you have no idea what you want your AI to do, why are you building it in the first place?

I'm not sure how you got from my question to your answer. I'm not talking at all about programmers not having intentions, and I agree with you that in pretty much all cases they do have intentions.

I'll assume that I wasn't clear, rather than that you're willing to ignore what's actually being said in favor of what lets you make a more compelling argument, and will attempt to be clearer.

You keep suggesting that there's no reason to worry about how to constrain the behavior of computer programs, because computer programs can only do what they are told to do.

At the same time, you admit that computer programs sometimes do things their programmers didn't intend for them to do. I might have written a stupid bug that causes the program to delete the contents of my hard drive, for example.

I agree completely that, in doing so, it is merely doing what I told it to do: I'm the one who wrote that stupid bug, it didn't magically come out of nowhere, the program doesn't have any mysterious kind of free will or anything. It's just a program I wrote.

But I don't see why that should be particularly reassuring. The fact remains that the contents of my hard drive are deleted, and I didn't want them to be. That I'm the one who told the program to delete them makes no difference I care about; far more salient to me is that I didn't intend for the program to delete them.

And the more a program is designed to flexibly construct strategies for achieving particular goals in the face of unpredictable environments, the harder it is to predict what it is that I'm actually telling my program to do, regardless of what I intend for it to do.

In other words: "I can't know what I'm telling it to do or be certain what I have told it to do."

Sure, once it deletes the files, I can (in principle) look back over the source code and say "Oh, I see why that happened." But that doesn't get me my files back.

Brains are self-organizing structures built by a combination of cellular signals and environmental cues. All that DNA does is to regulate what proteins the cell will manufacture. Development goes well beyond that.

And yet, remarkably, brains don't "self-organize" in the absence of that regulation.

You're right, of course, that the correct environment is also crucial; DNA won't magically turn into a brain without a very specific environment in which to manifest.

Then again, source code won't magically turn into a running program without a very specific environment either, and quite a lot of the information defining that running program comes from the compiler and the hardware platform rather than the source code... and yet we have no significant difficulty equating a running program with its source code.

(Sure, sometimes bugs turn out to be in the compiler or the hardware, but even halfway competent programmers don't look there except as a matter of last resort. If the running program is doing something I didn't intend, it's most likely that the source code includes an instruction I didn't intend to give.)