Artificial Addition

Well, shooting randomly at a distant target is more likely to produce a bulls-eye than not shooting at all, even though you're almost certainly going to miss (and probably shoot yourself in the foot while you're at it). It's probably better to try to find a way to take off that blindfold. As you suggest, we don't yet understand intelligence, so there's no way we're going to make an intelligent machine without either significantly improving our understanding or winning the proverbial lottery.

"Programming is the art of figuring out what you want so precisely that even a machine can do it." - Some guy who isn't famous

Well shooting randomly is perhaps a bad idea, but I think the best we can do is shoot systematically, which is hardly better (takes exponentially many bullets). So you either have to be lucky, or hope the target isn't very far, so you don't need to a wide cone to take pot shots at, or hope P=NP.

@Doug & Gray: AGI is a William Tell target. A near miss could be very unfortunate. We can't responsibly take a proper shot till we have an appropriate level of understanding and confidence of accuracy.

Eliezer,

Did you include your own answer to the question of why AI hasn't arrived yet in the list? :-)

This is a nice post. Another way of stating the moral might be: "If you want to understand something, you have to stare your confusion right in the face; don't look away for a second."

So, what is confusing about intelligence? That question is problematic: a better one might be "what isn't confusing about intelligence?"

Here's one thing I've pondered at some length. The VC theory states that in order to generalize well a learning machine must implement some form of capacity control or regularization, which roughly means that the model class it uses must have limited complexity (VC dimension). This is just Occam's razor.

But the brain has on the order of 10^12 synapses, and so it must be enormously complex. How can the brain generalize, if it has so many parameters? Are the vast majority of synaptic weights actually not learned, but rather preset somehow? Or, is regularization implemented in some other way, perhaps by applying random changes to the value of the weights (this would seem biochemically plausible)?

Also, the brain has a very high metabolic cost, so all those neurons must be doing something valuable.

That's not how William Tell managed it. He had to practice aiming at less-dangerous targets until he became an expert, and only then did he attempt to shoot the apple.

It is not clear to me that it is desirable to prejudge what an artificial intelligence should desire or conclude, or even possible to purposefully put real constraints on it in the first place. We should simply create the god, then acknowledge the truth: that we aren't capable of evaluating the thinking of gods.

Adding to DanBurFoot, is there a link you want to point to that shows your real, tangible results for AI, based on your superior methodology?

I think that one of the difficulties inherent in monotonous logics comes from the fact that real numbers are not very good a representing things continuous. In order to define a single point, an infinite number of digits are needed and thus an infinite amount of information. Often mathematicians ignore this. To them, using the symbol 2 to represent a continuous quantity is the same as the symbol 2.000... which seem to make for all kinds of weird paradoxes caused by the use of, often implied, infinite digits. For example, logicians seem to be unable to make a distinction between 1.999... and 2 (where they take two as meaning 2.000...) thus two different definable real numbers represent the same point.

When using real numbers that represent continuous value, I often wonder if we shouldn't always be using the number of digits to represent some kind of uncertainty. Using significant digits, is one of the first thing students learn in university, they are crucial for experiments of the real world, they allow us to quantify the uncertainty in the digits we write down. Yet mathematicians and logicians seem to ignore them in favor of paradoxical infinities. I wonder if by using uncertainty in this way, we might not do away with Godel's theorem and define arithmetics within a certain amount of relative uncertainty inherent to our measuring instruments and reasoning machinery.

For what it's worth, Benoit Essiambre, the things you have just said are nonsense. The reason logicians seem to be unable to make a distinction between 1.999... and 2 is that there is no distinction. They are not two different definable real numbers, they are the same definable real number.

Me: AGI is a William Tell target. A near miss could be very unfortunate. We can't responsibly take a proper shot till we have an appropriate level of understanding and confidence of accuracy.

Caledonian: That's not how William Tell managed it. He had to practice aiming at less-dangerous targets until he became an expert, and only then did he attempt to shoot the apple.

Yes, by "take a proper shot" I meant shooting at the proper target with proper shots. And yes, practice on less-dangerous targets is necessary, but it's not sufficient.

It is not clear to me that it is desirable to prejudge what an artificial intelligence should desire or conclude, or even possible to purposefully put real constraints on it in the first place. We should simply create the god, then acknowledge the truth: that we aren't capable of evaluating the thinking of gods.

I agree we can't accurately evaluate superintelligent thoughts, but that doesn't mean we can't or shouldn't try to affect what it thinks or what it's goals are.

I couldn't do this argument justice. I encourage interested readers to read Eliezer's paper on coherent extrapolated volition.

Nominull, I kind of agree that they are the same at the limit of infinite digits (assuming by 2 you mean 2.000...). It just seems to me that working with numbers that are subject to this kind of limit is the wrong approach to mathematics if we want maths to be tied to something real in this universe, especially when the limit is implicit and hidden in the notation.

No, by 2 I mean 1.999...

A_A

Benoit,

1,9999.... can only be the same (or equal) to 2 in some kind of imaginary world. The number 1,999... where there is an infinity of 9's does not "exist" in so far as it cannot be "represented" in a finite amount of space or time. The only way out is to "represent" infinity by (...). So you represent something infinite by something finite, thus avoiding a serious problem. But then stating that 1,999... is equal to 2 becomes a tautology.

Of course mathematicians now are used to deal with infinities. They can manipulate them any which way they want. But in the end, infinity has no equivalent in the "real" world. It is a useful abstraction.

So back to arithmetic. We can only "count" because our physical world is a quantum world. We have units because the basic elements are units, like elementary particles. If the real world were a continuum, there would be no arithmetic. Furthermore, arithmetic is a feature of the macroscopic world. When you look closer, it breaks down. In quantum physics, 1+1 is not always equal to two. You can have many particles in the same quantum state that are indistinguishable. How do you count sheep when you can't distinguish them?

I don't see anything "obvious" in stating that 1+1=2. It's only a convention. "1" is a symbol. "2" is another symbol. Trace it back to the "real" world, and you find that to have one object plus another of the same object (but distinct) requires subtle physical conditions.

On another note, arithmetic is a recent invention for humanity. Early people couldn't count to more than about 5, if not 3. Our brain is not that good at counting. That's why we learn arithmetic tables by heart, and count with our fingers. We have not "evolved" as arithmeticians.

I agree that infinity is an abstraction. What I'm trying to say is that this concept is often abused when it is taken as implicit in real numbers.

"We can only "count" because our physical world is a quantum world. We have units because the basic elements are units, like elementary particles. If the real world were a continuum, there would be no arithmetic."

I don't see it that way. In Euclid's book, variables are assigned to segment lengths and other geometries that tie algebra to geometric interpretations. IMO, when mathematics stray away from something that can be interpreted physically it leads to confusion and errors.

What I'd like to see is a definition of real numbers that is closer to reality and that allows us to encode our knowledge of reality more efficiently. A definition that does not allow abstract limits and infinite precision. Using the "significant digits" interpretation seems to be a step in the right direction to me as all of our measurement and knowledge is subject to some kind of error bar.

We could for example, define a set of real numbers such that we always use as many digit needed so that the quantization error from the limited number of digits is under a hundred times smaller than the error in the value we are measuring. This way, the error caused by the use of this real number system would always explain less than a 1% of the variance of our measurements based on it.

This also seem to require that we distinguish mathematics on natural numbers which represent countable whole items, and mathematics that represent continuous scales which would be best represented by the real numbers system with the limited significant digits.

Now this is just an idea, I'm just an amateur mathematician but I think it could resolve a lot of issues and paradoxes in mathematics.

1.9999... = 2 is not an "issue" or a "paradox" in mathematics.

If you use a limited number of digits in your calculations, then your quantization errors can accumulate. (And suppose the quantity you are measuring is the difference of two much larger numbers.)

Of course it's possible that there's nothing in the real world that corresponds exactly to our so-called "real numbers". But until we actually know what smaller-scale structure it is that we're approximating, it would be crazy to pick some arbitrary "lower-resolution" system and hope it matches the world better. That's doing for "finiteness" what Eliezer has somewhere or other complained about people doing for "complexity".

"...mathematics that represent continuous scales which would be best represented by the real numbers system with the limited significant digits."

If you limit the number of significant digits, your mathematics are discrete, not continuous. I'm guessing the concept you're really after is the idea of computable numbers. The set of computable numbers is a dense countable subset of the reals.

"Pocket calculators work by storing a giant lookup table of arithmetical facts".

you can't create a lookup table without proper math.

With the graphical-network insight in hand, you can give a mathematical explanation of exactly why first-order logic has the wrong properties for the job, and express the correct solution in a compact way that captures all the common-sense details in one elegant swoop.

Consider the following example, from Menzies's "Causal Models, Token Causation, and Processes"[*]:

An assassin puts poison in the king's coffee. The bodyguard responds by pouring an antidote in the king's coffee. If the bodyguard had not put the antidote in the coffee, the king would have died. On the other hand, the antidote is fatal when taken by itself and if the poison had not been poured in first, it would have killed the king. The poison and the antidote are both lethal when taken singly but neutralize each other when taken together. In fact, the king drinks the coffee and survives.

We can model this situation with the following structural equation system:

A = true G = A S = (A and G) or (not-A and not-G)

where A is a boolean variable denoting whether the Assassin put poison in the coffee or not, G is a boolean variable denoting whether the Guard put the antidote in the coffee or not, and S is a boolean variable denoting whether the king Survives or not.

According to Pearl and Halpern's definition of actual causation, the assassin putting poison in the coffee causes the king to survive, since changing the assassin's action changes the king's survival when we hold the guard's action fixed. This is clearly an incorrect account of causation.

IMO, graphical models and related techniques represent the biggest advance in thinking about causality since Lewis's work on counterfactuals (though James Heckman disagrees, which should make us a bit more circumspect). But they aren't the end of the line, even if we restrict our attention to manipulationist accounts of causality.

[*] The paper is found here. As an aside, I do not agree with Menzies's proposed resolution.

"But until we actually know what smaller-scale structure".

From http://en.wikipedia.org/wiki/Planck_Length: "Combined, these two theories imply that it is impossible to measure position to a precision greater than the Planck length, or duration to a precision greater than the time a photon traveling at c would take to travel a Planck length"

Therefore, one could in fact say that all time- and distance- derived measurements can in fact be truncated to a fixed number of decimal places without losing any real precision, by using precisions based on the Planck Length. There's no point in having precision smaller than the limits in the quote above, as anything smaller is unobservable in our current understanding of physics.

That length is approximately 1.6 x 10^-35, and the corresponding time duration is approximately 5.33702552 x 10^-44 seconds.

"When the basic problem is your ignorance, clever strategies for bypassing your ignorance lead to shooting yourself in the foot."

I like this lesson. It rings true to me, but the problem of ego is not one to be overlooked. People like feeling smart and having the status of being a "learned" individual. It takes a lot of courage to profess ignorance in today's academic climate. We are taught that we have such sophisticated techniques to solve really hard problems. There are armies of scientists and engineers working to advance our society every minute. But who stops and asks "if these guys (and gals) are so smart, why is it that such fundamental ignorance still exists in so many fields"? Yes, there are our current theories, but how many of them are truly impressive? How many logically follow from the context vs. how many took a truly creative breakthrough? The myth of reductionism promises steady progress, but it is the individual who gets inspired. It boils down to humility. Man is too arrogant to admit that he is still clueless on many fundamental problems. How could that possibly be true if we are all so smart in our modern age? Who amongst you will admit when something that seems very sophisticated actually makes no sense? You'll probably just feel stupid for not understanding, but the problem is not necessarily with you. Dogma creeps into any organization of people, and science is no different. We assume our level of understanding in certain subjects applies equally to all. Until people have the courage to question very fundamental assumptions on how we approach new problems, we will not progress, or worse, we will find much work has been done on a faulty foundation. Figuring out the right question to ask is the most important hurdle of all. But who has time when we are judged not by the quality of our thought but by the quantity? Some very important minds only produced a handful of papers, but they were worth reading...

anonymous--I'd like to second that motion

I read a book on the philosophy of set theory -- and I get lost right at the point where classical infinite thought was replaced by modern infinite thought. IIRC the problem was paradoxes based on infinite recursion (Zeno et. all) and finding mathematical foundations to satisfy calculus limits. Then something about Cantor, cardinality and some hand wavy 'infinite sets are real!'.

1.999... is just an infinite set summation of finite numbers 1 + 0.9 + 0.09 + ...

Now, how an infinite process on an infinite set can equal an integer is a problem I still grapple with. Classical theory said that this was nonsense since one would never finish the summation (if one were to begin). I tend to agree and I suppose one could say I see infinity as a verb and not a noun.

I suggest anyone who believes 1.999... === 2 really looks into what that means. The root of the argument isn't "What is the number between 1.999... and 2?" but rather "Can we say that 1.999... is a sensible theoretical concept?"

The issue with AI has nothing to do with ignorance or arrogance. The basic problem is that intelligence can't be meaningfully defined or meaningfully quantified. Documented fact: Richard Feynman had a measured I.Q. of 120. Documented fact: Marilyn Vos Savant had a measured I.Q. of 180 or 200, depending on which test you place more faith in. Documented fact: Feynman made a huge breakthrough in physics, Vos Savant has accomplished nothing worth mentioning in her life. I.Q. measurements fail to measure intelligence in any meaningful way.

Here's another fact for you. Louis Terman collected a group of so-called "geniuses" sieved by their high I.Q. scores. Two future nobel prize winners, Shockley and Alvarez, got tested but discarded by Terman's I.Q. tests and weren't part of the group.

Question: What does this tell you about current methods for measuring intelligence?

There is no evidence that people can meaningfully define or objectively measure intelligence. Rule of thumb: if you can't define it and you can't measure it objectively, you can't do science about it.

[Remainder of gigantic comment truncated by editor.]

Question: What does this tell you about current methods for measuring intelligence?

Better question: why do you insist that those examples are of failures to acknowledge intelligence when you also insist that we are unable to meaningfully define intelligence?

mclaren, your comment is way too long. I have truncated it and emailed you the full version. Feel free to post the comment to your blog, then post a link to the blog here.

Anonymous (re Planck scales etc.), sure you can truncate your representations of lengths at the Planck length, and likewise for your representations of times, but this doesn't simplify your number system unless you have acceptable ways of truncating all the other numbers you need to use. And, at present, we don't. Sure, maybe really the universe is best considered as some sort of discrete network with some funky structure on it, but that doesn't give us any way of simplifying (or making more appropriate) our mathematics until we know just what sort of discrete network with what funky structure. (And I think every sketch-of-a-theory we currently have along those lines still uses continuously varying quantities as quantum "amplitudes", too.)

James (re mathematics and infinite sets and suchlike), it seems unfair to criticize something as being handwavy when you demonstrably don't remember it clearly; how do you know that the vagueness is in the thing itself rather than your recollection? There is a perfectly clear and simple definition of what a sum like 1 + 9/10 + 9/100 + ... means (which, btw, is surely enough to call it "a sensible theoretical concept"), and what that particular one means is 2. If you have a different definition, or a different way of doing mathematics, that you like better, then feel free to adopt it and do mathematics that way; if you end up with a theory at least as coherent, useful and elegant as the usual one then perhaps it'll catch on.

Anonymous (re humility, reductionism, etc.): I think your comment consisted mostly of applause lights. Science is demonstrably pretty good at questioning fundamental assumptions (consider, say, heliocentricity, relativity, quantum mechanics, continental drift); what evidence have you that more effort should go into questioning them than currently does? (Clearly some should, and does. Clearly much effort spent that way is wasted, and produces pseudoscience or merely frustration. The question is how to apportion the effort.)

Thanks g for the tip about computable numbers, that's pretty much what I had in mind. I didn't quite get from the wikipedia article if these numbers could or could not replace the reals for all of useful mathematics but it's interesting indeed.

James, I share your feelings of uneasiness about infinite digits, as you said, the problem is not that these numbers will not represent the same points at the limit but that they shouldn't be taken to the limit so readily as this doesn't seem to add anything to mathematics but confusion.

@James:

If I recall my Newton correctly, the only way to take this "sum of an infinite series" business consistently is to interpret it as shorthand for the limit of an infinite series. (Cf. Newton's Principia Mathematica, Lemma 2. The infinitesimally wide parallelograms are dubitably real, but the area under the curve between the sets of parallelograms is clearly a real, definite area.)

@Benoit:

Why shouldn't we take 1.9999... as just another, needlessly complicated (if there's no justifying context) way of writing "2"? Just as I could conceivably count "1, 2, 3, 4, d(5x)/dx, 6, 7" if I were a crazy person.

Benquo, I see two possible reasons:

1) '2' leads to confusion as to whether we are representing a real or a natural number. That is, whether we are counting discrete items or we are representing a value on a continuum. If we are counting items then '2' is correct.

2) If it is clear that we are representing numbers on a continuum, I could see the number of significant digits used as an indication of the amount of uncertainty in the value. For any real problem there is always uncertainty caused by A) the measuring instrument and B) the representation system itself such as the computable numbers which are limited by a finite amount of digits (although we get to choose the uncertainty here as we choose the number of digits). This is one of the reason the infinite limits don't seem useful to me. They don't correspond to reality. The implicit limits seems to lead to sloppiness in dealing with uncertainty in number representation.

For example I find ambiguity in writing 1/3 = 0.333... However, 1.000/3.000 = 0.333 or even 1.000.../3.000...=0.333... make more sense to me as it is clear where there is uncertainty or where we are taking infinite limits.

Benoit Essiambre,

Right now Wikipedia's article is claiming that calculus cannot be done with computable numbers, but a Google search turned up a paper from 1968 which claims that differentiation and integration can be performed on functions in the field of computable numbers. I'll go and fix Wikipedia, I suppose.

eh? maths is well defined and well structured etc. intuitive thinking isn't and so can't be encoded into a computer program very easily, that was the whole point of minsky's paper! are you a bit thick or something??

Benoit Essiambre,

You say:

"1) '2' leads to confusion as to whether we are representing a real or a natural number. That is, whether we are counting discrete items or we are representing a value on a continuum."

If I recall correctly, this "confusion" is what allowed modern, atomic chemistry. Chemical substances -- measured as continuous quantities -- seem to combine in simple natural-number ratios. This was the primary evidence for the existence of atoms.

What is the practical negative consequence of the confusion you're trying to avoid?

You also say:

"2) If it is clear that we are representing numbers on a continuum, I could see the number of significant digits used as an indication of the amount of uncertainty in the value. For any real problem there is always uncertainty caused by A) the measuring instrument and B) the representation system itself such as the computable numbers which are limited by a finite amount of digits (although we get to choose the uncertainty here as we choose the number of digits). This is one of the reason the infinite limits don't seem useful to me. They don't correspond to reality. The implicit limits seems to lead to sloppiness in dealing with uncertainty in number representation."

But wouldn't good sig-fig practice round 1.999... up to something like 2.00 anyway?

Benoit, it was "Cyan" and not me who mentioned computable numbers.

Benoit, you assert that our use of real numbers leads to confusion and paradox. Please point to that confusion and paradox.

Also, how would your proposed number system represent pi and e? Or do you think we don't need pi and e?

Well, for example, the fact that two different real represent the same point. 2.00... 1.99... , the fact that they are not computable in a finite amount of time. pi and e are quite representable within a computable number system otherwise we couldn't reliably use pi and e on computers!

Benoit, those are two different ways of writing the same real, just like 0.333... and 1/3 (or 1.0/3.0, if you insist) are the same number. That's not a paradox. 2 is a computable number, and thus so are 2.000... and 1.999..., even though you can't write down those ways of expressing them in a finite amount of time. See the definition of a computable number if you're confused.

1.999... = 2.000... = 2. Period.

Benoit,

In the decimal numeral system, every number with a terminating decimal representation also has a non-terminating one that ends with recurring nines. Hence, 1.999... = 2, 0.74999... = 0.75, 0.986232999... = 0.986233, etc. This isn't a paradox, and it has nothing to do with the precision with which we measure actual real things. This sort of recurring representation happens in any positional numeral system.

You seem very confused as to the distinction between what numbers are and how we can represent them. All I can say is, these matters have been well thought out, and you'd profit by reading as much as you can on the subject and by trying to avoid getting too caught up in your preconceptions.

I could almost convince myself that you know something I don't about the way calculators work, but after the 12-year-old comment by "best experts" was never backed up by anything, I had to jump ship. Where are you pulling this stuff?

I completely don't understand this article, and I've been a (rather good) software developer for 10 years. Calculators can't add 200 + 200? What? Huh? I don't get it.

Their processors are also not using lookup tables. Long ago in the 70's there was a processor that did that, but it had too many limitations.

I have no idea what the hell you're talking about here.

Also why the fuck is my email address required? Why do weblogs do that...

This old post led me to an interesting question: will AI find itself in the position of our fictional philosophers of addition? The basic four functions of arithmetic are so fundamental to the operation of the digital computer that an intelligence built on digital circuitry might well have no idea of how it adds numbers together (unless told by a computer scientist, of course).

Bog: You are correct. That is, you do not understand this article at all. Pay attention to the first word, "Suppose..."

We are not talking about how calculators are designed in reality. We are discussing how they are designed in a hypothetical world where the mechanism of arithmetic is not well-understood.

"Like shooting blindfolded at a distant target"

So long as you know where the target is within five feet, it doesn't matter how small it is, how far away it is, whether or not you're blindfolded, or whether or not you even know how to use a bow. You'll hit it on a natural twenty. http://www.d20srd.org/srd/combat/combatStatistics.htm#attackRoll

Thread necromancy:

Suppose that human beings had absolutely no idea how they performed arithmetic. Imagine that human beings had evolved, rather than having learned, the ability to count sheep and add sheep. People using this built-in ability have no idea how it worked, the way Aristotle had no idea how his visual cortex supported his ability to see things. Peano Arithmetic as we know it has not been invented.

It occured to me that a real life example of this kind of thing is grammar. I don't know what the grammatical rules are for which of the words "I" or "me" should be used when I refer to myself, but I can still use those words with perfect grammar in everyday life*. This may be a better example to use since it's one that everyone can relate to.

*I do use a rule for working out whether I should say "Sarah and I" or "Sarah and me", but that rule is just "use whichever one you would use if you were just talking about youself". Thinking about it now I can guess at the "I/me" rule, but there's plenty of other grammar I have no idea about.

McDermott's old article, "Artificial Intelligence and Natural Stupidity" is a good reference for suggestively-named tokens and algorithms.

Someone needs to teach them how to count: {}, {{}}, {{},{{}}}, {{},{{}},{{},{{}}}}...

when you need to publish at least one paper per month

Gah! Any field with a publishing requirement like that... I shudder.

And... is it me, or is this one of the stupidest discussion threads on this site?

"I don't think we have to wait to scan a whole brain.  Neural networks are just like the human brain, and you can train them to do things without knowing how they do them.  We'll create programs that will do arithmetic without we, our creators, ever understanding how they do arithmetic."

This sort of anti-predicts the deep learning boom, but only sort of.

Fully connected networks didn't scale effectively; researchers had to find (mostly principled, but some ad-hoc) network structures that were capable of more efficiently learning complex patterns.

Also, we've genuinely learned more about vision by realizing the effectiveness of convolutional neural nets.

And yet, the state of the art is to take a generalizable architecture and to scale it massively, not needing to know anything new about the domain, nor learning much new about it. So I do think Eliezer loses some Bayes points for his analogy here, as it applies to games and to language.

I'd be interested to hear whether it is the case that people were saying things like this about AGI two, three, four, five decades ago:

You're all wrong.  Past efforts to create machine arithmetic were futile from the start, because they just didn't have enough computing power.  If you look at how many trillions of synapses there are in the human brain, it's clear that calculators don't have lookup tables anywhere near that large.  We need calculators as powerful as a human brain.  According to Moore's Law, this will occur in the year 2031 on April 27 between 4:00 and 4:30 in the morning.

I'm guessing the answer is yes?

If so, who were these people? It's looking like they are about to be vindicated. Maybe we can go give them a medal, and then learn from their thought processes.

At first I was perplexed, thinking that Yudkovsky for some reason wants to use programs for AI, and not neural networks. This article showed me very clearly why you need to understand the general principle first, and not try to do anything now. Even if you can randomly find answers to a specific quadratic equation, it won't solve even other quadratic equations, let alone cubic or any other problem in mathematics.