Performance Trends in AI

There are some problems with this analysis. First of all, translation is natural language processing. What task requires more understanding of natural language than translation? Second, the BLEU score mentioned is only a cheap and imperfect measure of translation quality. The best measure is human evaluation. And neural machine translation excels at that. Look at this graph. On all languages, the neural network is closer to human performance than the previous method. And on several languages it's extremely close to human performance, and it's translations would almost be indistinguishable from human. That's incredible! And it shows that NNs can handle symbolic problems, which the author disputes.

The biggest problem though, is that all machine learning tasks are expected to have diminishing returns.You can't do better on a classification task than 0% error, for example. You might have an algorithm that is 10x better than another. But it may only do 1% better on some chosen dataset. Because there isn't more than 1% progress that can be made! Just looking at benchmark scores is going to really underestimate the rate of progress.

I might wonder if there are things humans can do with concepts and symbols and principles, the traditional tools of the “higher intellect”, the skills that show up on highly g-loaded tasks, that deep learning cannot do with current algorithms. ... So far, I think there is no empirical evidence from the world of deep learning to indicate that today’s deep learning algorithms are headed for general AI in the near future.

I strongly agree, and I think people at Deepmind already get this because they are working on differentiable neural computers.

Another key point here is that hardware gains such as GPUs and Moore's law increase the returns to investing time and effort into software and research.