The AI Timelines Scam

1. For reasons discussed on comments to previous posts here, I'm wary of using words like "lie" or "scam" to mean "honest reporting of unconsciously biased reasoning". If I criticized this post by calling you a liar trying to scam us, and then backed down to "I'm sure you believe this, but you probably have some bias, just like all of us", I expect you would be offended. But I feel like you're making this equivocation throughout this post.

2. I agree business is probably overly optimistic about timelines, for about the reasons you mention. But reversed stupidity is not intelligence [LW · GW]. Most of the people I know pushing short timelines work in nonprofits, and many of the people you're criticizing in this post are AI professors. Unless you got your timelines from industry, which I don't think many people here did, them being stupid isn't especially relevant to whether we should believe the argument in general. I could find you some field (like religion) where people are biased to believe AI will never happen, but unless we took them seriously before this, the fact that they're wrong doesn't change anything.

3. I've frequently heard people who believe AI might be near say that their side can't publicly voice their opinions, because they'll get branded as loonies and alarmists, and therefore we should adjust in favor of near-termism because long-timelinists get to unfairly dominate the debate. I think it's natural for people on all sides of an issue to feel like their side is uniquely silenced by a conspiracy of people biased towards the other side. See Against Bravery Debates for evidence of this.

4. I'm not familiar with the politics in AI research. But in medicine, I've noticed that doctors who go straight to the public with their controversial medical theory are usually pretty bad, for one of a couple of reasons. Number one, they're usually wrong, people in the field know they're wrong, and they're trying to bamboozle a reading public who aren't smart enough to figure out that they're wrong (but who are hungry for a "Galileo stands up to hidebound medical establishment" narrative). Number two, there's a thing they can do where they say some well-known fact in a breathless tone, and then get credit for having blown the cover of the establishment's lie. You can always get a New Yorker story by writing "Did you know that, contrary to what the psychiatric establishment wants you to believe, SOME DRUGS MAY HAVE SIDE EFFECTS OR WITHDRAWAL SYNDROMES?" Then the public gets up in arms, and the psychiatric establishment has to go on damage control for the next few months and strike an awkward balance between correcting the inevitable massive misrepresentations in the article while also saying the basic premise is !@#$ing obvious and was never in doubt. When I hear people say something like "You're not presenting an alternative solution" in these cases, they mean something like "You don't have some alternate way of treating diseases that has no side effects, so stop pretending you're Galileo for pointing out a problem everyone was already aware of." See Beware Stephen Jay Gould [LW · GW] for Eliezer giving an example of this, and Chemical Imbalance and the followup post for me giving an example of this. I don't know for sure that this is what's going on in AI, but it would make sense.

I'm not against modeling sociopolitical dynamics. But I think you're doing it badly, by taking some things that people on both sides feel, applying it to only one side, and concluding that means the other is involved in lies and scams and conspiracies of silence (while later disclaiming these terms in a disclaimer, after they've had their intended shocking effect).

I think this is one of the cases where we should use our basic rationality tools like probability estimates. Just from reading this post, I have no idea what probability Gary Marcus, Yann LeCun, or Steven Hansen has on AGI in ten years (or fifty years, or one hundred years). For all I know all of them (and you, and me) have exactly the same probability and their argument is completely political about which side is dominant vs. oppressed and who should gain or lose status (remember the issue where everyone assumes LWers are overly certain cryonics will work, whereas in fact they're less sure of this than the general population and just describe their beliefs differently [LW · GW] ). As long as we keep engaging on that relatively superficial monkey-politics "The other side are liars who are silencing my side!" level, we're just going to be drawn into tribalism around the near-timeline and far-timeline tribes, and our ability to make accurate predictions is going to suffer. I think this is worse than any improvement we could get by making sociopolitical adjustments at this level of resolution.

I agree with:

• Most people trying to figure out what's true should be mostly trying to develop views on the basis of public information and not giving too much weight to supposed secret information.
• It's good to react skeptically to someone claiming "we have secret information implying that what we are doing is super important."
• Understanding the sociopolitical situation seems like a worthwhile step in informing views about AI.
• It would be wild if 73% of tech executives thought AGI would be developed in the next 10 years. (And independent of the truth of that claim, people do have a lot of wild views about automation.)

I disagree with:

• Norms of discourse in the broader community are significantly biased towards short timelines. The actual evidence in this post seems thin and cherry-picked. I think the best evidence is the a priori argument "you'd expect to be biased towards short timelines given that it makes our work seem more important." I think that's good as far as it goes but the conclusion is overstated here.
• "Whistleblowers" about long timelines are ostracized or discredited. Again, the evidence in your post seems thin and cherry-picked, and your contemporary example seems wrong to me (I commented separately). It seems like most people complaining about deep learning or short timelines have a good time in the AI community, and people with the "AGI in 20 years" view are regarded much more poorly within academia and most parts of industry. This could be about different fora and communities being in different equilibria, but I'm not really sure how that's compatible with "ostracizing." (It feels like you are probably mistaken about the tenor of discussions in the AI community.)
• That 73% of tech executives thought AGI would be developed in the next 10 years. Willing to bet against the quoted survey: the white paper is thin on details and leaves lots of wiggle room for chicanery, while the project seems thoroughly optimized to make AI seem like a big deal soon. The claim also just doesn't seem to match my experience with anyone who might be called tech executives (though I don't know how they constructed the group).

Hi Jessica. Nice post and I agree with many of your points. Certainly, I believe—as you do—that a number of bad actors are wielding the specter of AGI sloppily and irresponsibly, either to consciously defraud people or on account of buying into something that speaks more to the messianic than to science. Perhaps ironically, one frequent debate that I have had with Gary in the past is that while he is vocally critical of exuberance over deep learning, he is himself partial to speaking rosily of nearish-term AGI, and of claiming progress (or being on the verge of progress) towards it. On the other hand, I am considerably more skeptical.

While I enjoyed the post and think we agree on many points, if you don’t mind I would like to respectfully note that I’ve been quoted here slightly out of context and would like to supply that missing context. To be sure, I think your post is written well and with honest intentions, and I know how easy it is to miss some context in Twitter threads [especially as it seems that many tweets have been deleted from this thread].

Regarding my friend Gary Marcus. I like Gary and we communicate fairly regularly, but we don’t always agree on the science or on the discourse.

In this particular case, he was specifically insisting to be “the first” to make a set of arguments (which contradicted my understanding of history). When I say “Saying it louder ≠ saying it first.”, I am truly just pushing back on this specific point—the assertion that he said it first. Among others, Judea Pearl has argued the limits of curve fitting far more rigorously and far earlier in history.

There is nothing dishonest or contradictory about agreeing with a broader point and simultaneously disagreeing with a man’s claim to have originated it. I took exception to the hyperbole, not the message.

After the quote, your post notes: “but, this is essentially admitting that Marcus is correct, while still criticizing him for saying it”—what is there to “admit”? I myself have made similar critical arguments both in technical papers, position papers, blog posts, and the popular press. Characterizing “agreement” as “admitting” something, makes the false insinuation that somehow I have been on the wrong side of the debate.

For the record, I am acknowledged in Gary’s paper on limitations of deep learning (which you reference here) for giving him a large amount of constructive feedback and have myself, perhaps in defiance of Adam Smith’s aphorism, been vocally critical of my own community within the technical forums, recently publishing “Troubling Trends in Machine Learning Scholarship” at the ICML debates (https://arxiv.org/abs/1807.03341) which was subsequently published by CACM. I suspect this piece (as well as much of my writing at http://approximatelycorrect.com and in other formal position papers) is in the spirit of the sort of critical writing that you appear to encourage.

Further, I’d like to address the second part of the discussion. When I say “Saying X doesn’t solve Y is pretty easy. But where are your concrete solutions for Y?” My point here is that Gary doesn’t just push back on the false claims made about current technology that doesn’t do Y. He also sometimes makes public attacks on the people working on X. It would seem that their crime is that they haven’t developed technical solutions to the grand challenge of Y. If failing to solve these particular moonshots (true reasoning, solving common sense, an elegant formulation of symbol manipulation and synthesis with pattern recognition) is a crime, then Gary too is just as guilty and the attack ought to be levied with greater humility. These attacks strike me as inappropriate and misplaced (compared to the more reasonable push-back on misinformation in the public sphere. ***To be clear, while I understand why you might have drawn that conclusion from this half-tweet, I do not believe that one must have a solution in hand to levy criticism and my writing and technical papers attest to this.***

Basically, AI professionals seem to be trying to manage the hype cycle carefully.

Ignorant people tend to be more all-or-nothing than experts. By default, they'll see AI as "totally unimportant or fictional", "a panacea, perfect in every way" or "a catastrophe, terrible in every way." And they won't distinguish between different kinds of AI.

Currently, the hype cycle has gone from "professionals are aware that deep learning is useful" (c. 2013) to "deep learning is AI and it is wonderful in every way and you need some" (c. 2015?) to "maybe there are problems with AI? burn it with fire! Nationalize! Ban!" (c. 2019).

Professionals who are still working on the "deep learning is useful for certain applications" project (which is pretty much where I sit) are quite worried about the inevitable crash when public opinion shifts from "wonderful panacea" to "burn it with fire." When the public opinion crash happens, legitimate R&D is going to lose funding, and that will genuinely be unfortunate. Everyone savvy knows this will happen. Nobody knows exactly when. There are various strategies for dealing with it.

Accelerate the decline: this is what Gary Marcus is doing.

Carve out a niche as an AI Skeptic (who is still in the AI business himself!) Then, when the funding crunch comes, his companies will be seen as "AI that even the skeptic thinks is legit" and have a better chance of surviving.

Be Conservative: this is a less visible strategy but a lot of people are taking it, including me.

Use AI only in contexts that are well justified by evidence, like rapid image processing to replace manual classification. That way, when the funding crunch happens, you'll be able to say you're not just using AI as a buzzword, you're using well-established, safe methods that have a proven track record.

Pivot Into Governance: this is what a lot of AI risk orgs are doing

Benefit from the coming backlash by becoming an advisor to regulators. Make a living not by building the tech but by talking about its social risks and harms. I think this is actually a fairly weak strategy because it's parasitic on the overall market for AI. There's no funding for AI think tanks if there's no funding for AI itself. But it's an ideal strategy for the cusp time period when we're just shifting between blind enthusiasm to blind panic.

Preserve Credibility: this is what Yann LeCun is doing and has been doing from day 1 (he was a deep learning pioneer and promoter even before the spectacular empirical performance results came in)

Try to forestall the backlash. Frame AI as good, not bad, and try to preserve the credibility of the profession as long as you can. Argue (honestly but selectively) against anyone who says anything bad about deep learning for any reason.

Any of these strategies may say true things! In fact, assuming you really are an AI expert, the smartest thing to do in the long run is to say only true things, and use connotation and selective focus to define your rhetorical strategy. Reality has no branding; there are true things to say that comport with all four strategies. Gary Marcus is a guy in the "AI Skeptic" niche saying things that are, afaik, true; there are people in that niche who are saying false things. Yann LeCun is a guy in the "Preserve AI Credibility" niche who says true things; when Gary Marcus says true things, Yann LeCun doesn't deny them, but criticizes Marcus's tone and emphasis. Which is quite correct; it's the most intellectually rigorous way to pursue LeCun's chosen strategy.

[Note: this, and all comments on this post unless specified otherwise, is written with my 'LW user' hat on, not my 'LW Admin' or 'MIRI employee' hat on, and thus is my personal view instead of the LW view or the MIRI view.]

As someone who thinks about AGI timelines a lot, I find myself dissatisfied with this post because it's unclear what "The AI Timelines Scam" you're talking about, and I'm worried if I poke at the bits it'll feel like a motte and bailey, where it seems quite reasonable to me that '73% of tech executives thinking that the singularity will arrive in <10 years is probably just inflated 'pro-tech' reasoning,' but also it seems quite unreasonable to suggest that strategic considerations about dual use technology should be discussed openly (or should be discussed openly because tech executives have distorted beliefs). It also seems like there's an argument for weighting urgency in planning that could lead to 'distorted' timelines while being a rational response to uncertainty.

On the first point, I think the following might be a fair description of some thinkers in the AGI space, but don't think this is a fair summary of MIRI (and I think it's illegible, to me at least, whether you are intending this to be a summary of MIRI):

This bears similarity to some conversations on AI risk I've been party to in the past few years. The fear is that Others (DeepMind, China, whoever) will develop AGI soon, so We have to develop AGI first in order to make sure it's safe, because Others won't make sure it's safe and We will. Also, We have to discuss AGI strategy in private (and avoid public discussion), so Others don't get the wrong ideas. (Generally, these claims have little empirical/rational backing to them; they're based on scary stories, not historically validated threat models)

I do think it makes sense to write more publicly about the difficulties of writing publicly, but there's always going to be something odd about it. Suppose I have 5 reasons for wanting discussions to be private, and 3 of them I can easily say. Discussing those three reasons will give people an incomplete picture that might seem complete, in a way that saying "yeah, the sum of factors is against" won't. Further, without giving specific examples, it's hard to see which of the ones that are difficult to say you would endorse and which you wouldn't, and it's not obvious to me legibility is the best standard here.

But my simple sense is that openly discussing whether or not nuclear weapons were possible (a technical claim on which people might have private information, including intuitions informed by their scientific experience) would have had costs and it was sensible to be secretive about it. If I think that timelines are short because maybe technology X and technology Y fit together neatly, then publicly announcing that increases the chances that we get short timelines because someone plugs together technology X and technology Y. It does seem like marginal scientists speed things up here.

Now, I'm paying a price here; it may be the case that people have tried to glue together technology X and technology Y and it won't work. I think private discussions on this are way better than no discussions on this, because it increases the chances that those sorts of crucial facts get revealed. It's not obvious that public discussions are all that much better on these grounds.

On the second point, it feels important to note that the threshold for "take something seriously" is actually quite small. I might think that the chance that I have Lyme disease is 5%, and yet that motivates significant action because of hugely asymmetric cost considerations, or rapid decrease in efficacy of action. I think there's often a problem where someone 'has short timelines' in the sense that they think 10-year scenarios should be planned about at all, but this can be easily mistaken for 'they think 10-year scenarios are most likely' because often if you think both an urgent concern and a distant concern are possible, almost all of your effort goes into the urgent concern instead of the distant concern (as sensible critical-path project management would suggest).

For reference, the Gary Marcus tweet in question is:

“I’m not saying I want to forget deep learning... But we need to be able to extend it to do things like reasoning, learning causality, and exploring the world .” - Yoshua Bengio, not unlike what I have been saying since 2012 in The New Yorker.

I think Zack Lipton objected to this tweet because it appears to be trying to claim priority. (You might have thought it's ambiguous whether he's claiming priority, but he clarifies in the thread: "But I did say this stuff first, in 2001, 2012 etc?") The tweet and his writings more generally imply that people in the field have recently changed their view to agree with him, but many people in the field object strongly to this characterization.

The tweet is mostly just saying "I told you so." That seems like a fine time for people to criticize him about making a land grab rather than engaging on the object level, since the tweet doesn't have much object-level content. For example:

"Saying it louder ≠ saying it first. You can't claim credit for differentiating between reasoning and pattern recognition." [...] is essentially a claim that everybody knows that deep learning can't do reasoning. But, this is essentially admitting that Marcus is correct, while still criticizing him for saying it.

Hopefully Zack's argument makes more sense if you view it as a response to Gary Marcus claiming priority. Which is what Gary Marcus was doing and clearly what Zack is responding to. This is not a substitute for engagement on the object level. Saying "someone else, and in fact many people in the relevant scientific field, already understood this point" is an excellent response to someone who's trying to claim credit for the point.

There are reasonable points to make about social epistemology here, but I think you're overclaiming about the treatment of critics, and that this thread in particular is a bad example to point to. It also seems like you may be mistaken about some of the context. (Zack Lipton has no love for short-timelines-pushers and isn't shy about it. He's annoyed at Gary Marcus for making bad arguments and claiming unwarranted credit, which really is independent of whether some related claims are true.)

in part because I don't have much to say on this issue that Gary Marcus hasn't already said.

It would be interesting to know which particular arguments made by Gary Marcus you agree with, and how you think they relate to arguments about timelines.

In this preliminary doc, it seems like most the disagreement is driven by saying there is a 99% probability that training a human-level AI would take more than 10,000x more lifetimes than AlphaZero took games of go (while I'd be at more like 50%, and have maybe 5-10% chance that it will take many fewer lifetimes). Section 2.0.2 admits this is mostly guesswork, but ends up very confident the number isn't small. It's not clear where that particular number comes from, the only evidence gestured at is "the input is a lot bigger, so it will take a lot more lifetimes" which doesn't seem to agree with our experience so far or have much conceptual justification. (I guess the point is that the space of functions is much bigger? but if comparing the size of the space of functions, why not directly count parameters?) And why is this a lower bound?

Overall this seems like a place you disagree confidently with many people who entertain shorter timelines, and it seems unrelated to anything Gary Marcus says.

Exercise for those (like me) who largely agreed with the criticism that the usage of "scam" in the title was an instance of the noncentral fallacy (drawing the category boundaries of "scam" too widely in a way that makes the word less useful): do you feel the same way about Eliezer Yudkowsky's "The Two-Party Swindle" [LW · GW]? Why or why not?

I almost wrote a post today with roughly the following text. It seems highly related so I guess I'll write a short version.

The ability to sacrifice today for tomorrow is one of the hard skills all humans must learn. To be able to make long-term plans is not natural. Most people around me seem to be able to think about today and the next month, and occasionally the next year (job stability, housing stability, relationship stability, etc), but very rarely do I see anyone acting on plans with timescales of decades (or centuries). Robin Hanson has written about how humans naturally think in a very low-detail and unrealistic mode in far (as opposed to near) thinking, and I know that humans have a difficult time with scope sensitivity.

It seems to me that is is a very common refrain in the community to say "but timelines are short" in response to someone's long-term thinking or proposal, to suggest somewhere in the 5-25 year range until no further work matters because an AGI Foom has occured. My epistemic state is that even if this is true (which it very well may be), most people who are thinking this way are not in fact making 10 year plans. They are continuing to make at most 2 year plans, while avoiding learning how to make longer term plans.

There is a two-step form of judo required to first learn to make 50 year plans and then secondarily restrict yourself to shorter-term plans. It is not one move, and I often see "but timelines are short" used to prevent someone from learning the first move.

I had not considered until the OP that this was actively adversarially selected for, certainly in industry, but it does seem compelling as a method for making today very exciting and stopping people from taking caution for the long-term. Will think on it more.

[purely personal view]

It seems quite easy to imagine similarly compelling socio-political and subconscious reasons why people working on AI could be biased against short AGI timelines. For example

• short timelines estimates make broader public agitated, which may lead to state regulation or similar interference [historical examples: industries trying to suppress info about risks]
• researchers mostly want to work on technical problems, instead of thinking about nebulous future impacts of their work; putting more weight on short timelines would force some people to pause and think about responsibility, or suffer some cognitive dissonance, which may be unappealing/unpleasant for S1 reasons [historical examples: physicists working on nuclear weapons]
• fears claims about short timelines would get pattern-matched as doomsday fear-mongering / sensationalist / subject of scifi movies ...

While I agree motivated reasoning is a serious concern, I don't think it's clear how do the incentives sum up. If anything, claims like "AGI is unrealistic or very far away, however practical applications of narrow AI will be profound" seems to capture most of the purported benefits (AI is important) and avoid the negatives (no need to think).

Planned summary:

This post argues that AI researchers and AI organizations have an incentive to predict that AGI will come soon, since that leads to more funding, and so we should expect timeline estimates to be systematically too short. Besides the conceptual argument, we can also see this in the field's response to critics: both historically and now, criticism is often met with counterarguments based on "style" rather than engaging with the technical meat of the criticism.

Planned opinion:

I agree with the conceptual argument, and I think it does hold in practice, quite strongly. I don't really agree that the field's response to critics implies that they are biased towards short timelines -- see these [LW(p) · GW(p)] comments [LW(p) · GW(p)]. Nonetheless, I'm going to do exactly what this post critiques, and say that I put significant probability on short timelines, but not explain my reasons (because they're complicated and I don't think I can convey them, and certainly can't convey them in a small number of words).

I see clear parallels with the treatment of Sabine Hossenfelder blowing the whistle on the particle physics community pushing for a new $20B particle accelerator. She has been going through the same adversity as any high-profile defector from a scientific community, and the arguments against her are the same ones you are listing.

This is a cogent, if sparse, high-level analysis of the epistemic distortions around megaprojects [LW · GW] in AI and other fields.

It points out that projects like the human brain project and the fifth generation computer systems project made massive promises, raised around a billion dollars, and totally flopped. I don't expect this was a simple error, I expect there were indeed systematic epistemic distortions involved, perpetuated at all levels.

It points out that similar scale projects are being evaluated today involving various major AI companies globally, and points out that the sorts of distortionary anti-epistemic tendencies can still be observed. Critics of the ideas that are currently getting billions of dollars (deep learning leading to AGI) are met with replies that systematically exclude the possibility of 'stop, halt, and catch fire' but instead only include 'why are you talking about problems and not solutions' and 'do this through our proper channels within the field and not in this unconstrained public forum', which are clearly the sorts you'd expect to see when a megaproject is protecting itself.

The post briefly also addresses why it's worth modeling the sociopolitical arguments, and not just the technical arguments. I think it's clear that megaprojects like this are subject to major distortionary forces – at the point where you're talking about arguments against the positions that is literally funding the whole field, it is obviously not acceptable to constrain dialogue to the channels that field controls, this is a mechanism that is open to abuse of power. I like this short section.

The post ends with the claim that 'people are being duped into believing a lie'. I don't feel convinced of this.

I tried to write down why simply, but I'm not having the easiest time. A few pointers:

• A chain is as strong as its weakest link, but not all organisations are chains. Many mathematicians can be doing nonsense symbol-manipulation while Andrew Wiles solves Fermat's Last Theorem. I expect there was an overlap in the time today when science is substantially broken, and the time when Feynman was around making diagrams and building the atom bomb. In the intermediary time a lot of 'science' you could point to as not actually science and supported by anti-epistemic arguments, but this was somewhat separable from Feynman who was still doing real work.
• There can be many levels of fraud, combined with many levels of actual competence at the object level. The modern field of ML has mightily impressed me with AlphaGo and GPT and so on. I think that the "full scam" position is that this is entirely a consequence of increased compute and not ML expertise, and that basically there is not much expertise at all in these fields. I find this plausible but not at the 50% level. So just because there's evidence of anti-epistemic and adversarial behavior, this does not preclude real work from being done.
• I do think it's pretty normal for projects to have marketing, run in an epistemically adversarial way, kept in arms length and bringing in resources.
• I also think that sometimes very competent people are surrounding by distortionary forces. I think I should be able to come up with strong examples here, and I thought a bit about making the case for Thiel or Cummings (who've both shown ability to think clearly but also engaged in somewhat political narrative-building). Perhaps Hoover is an example? Still, I think that sometimes a project can be engaged adversarially with the outside world and still be competent at its work. But I don't think I've shown that strongly, and in most actual cases I am repulsed by projects that do the adversarial stuff and think it's delusional to be holding out hope. I also think it's especially delusional to think this about science. Science isn't supposed to be a place where the real conversation happens in private.

Conclusion

I think this post raises a straightforward and valid hypothesis to evaluate the field against as a whole. I don't think it's sufficiently detailed to convince me that the overall hypothesis holds. I do think it's a valuable conversation to have, it's such an important topic, especially for this community. I think this post is valuable, and I expect I will give it a small positive vote in the review, around +2 or +3.

Further Work

Here are some further questions I'd like to see discussed and answered, to get a better picture of this:

• What are a few other examples of criticism of the current wave of AI hype, and how were they dealt with?
• What do leaders of these projects say on this topic, and in response to criticism?
  • (I recall an FLI panel with Demis Hassabis on, where the on detailed argument he made about the decision to put more/less funding into AGI right now was saying that it would be easier for lots of groups to do it in the future as compute gets cheaper, so in order to have centralized control and be able to include time for safety we should push as fast as we can on AGI now. I don't think it's an unreasonable argument but I was hardly surprised to hear it coming from him.)
• How open are the channels of communication with the field? How easy is it for an outsider to engage with the people in the field?
• Who are the funders of AI? To what extent are they interested in public discourse around this subject?
  • (My guess is that the answer here is something like "the academic field and industry have captured the prestige associated with it so that nobody else is considered reasonable to listen to".)
• What is the state of the object level arguments around the feasibility of AGI?
• Does the behavior of the people who lead the field matches up with their claims?
• What are some other megaprojects or fields with billions of dollars going into projects, and how are these dynamics playing out in those areas?

Hubert Dreyfus, probably the most famous historical AI critic, published "Alchemy and Artificial Intelligence" in 1965, which argued that the techniques popular at the time were insufficient for AGI.

That is not at all what the summary says. Here is roughly the same text from the abstract:

Early successes in programming digital computers to exhibit simple forms of intelligent behavior, coupled with the belief that intelligent activities differ only in their degree of complexity, have led to the conviction that the information processing underlying any cognitive performance can be formulated in a program and thus simulated on a digital computer. Attempts to simulate cognitive processes on computers have, however, run into greater difficulties than anticipated. An examination of these difficulties reveals that the attempt to analyze intelligent behavior in digital computer language systematically excludes three fundamental human forms of information processing (fringe consciousness, essence/accident discrimination, and ambiguity tolerance). Moreover, there are four distinct types of intelligent activity, only two of which do not presuppose these human forms of information processing and can therefore be programmed. Significant developments in artificial intelligence in the remaining two areas must await computers of an entirely different sort, of which the only existing prototype is the little-understood human brain.

In case you thought he just meant greater speed, he says the opposite on PDF page 71. Here is roughly the same text again from a work I can actually copy and paste:

It no longer seems obvious that one can introduce search heuristics which enable the speed and accuracy of computers to bludgeon through in those areas where human beings use more elegant techniques. Lacking any a priori basis for confidence, we can only turn to the empirical results obtained thus far. That brute force can succeed to some extent is demonstrated by the early work in the field. The present difficulties in game playing, language translation, problem solving, and pattern recognition, however, indicate a limit to our ability to substitute one kind of "information processing** for another. Only experimentation can determine the extent to which newer and faster machines, better programming languages, and cleverer heuristics can continue to push back the frontier. Nonetheless, the dra- matic slowdown in the fields we have considered and the general failure to fulfill earlier predictions suggest the boundary may be near. Without the four assumptions to fall back on, current stagnation should be grounds for pessimism.

This, of course, has profound implications for our philosophical tradi- tion. If the persistent difficulties which have plagued all areas of artificial intelligence are reinterpreted as failures, these failures must be interpre- ted as empirical evidence against the psychological, epistemological, and ontological assumptions. In Heideggerian terms this is to say that if Western Metaphysics reaches its culmination in Cybernetics, the recent difficulties in artificial intelligence, rather than reflecting technological limitations, may reveal the limitations of technology.

If indeed Dreyfus meant to critique 1965's algorithms - which is not what I'm seeing, and certainly not what I quoted - it would be surprising for him to get so much wrong. How did this occur?

The missile gap was a lie by the US Air Force to justify building more nukes, by falsely claiming that the Soviet Union had more nukes than the US

This statement is not supported by the link used as a reference.  Was it a lie?  The reference speaks to failed intelligence and political manipulation using the perceived gap. The phrasing above suggests conspiracy.

This post was very helpful to me when I read it, in terms of engaging more with this hypothesis. The post isn't very rigorous and I think doesn't support the hypothesis very well, but nonetheless it was pretty helpful to engage with the perspective (I also found the comments valuable), so I'm nominating it for its positive effects for me personally.

I think a lot of the animosity that Gary Markus drew was less that some of his points were wrong, and more that he didn't seem to have a full grasp of the field before criticizing it. Here's an r/machinelearning thread on one of his papers. Granted, r/ML is not necessarily representative of the AI community, especially now, but you see some people agreeing with some of his points, and others claiming that he's not up to date with current ML research. I would recommend people take a look at the thread, to judge for themselves.

I'm also not inclined to take any twitter drama as strong evidence of the attitudes of the general ML community, mainly because twitter seems to strongly encourage/amplify the sort of argumentative shit-flinging pointed out in the post.

This was important to the discussions around timelines at the time, back when the talk about timelines felt central. This felt like it helped give me permission to no longer consider them as central, and to fully consider a wide range of models of what could be going on. It helped make me more sane, and that's pretty important.

It was also important for the discussion about the use of words and the creation of clarity. There's been a long issue of exactly when and where to use words like "scam" and "lie" to describe things - when is it accurate, when is it useful, what budgets does it potentially use up? How can we describe what we see in the world in a way that creates common knowledge if we can't use words that are literal descriptions? It's something I still struggle with, and this is where the key arguments got made. 

Thus, on reflection, I'd like to see this included.

I liked the comments on this post more than I liked the post itself. As Paul commented, there's as much criticism of short AGI timelines as there is of long AGI timelines; and as Scott pointed out, this was an uncharitable take on AI proponents' motives.

Without the context of those comments, I don't recommend this post for inclusion.

At the time, I argued pretty strongly against parts of this post, and I still think my points are valid and important. That said, I think in retrospect this post had a large impact; I think it kicked off several months of investigation of how language works and what discourse norms should be in the presence of consequences. I'm not sure it was the best of 2019, but it seems necessary to make sense of 2019, or properly trace the lineage of ideas?

Which is not to say that modeling such technical arguments is not important for forecasting AGI. I certainly could have written a post evaluating such arguments, and I decided to write this post instead, in part because I don’t have much to say on this issue that Gary Marcus hasn’t already said.

Is he an AI researcher though? Wikipedia says he's a psychology professor, and his arXiv article criticizing deep learning doesn't seem to have much math. If you have technical arguments, maybe you could state them?

The key sentiment of this post that I currently agree with:

• There's a bit of a short timelines "bug" in the Berkeley rationalist scene, where short timelines have become something like the default assumption (or at least are not unusual). 
• There don't seem to be strong, public reasons for this view. 
• It seems like most people who are sympathetic to short timelines are sympathetic to it mainly as the result of a social proof cascade. 
• But this is obscured somewhat, because some folks who opinions are being trusted, don't show their work (rightly or wrongly), because of info security considerations.

This reminds me of related questions around slowing down AI, discussing AI with a mass audience, or building public support for AI policy (e.g. https://forum.effectivealtruism.org/posts/pm6Mn4a3h4oekCCay/two-strange-things-about-ai-safety-policy, [EA · GW] http://www.zachgroff.com/2017/08/does-ai-safety-and-effective-altruist.html). A lot of the arguments against doing these things have this same motivation that we are concerned about the others for reasons that are somewhat abstruse. Where would these "sociopolitical" considerations get us on these questions?

Yeah, 10/10 agreement on this. Like it'd be great if you could "just" donate to some AI risk org and get the promised altruistic benefits, but if you actually care about "stop all the fucking suffering I can", then you should want to believe AI risk research is a scam if it is a scam.

At which point you go oh fuck, I don't have a good plan to save the world anymore. But not having a better plan shouldn't change your beliefs on whether AI risk research is effective.

Why not treat this in Bayesian way? There is 50 per cent a priory credence in short time line, and 50 per cent on long one. In that case, we still need to get working AI safety solutions ASAP, even if there is 50 per cent chance that the money on AI safety will be just lost? (Disclaimer: I am not paid for any AI safety research, except Good AI prize of 1500 USD, which is not related to timelines.)