Thoughts on Ben Garfinkel's "How sure are we about this AI stuff?"

RE Sarah: Longer timelines don't change the picture that much, in my mind. I don't find this article to be addressing the core concerns. Can you recommend one that's more focused on "why AI-Xrisk isn't the most important thing in the world"?

RE Robin Hanson: I don't really know much of what he thinks, but IIRC his "urgency of AI depends on FOOM" was not compelling.

What I've noticed is that critics are often working from very different starting points, e.g. being unwilling to estimate probabilities of future events, using common-sense rather than consequentialist ethics, etc.

(I do also think I've gotten some reasonable updates from Jeff Kaufman's discussion and arguments (link to the most recent summary he gave but I'm thinking more of various thoughts I've heard him give over the years. I think my notion that X-Risk efforts are hard due to poor feedback loops came from him, which I think is one of the most important concerns to address. Although my impression is he got it from somewhere else originally)

[Edit: much of what Jeff did in his "official research project" was talk to lots of AI professionals, who theoretically had more technically relevant expertise. I'm not sure how many of them fit the criteria I listed in my reply to Ted. My impression is they were some mix of "already involved in the AI Safety field", and "didn't really meet the criteria I think are important for critics", although I don't remember the details offhand.

I do observe that the best internal criticism we have comes from people who roughly buy the Superintelligence arguments and are involved with the AI Alignment field but who disagree a lot on the particulars a la Paul]

Although I'm not the one trying to run a project, there are a couple credentials I'd be looking for to evaluate in a serious critic. (I very much agree with the OP that "serious critics" are an important thing to have more of).

Not meant to be a comment one way or another on whether you fit this, just that you didn't mention it yet:

• Fluency in the arguments presented in Superintelligence (ideally, fluency in the broader spectrum of arguments relating to AI and X-Risk, but Superintelligence does a thorough enough job that it works okay as a litmus test). I stopped paying much attention to AI Safety critics because they didn't seem knowledgeable about the basic arguments, let alone up to speed on the state of the field.
• "Seriousness of trying to solve the problem." I.e if we knew that aliens were coming in 30 years (or 10, or even 300), we wouldn't just shrug if we couldn't come up with tractable things to do. We'd be building some kind of model of whatever-the-best-course-of-action was. If the correct course of action is "wait until the field has progressed more", there should be a clear sense of when you have waited long enough and what to do when things change. (Maybe this point could be summarized as "familiar with and responding to the concerns in No Fire Alarm for AI Safety [LW · GW].")
• The criticism is expecting counter-criticism. i.e. What I think we're missing is critics who are in it for the long haul, who see their work as the first step of an iterative process, with an expectation that the AI safety field will respond and/or update to their critiques.

OTTOMH - Off the top of my head

I feel it's like "A -> likely B" being an evidence for "B -> likely A"; generally true, but it could be either very strong or very weak evidence depending on the base rates of A and B.

Not having knowledgeable criticism against position "2 + 2 = 4" is strong evidence, because many people are familiar with the statement, many use it in their life or work, so if it is wrong, it would be likely someone would already offer some solid criticism.

But for statements that are less known or less cared about, it becomes more likely that there are good arguments against them, but no one noticed them yet, or no one bothered to write a solid paper about them.