Thanks for the post. It'd be helpful to have a TL;DR for this (an abstract), since it's kinda long - what are the main points you're trying to get across?

I mean, yes, humans make mistakes too. Do our most high level mistakes like "Andrew Wiles' first proof of Fermat's Theorem was wrong" affect much our ability to be vastly superior to chimpanzees in any conflict with them?

Overall I agree with the statements here in the mathematical sense, but I disagree about how much to index on them for practical considerations. Upvoted because I think it is a well-laid-out description of a lot of peoples' reasons for believing AI will not be as dangerous as others fear.

First, do you agree that additional knowing-that reduces the amount of failure needed to achieve knowing-how? 

If not, are you also of the opinion that schools, education as a concept, books and similar storage media, or other intentional methods of imparting know-how between humans to have zero value? My understanding is that dissemination of information enabling learning from other people's past failures is basically the fundamental reason for humanity's success following the inventions of language, writing, and the printing press.

If so, where do you believe the upper bound on that failure-reduction-potential lies, in the limit of very high intelligence coupled with very high computing power? With how large an error bar on said upper bound? Why there? And does your estimate imply near-zero potential for the limit to be high enough to create catastrophic or existential risk?

Second, I agree that there is always a harder problem, and that such problems will still exist for anything that, to a human, would count as ASI. How certain are you that any given AI's limits will (in the important cases) only include things recognizable by humans in advance during planning or later during action as mistakes, in a way that reliably provides us opportunity to intervene in ways the AI did not anticipate as plausible failure modes? In other words, the universe may have limitless complexity, but it is not at all clear to me that the kinds of problems an AI would need to want to solve to present an existential risk to humans would require it to tackle much of that complexity. They may be problems a human could reliably solve given 1000 years subjective thinking time followed by 100 simultaneous "first tries" of various candidate plans, only one of which needs to succeed. If even one such case exists, I would expect anything worth calling an ASI to be able to figure such plans out in a matter of minutes, hours at most, maybe days if trying to only use spare compute that won't be noticed.

Third, I agree that it will often be in an AI's best interests, especially early on, to do nothing and bide its time, even if it thinks some plan could probably succeed but might become visible and get it destroyed or changed. This is where the concepts of deceptive alignment and a sharp left turn came from, 15-20 years ago IIRC, though the terminology and details have changed over time. However, at this point I expect that within the next couple of years millions of people will eagerly hand various AI systems near-unfettered access to their email, social media, bank and investment accounts, and so on. GPT-6 and its contemporaries will have access to millions of legal identities and many billions of dollars belonging to the kinds of people willing to mostly let an AI handle many details of their lives with minimal oversight. I see little reason to expect these systems will be significantly harder to jailbreak than all the releases so far. 

Fourth, even if it does take many years for any AI to feel established enough to risk enacting a dangerous plan, humans are human. Each year it doesn't happen will be taken as evidence it won't, and (some) people will be even less cautious than they are now. It seems to me that the baseline path is that the humans, and then the AIs, will be on the lookout for likely catastrophic capabilities failures, and iteratively fix them in order to make the AI more instrumentally useful, until the remaining failure modes that exist are outside our ability to anticipate or fix; then things chug along seeming to have gone well for some length of time, and we just kinda have to hope that length of time is very very long or infinite.

I agree that a superintelligence might make mistakes. In fact I still believe the first AGI may be a good engineer but bad strategist [LW · GW]. I completely agree a smart but unwise superintelligence is dangerous, and may build a greater superintelligence misaligned to even it.

However I think mistakes will almost completely disappear above a certain level of extreme superintelligence.

A truly intelligent being doesn't just fit models to empirical data, but fits simulations to empirical data.

After it fits a simulation to the empirical data, it then fits a model to the simulation. This "model fitted to a simulation fitted to empirical data" will generalize far better than a model directly fitted to empirical data.

It can then use this model to run "cheap simulations" of higher level phenomena, e.g. it models atoms to run cheap simulations of molecules, it models molecules to run cheap simulations of cells, it models cells to run cheap simulations of humans, it models humans to run cheap simulations of the world.

Simulations fit empirical data much better than directly fitting models to empirical data, because a simulation may be made out of millions of "identical" objects. Each "identical" object has the same parameters. This means the independent parameters of the simulation may be a millionfold fewer than a model with the same number of moving parts. This means you need far fewer empirical data to fit the simulation to empirical data, assuming the real world has the same shape as the simulation, and is also made up of many moving parts "with the same parameters."

I copied this from my own post about Scanless Whole Brain Emulation [LW · GW], but that's very off topic :)

EDIT: one thing simulations cannot predict are even smarter superintelligences, since simulating them equals building them. As long as it is wise enough to understand this, it can find solutions which prevent smarter superintelligences from being built, and then do a risk analysis. The universe might last for a trillion years, so building a smarter superintelligence 1000 years sooner has negligible benefit. The more time it spends planning how to build it safely, the lower the risk it gets it wrong.