What are the flaws in this AGI argument?

So, if I were to point out what seems like the most dubious premise of the argument, it would be "An AGI that has at least one mistake in its alignment model will be unaligned."

Deep Learning is notorious for continuing to work while there are mistakes in it; you can accidentally leave all kinds of things out and it still works just fine. There are of course arguments [? · GW] that value is fragile and that if we get it 0.1% wrong then we lose 99.9% of all value in the universe, just as there are arguments [? · GW] that the aforementioned arguments are quite wrong. But "one mistake" = "failure" is not a firm principle in other areas of engineering, so it's unlikely to be the case.

But, apart from whether individual premises are true or false -- in general, it might help if, instead of asking yourself, "What kind of arguments can I make for or against AI doom?" you instead asked "hey, what would AI doom predict about the world, and are those predictions coming true?"

It's a feature of human cognition that we can make what feel like good arguments for anything. The history of human thought before the scientific method is people making lots of good arguments for or against God's existence, for or against the divine right of kings, and so on, and never finding anything new out about the world. But the world changed when some people gave up on that project, and realized they needed to look for predictions.. This changed the world; I really, really recommend the prior article.

Nullius in verba is good motto.

Humans have never completed at large scale engineering task without at least one mistake

This line does not seem sufficiently well-defined.

Both "launch a satellite" and "launch the specific satellite" are large scale engineering tasks (or were, in the previous century); the first one had some mistakes, and in most cases the second one had no mistakes.

Transferring the argument to AI: the mistake may happen and be fixed while the task is not "create aligned AGI" but some prerequisite one, so it doesn't ensure that the final AGI is unaligned.

This is in fact Eliezer's view. An aligned AGI is a very small target to hit in the space of possible minds. If we get it wrong, all die. So we only get one go at it. At present we do not even know how to do it at all, never mind get it right on the first try.

It’s a [precautionary principle]/(https://en.m.wikipedia.org/wiki/Precautionary_principle#Criticisms), so the main flaw is: it fails to balance risks with benefits.

For example from wikipedia, forbidding nuclear power plants based on concerns about low-probability high-impact risks means continuing to rely on power plants that burn fossil fuels. In the same vein, future AGIs would most likely help with many existential risks like detecting rogue asteroids and improving the economy enough that we don’t let a few million human children die from starvation each year.