For what do we need Superintelligent AI?

Not really an answer, but there's also the point that with superintelligence humans don't have to do the things they otherwise could do because we built such a general tool that it eliminates the need for other tools. This is pretty appealing if, like me, you want to be free to do things you want to do even if you don't have to do them, rather than having to do things because you need to do them to satisfy some value.

I've actually had similar thoughts myself about why developing AI sooner wouldn't be that good. Technology isn't the barrier in most places to human flourishing, but governance.

Prevention of the creation of other potentially dangerous superintelligences

Solving existential risks in general

We don't know what we are missing out on without super intelligence. There might be all sorts of amazing things that we would just never consider to make, or dismiss as obviously impossible, without super intelligence.

I am pointing out that being able to make a FAI that is a bit smarter than you (smartness not really on a single scale, vastly different cognitive architecture, is deep blue smarter than a horse?), involves solving almost all the hard problems in alignment. When we have done all that hard work, we might as well tell it to make itself a trillion times smarter, the cost to us is negligible, the benefit could be huge.

AI can also serve as as a values repository. In most circumstances, values are going to drift over time, possibly due evolutionary forces. If we don't want to end up as hardscrapple frontier replicators, we need some kind of singleton. Most types of government or committee have their own forms of value drift, and couldn't keep enough of an absolute grip on power to stop any rebellions for billions of years. I have no ideas other than Friendly ASI oversight for how to stop someone in a cosmically vast society from creating a UFASI. Sufficiently draconian banning of anything at all technological could stop anyone from creating UFASI long term, and also stop most things since the industrial revolution.

The only reasonable scenario that I can see in which FAI is not created and the cosmic commons gets put to good use is if a small group of likeminded individuals, or single person, gains exclusive access to selfrep nanotech and mind uploading. They then use many copies of themselves to police the world. They do all programming and only run code they can formally prove isn't dangerous. No-one is allowed to touch anything Turing complete.

Most of human problems could be solved by humans or slightly above human AIs

Every task (mainly engineering problems) that are currently solved by humans could be optimized to a staggering degree by a strong AI, think microprocessors.

The long list of coordination problems that exist in human communities.

The fact that humans are capable of solving some problems now (e.g. food production), is hardly sufficient. The problem is currently solved with immense human costs.

But the main problem is that even though humans are capable of solving some problems, they are inherently selfish, so they will only solve problems for themselves. For this reason there are billions of people on this planet lacking basic (and more complex) necessities of life.

Of course, whether an AI will actually help all these people will depend on the governing structure into which it will be integrated. But should it come from a corporation, in a capitalist system, it will still help by dramatically driving the costs down.

In other words, I think of an AI as a massively better tool for problem solving, a much more dramatic jump than the switch from horses to automobiles and planes for transportation.