Aleksi Liimatainen's Shortform

My guess: goal specification is used to disambiguate communication rather than for optimization. The agents have some relatively fixed modes of "what they do" and the principals will is "only" used to pick one of those. If the peak of the principals will falls "inbetween" options, so sad, it will be rounded off to the nearest "supported" option. With this agents make a sensible action even if used by multiple principles which would like to "customise" its behaviour in weird ways, each off course in a different manner. Clunky but uncorrupted. You don't have to watch out what you wish for you don't get what you wish for.

(I upvote you writing this as a main-page question or post.)

(copied from my tweet)

If you raise an ape as a child, you don't get a child. You just get an ape.

I suspect that you're right, but that this isn't as comforting as might be expected in the event of AGI.

The flexibility and capacity of human brains to learn skills and acquire knowledge has allowed us to shorten the timelines for developing skills and knowledge for members of our species by many orders of magnitude. Instincts honed by evolution take a long time to develop further. Knowledge acquired from literacy and mass distribution takes a lot less, but still limited by the couple of decades it takes for children to start with a blank slate and learn through to adulthood so that they (hopefully) have enough skills to function in society. Some few of them can learn new things to add to and refine the collection. Then they die, only a few decades after building up to basic competence, and any skills and knowledge not explicitly passed on is lost. Still, by the standards of evolution this is lightning fast.

The development of AGI promises that some entities will be capable of learning much faster than we do, likely in months at most instead of decades. Later versions may be able to learn much faster still. They will be able to be copied in a trained state and the copies learning further from there without inevitable death imposing an upper bound.

This will close a completely new positive feedback loop, even without considering recursive self-improvement of underlying software or hardware. We should expect that closing additional - and faster - feedback loops in knowledge and skill acquisition to lead to unpredictable capabilities, beyond anything previously seen. The likelihood of additional improvements in underlying software and hardware just add two more new positive feedback loops in capability gain.

This may be a case of a threshold effect.  I'm not sure what your definition of "strongly superintelligent relative to" is, but unlocking language and social networking and planning is definitely a superpower.  And one that would be hard to predict until it started to appear.

It's quite possible that the next jump in ability to optimize the future by application of models (intelligence) is small by some measures, but large in impact due to some capability we currently undervalue.  

Humans in modern societies cannot be disconnected from financial pressures (or, in some communist experiences, pressures of material needs intermediated by not-quite-financial mechanisms).  Pure insulation from such is unlikely to be possible at any scale, and probably not desirable for most things.

The common concern here is not "connection with reality", but "short-timeframe focus".  With some amount of "ability to take long-shot bets that probably won't pay off".  I know of no way to make individuals or organizations resistant to the pressures of having to eat and enjoying leisure activities which require other humans to be paid.  

You can seek out and attract members who are naturally more aligned with your mission than with their personal well-being, but that doesn't get you to 100%.  Probably enough for most purposes.  You can attract sponsors and patrons who've overindexed on their personal wealth and are willing to share some of it with you, in pursuit of your mission. 

In all cases, if the mission becomes unpopular or if those sacrificing to further it stop believing it's worth your expense, it'll stop.  This can take some time, and isn't terribly well-measured, so it often affects topics rather than organizations or individuals.  That sucks, but over time it definitely happens.