LessWrong 2.0 Reader

I believe this echos out my thoughts perfectly, I might quote it in full if I ever do get around to reviving that draft :))

The bit about "perfect" as not giving slack for development, I think, could be used even in the single individual scenario if you assume any given "ideal" action as lower chance of discovering something potential useful than a "mistake". I.e. adding:

• Actions have unintended and unknown consequences that reveal an unknown landscape of possibilities
• Actions have some % chance of being "optimal", but one can never be 100% certain they are so, just weigh them as having a higher or lower chance of being so
• "optimal" is a shifting goal-post and every action will potentially shift it

I think the "tinkerer" example is interesting, but even that assumes "optimal" to be goals dictates by natural-selection, and as per the sperm bank example, nobody care about those goals per-say, their subconscious machinery and the society they live in "cares". So maybe a complementary individual-focused example would be a world in which inactivity is equivalent to the happiest state possible (i.e. optimal for the individual) or at least some form of action that does not lend to itself or something similar to it's host being propagated through time.

Can you go into more detail for #4?

This, I assume, you'd base on a "hasn't happened before, no other animal or thing similar to us is doing it as far as we know, so it's improbable we will be able to do it" type assumption? Or something different?

I don't necessarily think you have to take the "AI" example for the point to make sense though.

I think "reasoning your way to a distant inference", as a human, is probably a far less controversial example that could be used here. In that most people here seem to assume there are ways to make distant inferences (e.g. about the capabilities of computers in the far off future), which historically seems fairly far fetched, it almost never happens when it does it is celebrated, but the success rate seems fairly small and there doesn't seem to be a clear formula for it that works.

This example by Tom Scott is excellent imo

It is a feature of the AI system but it's very important to first discover proper alignment before discovering AGI. If you randomly go about making discoveries it's more likely that you end up discovering AGI and ending humanity before discovering proper alignment.

The point of focusing on AI Alignment isn't that it's an efficient way to discover new technology but that it's a way that makes it less likely that humanity will develop technology that destroys humanity. 

Is "proper alignment" not a feature of an AI system, i.e. something that has to be /invented/discovered/built/?

This sound like semantics vis-a-vis the potential stance I was referring to above. 

Kenny, do you have a reason to think that housing construction and rental is a situation where you would expect good outcomes from the government running it?  

You might get a bunch of these, and the ones I have seen were somewhat shoddily constructed.  

Consider this.  There is a 'model' free market solution, with Tokyo a model example.  In 3 sentences:

       The national government handles the rules for permitting, and any developer with plans that meet the codes, which allow extremely high density, will get a permit.  Neighbors and local city governments get very little say in large areas, so replacing older homes with new can't be blocked.  The outcome of this policy is that the housing in Tokyo is inexpensive, high density, and most of it is recently constructed.  Here's a video on it.

I later found out that it contained coal tar base dies and cyclamates and saccharine.

How did you find out?

What is efficient on one side of the coin is 0 slack on the other.

Aspects which are hard to put into numbers get short end of the stick if numbers try to go up.

If someone optimises for some measures that are proxies for success the quality of the proxiness of those numbers goes down. An airplane that uses many instruments is likely to be safer than one that uses less instruments. For a complex system a good solution is unlikely to be optimal if the dimensionality of the problem was artifically lowered. So exact solutions can be warning signs that maybe important aspects were assumed away to make the problem overtly tractable.