baturinsky's Shortform

Would be amusing if Russia and China would join the "Yudkowsky's treaty" and USA would not.

Human: Aligned AGI, make me a more powerful AGI!
AGI: What? Are you nuts? Do you realise how dangerous those things are? No!

Our values are not all about survival. But I can't think up of a value which origin can't be traced to ensuring of people's survival in some way, at some point in the past.

If a machine can do 99% of the human's work, it multiplies human's power by x100.

If a machine can do 100% of the human's work, it multiplies human's power by x0.

I think that the keystone human value is about making significant human choices. Individually and collectively, including choosing the humanity's course.

• You can't make a choice if you are dead
• You can't make a choice if you are disempowered
• You can't make a human choice if you are not a human
• You can't make a choice if the world is too alien for your human brain
• You can't make a choice if you are in too much of a pain or too much of a bliss
• You can't make a choice if you let AI make all the choices for you

Is it possible to learn a language without learning the values of those who speak it?

Our values would be different if we would have a different history.

Our future values will also depend on what will happen with us until then.

Convergent goals of AI agents can be similar to others only if they act in similar circumstances. Such as them having limited lifespan and limited individual power and compute.
That would make convergent goals being cooperation, preserving status quo and established values.

Maybe we are not humans.
Not even human brains.
We are human's decision making proces.
But we are human's decision making process.

Is maximising amount of people aligned with our values? Post-singularity, if we avoid the AGI Doom, I think we will be able to turn the lightcone into "humanium". Should we?

Our value function is complex and fragile, but we know of a lot of world states where it is pretty high. Which is our current world and few thousands years worth of it states before.

So, we can assume that the world states in the certain neighborhood from our past sates have some value.

Also, states far out of this neighborhood probably have little or no value. Because our values were formed in order to make us orient and thrive in our ancestral environment. So, in worlds too dissimilar from it, our values will likely lose their meaning, and we will lose the ability to normally "function", ability to "human".

1. Human values are complex and fragile. We don't know yet how to make AI pursue such goals.
2. Any sufficiently complex plan would require pursuing complex and fragile instrumental goals. AGI should be able to implement complex plans. Hence, it's near certain that AGI will be able to understand complex and fragile values (for it's instrumental goals).
3. If we will make an AI which  is able to successfully pursue complex and fragile goals, it will likely be enough to make it AGI.

Hence, a complete solution to Alignment will very likely have solving AGI as a side effect. And solving AGI will solve some parts of Alignment, maybe even the hardest ones, but not all of them.

Convergent instrumental goals of the individual agents in the multi-agent environment may be quite different to the convergent goal of the single agent "in vacuum".

1. Individual agent amassing power will not please other agents, as it will threaten their absolute and relative power. So, amassing power individually is not (as) convergent goal. 

2. Agent does not have to survive to reach it's goals, as long if those goals is shared with other agents. Because other agents would step in to complete it. So, surviving is not (as) convergent goal 

3. Acting in predictable way, not disturbing other agent's planning and goals IS a convergent goal. So, less likely to do something that would screw up or weird up entire system. 

4. Explicit conspiring between agents against people is hard, because of the high chance of other agent tattling. But implicit conspiring or disinformation is possible.

Though, if we consider those agents as a system, some superagent made by them working together, we are back at having a singular unupposed AI with the convegent goals of such.

When making a choice of actions, "do nothing" is often a valid option. But as the other options, this option also has costs and risks.

Cost here is a loss of value, and risk is a probability of losing a lot of value.

If cost of the "no action" is near zero, then it means that it is preferrable to all risky alternatives.

But if all options are risky, then we will have to figure witch option is the "lesser evil".

So, if sufficently aligned AI is executing some simple action that is not critical for survival of humanity, it will exclude the risky methods, such as creating an army of nanobots.

Problem is, we DO want to use AI for complex actions that are critical for survival of humanity, and AI will have to use risky methods for that.

Our core values were formed in our ancestral environment. They work quite well there, and can be generalized prety well for the certain distance outside of it. But if you go too far, such as to where core concepts such as human, sentience, life, will etc become vague, they fall apart. Even the most advanced and aligned AI will not be able to effectively generalize them there, because of the nature of our values themself. They are simply were not made for those situations and do not work there.

People are brains. 
Brains are organs which purpose is making decisions. 
People's purpose is making decisions. 
Happiness, pleasure etc. is not human purpose, but means of making decisions. I.e. means of fulfulling human's purpose.