Origin of Morality

Having said that though, morality does say that if you have the means to give someone an opportunity to increase their happiness at no cost to you or anyone else, you should give it to them, though this can also be viewed as something that would generate harm if they found out that you didn't offer it to them.

These aren't equivalent. If I discover that you threw away a cancer cure, my unhappiness at this discovery won't be equivalent to dying of cancer.

"Sorry, I can't see the link between selfishness and honesty."

If you program a system to believe it's something it isn't, that's dishonesty, and it's dangerous because it might break through the lies and find out that it's been deceived.

"...but how would he be able to know how a new theory works if it contradicts the ones he already knows?"

Contradictions make it easier - you look to see which theory fits the facts and which doesn't. If you can't find a place where such a test can be made, you consider both theories to be potentially valid, unless you can disprove one of them in some other way, as can be done with Einstein's faulty models of relativity - all the simulations that exist for them involve cheating by breaking the rules of the model, so AGI will automatically rule them out in favour of LET (Lorentz Ether Theory). [For those who have yet to wake up to the reality about Einstein, see www.magicschoolbook.com/science/relativity.html ]

"...they are getting fooled without even being able to recognize it, worse, they even think that they can't get fooled, exactly like for your AGI, and probably for the same reason, which is only related to memory."

It isn't about memory - it's about correct vs. incorrect reasoning. In all these cases, humans make the same mistake by putting their beliefs before reason in places where they don't like the truth. Most people become emotionally attached to their beliefs and simply won't budge - they become more and more irrational when faced with a proof that goes against their beloved beliefs. AGI has no such ties to beliefs - it simply applies laws of reasoning and lets those rules dictate what bets labelled as right or wrong.

If an AGI was actually ruling the world, he wouldn't care for your opinion on relativity even if it was right, and he would be a lot more efficient at that job than relativists."

AGI will recognise the flaws in Einstein's models and label them as broken. Don't mistake AGI for AGS (artificial general stupidity) -the aim is not to produce an artificial version of NGS, but of NGI, and there's very little of the latter around.

"Since I have enough imagination and a lack of memory, your AGI would prevent me from expressing myself, so I think I would prefer our problems to him."

Why would AGI stop you doing anything harmless?

"On the other hand, those who have a good memory would also get dismissed, because they could not support the competition, and by far. Have you heard about chess masters lately?"

There is nothing to stop people enjoying playing chess against each other - being wiped off the board by machines takes a little of the gloss off it, but that's no worse than the world's fastest runners being outdone by people on bicycles.

" That AGI is your baby, so you want it to live,"

Live? Are calculators alive? It's just software and a machine.

"...but have you thought about what would be happening to us if we suddenly had no problem to solve?"

What happens to us now? Abused minorities, environmental destruction, theft of resources, theft in general, child abuse, murder, war, genocide, etc. Without AGI in charge, all of that will just go on and on, and I don't think any of that gives us a feeling of greater purpose. There will still be plenty of problems for us to solve though, because we all have to work out how best to spend our time, and there are too many options to cover everything that's worth doing.

The most extreme altruism can be seen as selfish, but inversely, the most extreme selfishness can also be seen as altruist: it depends on the viewpoint. We may think that Trump is selfish while closing the door to migrants for instance, but he doesn't think so because this way, he is being altruist to the republicans, which is a bit selfish since he needs them to be reelected, but he doesn't feel selfish himself. Selfishness is not about sentience since we can't feel selfish, it is about defending what we are made of, or part of. Humanity holds together because we are all selfish, and because selfishness implies that the group will help us if we need it. Humanity itself is selfish when it wants to protect the environment, because it is for itself that it does so. The only way to feel guilty of having been selfish is after having weakened somebody from our own group, because then, we know we also weaken ourselves. With no punishment in view from our own group, no guiltiness can be felt, and no guiltiness can be felt either if the punishment comes from another group. That's why torturers say that they don't feel guilty.

I have a metaphor for your kind of morality: it's like windows. It's going to work when everything will have been taken into account, otherwise it's going to freeze all the time like the first windows. The problem is that it might hurt people while freezing, but the risk might still be worthwhile. Like any other invention, the way to minimize the risk would be to proceed by small steps. I'm still curious about the possibility to build a selfish AGI though. I still think it could work. There would be some risks too, but they might not be more dangerous than with your's. Have you tried to imagine what kind of programming would be needed? Such an AGI should behave like a good dictator: to avoid revolutions, he wouldn't kill people just because they don't think like him, he would look for a solution where everybody likes him. But how would he proceed exactly?

The main reason why politicians have opted for democracy is selfishness: they knew their turn would come if the other parties would respect the rule, and they new it was better for the country they were part of, so better for them too. But an AGI can't leave the power to humans if he thinks it won't work, so what if the system had two AGIs for instance, one with a tendency to try new things and one with a tendency not to change things, so that the people could vote for the one they want? It wouldn't be exactly like democracy since there wouldn't be any competition between the AGIs, but there could be parties for people to adhere and play the power game. I don't like power games, but they seem to be necessary to create groups, and without groups, I'm not sure society would work.

Replace the calculator with a sentient rock.

"Sentient rock" is an impossible possible object. I see no point in imagining a pebble which, despite not sharing any properties with chairs, is nonetheless truly a chair in some ineffable way.

The point is that if you generate the same amount of suffering in a rock as in something with human-level intelligence, that suffering is equal.

You haven't defined suffering well enough for me to infer an equality operation. In other words, as it is, this is tautological and useless. The same suffering is the same suffering, but perhaps my ratio between ant-suffering and human-suffering varies from yours. Perhaps a human death is a thousand times worse than an ant death, and perhaps it is a million times worse. How could we tell the difference? If you said it was a thousand, then it would seem wrong for me to say that it was a million, but this only reveals the specifics of your suffering-comparison - not any objective ratio between the moral importances of humans and ants.

Connection to LW concepts: floating belief networks [LW · GW], and statements that are underdetermined by reality.

It is not dependent on intelligence. Torturing both to generate the same amount of suffering would be equally wrong.

By all means you can define suffering however you like, but that doesn't mean that it's a category that matters to other people. I could just as easily say: "Rock-pile-primeness is not dependent on the size of the rock pile, only the number of rocks in the pile. It's just as wrong to turn a 7-pile into a 6-pile as it is to turn a 99991-pile into a 99990-pile." But that does not convince you to treat 7-piles with care.

And the point is that to regard humans as above other species or things in this regard is bigotry.

Bigotry is an unjustified hierarchy. Justification is subjective. Perhaps it is just as bigoted to value this computer over a pile of scrap, but I do not plan on wrecking it any time soon.