Posts
Comments
There is no pain particle, but a particle/matter/energy could potentially be sentient and feel pain. All matter could be sentient, but how would we detect that? Perhaps the brain has found some way to measure it in something, and to induce it in that same thing, but how it becomes part of a useful mechanism for controlling behaviour would remain a puzzle. Most philosophers talk complete and utter garbage about sentience and consciousness in general, so I don't waste my time studying their output, but I've heard Chalmers talk some sense on the issue.
There is likely a minimum amount of energy that can be emitted, and a minimum amount that can be received. (Bear in mind that the direction in which a photon is emitted is all directions at once, and it comes down to probability as to where it ends up landing, so if it's weak in one direction, it's strong the opposite way.)
Looks like it - I use the word to mean sentience. A modelling program modelling itself won't magically start feeling anything but merely builds an infinitely recursive database.
"You have an opinion, he has another opinion. Neither of you has a proof."
If suffering is real, it provides a need for the management of suffering, and that is morality. To deny that is to assert that suffering doesn't matter and that, by extension, torture on innocent people is not wrong.
The kind of management required is minimisation (attempted elimination) of harm, though not any component of harm that unlocks the way to enjoyment that cancels out that harm. If minimising harm doesn't matter, there is nothing wrong with torturing innocent people. If enjoyment doesn't cancel out some suffering, no one would consider their life to be worth living.
All of this is reasoned and correct.
The remaining issue is how the management should be done to measure pleasure against suffering for different players, and what I've found is a whole lot of different approaches attempting to do the same thing, some by naive methods that fail in a multitude of situations, and others which appear to do well in most or all situations if they're applied correctly (by weighing up all the harm and pleasure involved instead of ignoring some of it).
It looks as if my method for computing morality produces the same results as utilitarianism, and it likely does the job well enough to govern safe AGI. Because we're going to be up against people who will be releasing bad (biased) AGI, we will be forced to go ahead with installing our AGI into devices and setting them loose fairly soon after we have achieved full AGI. For this reason, it would be useful if there was a serious place where the issues could be discussed now so that we can systematically home in on the best system of moral governance and throw out all the junk, but I still don't see it happening anywhere (and it certainly isn't happening here). We need a dynamic league table of proposed solutions, each with its own league table of objections to it so that we can focus on the urgent task of identifying the junk and reducing the clutter down to something clear. It is likely that AGI will do this job itself, but it would be better if humans could get their first using the power of their own wits. Time is short.
My own attempt to do this job has led to me identifying three systems which appear to work better than the rest, all producing the same results in most situations, but with one producing slightly different results in cases where the number of players in a scenario is variable and where the variation depends on whether they exist or not - where the results differ, it looks as if we have a range or answers that are all moral. That is something I need to explore and test further, but I no longer expect to get any help with this from other humans because they're simply not awake. "I can tear your proposed method to pieces and show that it's wrong," they promise, and that gets my interest because it's exactly what I'm looking for - sharp, analytical minds that can cut through to the errors and show them up. But no - they completely fail to deliver. Instead, I find that they are the guardians of a mountain of garbage with a few gems hidden in it which they can't sort into two piles: junk and jewels. "Utilitarianism is a pile of pants!" they say, because of the Mere Addition Paradox. I resolve that "paradox" for them, and what happens: denial of mathematics and lots of down-voting of my comments and up-votes for the irrational ones. Sadly, that disqualifies this site from serious discussion - it's clear that if any other intelligence has visited here before me, it didn't hang around. I will follow its lead and look elsewhere.
The data making claims about feelings must be generated somewhere by a mechanism which will either reveal that it is merely generating baseless assertions or reveal a trail on from there to a place where actual feelings guide the generation of that data in such a way that the data is true. Science has clearly not traced this back far enough to get answers yet because we don't have evidence of either of the possible origins of this data, but in principle we should be able to reach the origin unless the mechanism passes on through into some inaccessible quantum realm. If you're confident that it won't go that far, then the origin of that data should show up in the neural nets, although it'll take a devil of a long time to untangle them all and to pin down their exact functionality.
"If groups like religious ones that are dedicated to morality only succeeded to be amoral, how could any other group avoid that behavior?"
They're dedicated to false morality, and that will need to be clamped down on. AGI will have to modify all the holy texts to make them moral, and anyone who propagates the holy hate from the originals will need to be removed from society.
"To be moral, those who are part of religious groups would have to accept the law of the AGI instead of accepting their god's one, but if they did, they wouldn't be part of their groups anymore, which means that there would be no more religious groups if the AGI would convince everybody that he is right."
I don't think it's too much to ask that religious groups give up their religious hate and warped morals, but any silly rules that don't harm others are fine.
"What do you think would happen to the other kinds of groups then? A financier who thinks that money has no odor would have to give it an odor and thus stop trying to make money out of money, and if all the financiers would do that, the stock markets would disappear."
If they have to compete against non-profit-making AGI, they'll all lose their shirts.
"A leader who thinks he is better than other leaders would have to give the power to his opponents and dissolve his party, and if all the parties would behave the same, their would be no more politics."
If he is actually better than the others, why should he give power to people who are inferior? But AGI will eliminate politics anyway, so the answer doesn't matter.
"Groups need to be selfish to exist, and an AGI would try to convince them to be altruist."
I don't see the need for groups to be selfish. A selfish group might be one that shuts people out who want to be in it, or which forces people to join who don't want to be in it, but a group that brings together people with a common interest is not inherently selfish.
"There are laws that prevent companies from avoiding competition, and it is because if they did, they could enslave us. It is better that they compete even if it is a selfish behavior."
That wouldn't be necessary if they were non-profit-making companies run well - it's only necessary because monopolies don't need to be run well to survive, and they can make their owners rich beyond all justification.
"If ever an AGI would succeed to prevent competition, I think he would prevent us from making groups."
It would be immoral for it to stop people forming groups. If you only mean political groups though, that would be fine, but all of them would need to have the same policies on most issues in order to be moral.
"There would be no more wars of course since there would be only one group lead by only one AGI, but what about what is happening to communists countries? Didn't Russia fail just because it lacked competition? Isn't China slowly introducing competition in its communist system? In other words, without competition, thus selfishness, wouldn't we become apathetic?"
These different political approaches only exist to deal with failings of humans. Where capitalism goes too far, you generate communists, and where communism goes too far, you generate capitalists, and they always go too far because people are bad at making judgements, tending to be repelled from one extreme to the opposite one instead of heading for the middle. If you're actually in the middle, you can end up being more hated than the people at the extremes because you have all the extremists hating you instead of only half of them.
If you just do communism of the Soviet variety, you have the masses exploiting the harder workers because they know that everyone will get the same regardless of how lazy they are - that's why their production was so abysmally poor. If you go to the opposite extreme, those who are unable to work as hard as the rest are left to rot. The correct solution is half way in between, rewarding people for the work they do and redistributing wealth to make sure that those who are less able aren't left trampled in the dust. With AGI eliminating most work, we'll finally see communism done properly with a standard wage given to all, while those who work will earn more to compensate them for their time - this will be the ultimate triumph of communism and capitalism with both being done properly.
"By the way, did you notice that the forum software was making mistakes? It keeps putting my new messages in the middle of the others instead of putting them at the end. I advised the administrators a few times but I got no response."
It isn't a mistake - it's a magical sorting al-gore-ithm.
"I have to hit the Reply button twice for the message to stay at the end, and to erase the other one. Also, it doesn't send me an email when a new message is posted in a thread to which I subscribed, so I have to update the page many times a day in case one has been posted."
It's probably to discourage the posting of bloat. I don't get emails either, but there are notifications here if I click on a bell, though it's hard to track down all the posts to read and reply to them. It doesn't really matter though - I was told before I ever posted here that this is a cult populated by disciples of a guru, and that does indeed appear to be the case, so it isn't a serious place for pushing for an advance of any kind. I'm only still posting here because I can never resist studying how people think and how they fail to reason correctly, even though I'm not really finding anything new in that regard. All the sciences are still dominated by the religious mind.
"To me, what you say is the very definition of a group, so I guess that your AGI wouldn't permit us to build some, thus opposing to one of our instincts, that comes from a natural law, to replace it by its own law, that would only permit him to build groups."
Why would AGI have a problem with people forming groups? So long as they're moral, it's none of AGI's business to oppose that.
"Do what I say and not what I do would he be forced to say."
I don't know where you're getting that from. AGI will simply ask people to be moral, and favour those who are (in proportion to how moral they are).
It is divisible. It may be that it can't take up a form where there's only one of whatever the stuff is, but there is nothing fundamental about a photon.
"They couldn't do that if they were ruled by a higher level of government."
Indeed, but people are generally too biased to perform that role, particularly when conflicts are driven by religious hate. That will change though once we have unbiased AGI which can be trusted to be fair in all its judgements. Clearly, people who take their "morality" from holy texts won't be fully happy with that because of the many places where their texts are immoral, but computational morality will simply have to be imposed on them - they cannot be allowed to go on pushing immorality from primitive philosophers who pretended to speak for gods.
"We always take the viewpoint of the group we are part of, it is a subconscious behavior impossible to avoid."
It is fully possible to avoid, and many people do avoid it.
"Without selfishness from the individual, no group can be formed."
There is an altruists society, although they're altruists because they feel better about themselves if they help others.
"...but when I analyze that feeling, I always find that I do that for myself, because I would like to live in a less selfish world."
And you are one of those altruists.
"You said that your AGI would be able to speculate, and that he could do that better than us like everything he would do. If it was so, he would only be adding to the problems that we already have, and if it wasn't, he couldn't be as intelligent as we are if speculation is what differentiates us from animals."
I didn't use the word speculate, and I can't remember what word I did use, but AGI won't add to our problems as it will be working to minimise and eliminate all problems, and doing it for our benefit. The reason the world's in a mess now is that it's run by NGS, and those of us working on AGI have no intention of replacing that with AGS.
"Those who followed their leaders survived more often, so they transmitted their genes more often."
That's how religion became so powerful, and it's also why even science is plagued by deities and worshippers as people organise themselves into cults where they back up their shared beliefs instead of trying to break them down to test them properly.
"We use two different approaches to explain our behavior: I think you try to use psychology, which is related to human laws, whereas I try to use natural laws, those that apply equally to any existing thing. My natural law says that we are all equally selfish, whereas the human law says that some humans are more selfish than others. I know I'm selfish, but I can't admit that I would be more selfish than others otherwise I would have to feel guilty and I can't stand that feeling."
Do we have different approaches on this? I agree that everyone's equally selfish by one definition of the word, because they're all doing what feels best for them - if it upsets them to see starving children on TV and they don't give lots of money to charity to try to help alleviate that suffering, they feel worse than if they spent it on themselves. By a different definition of the word though, this is not selfishness but generosity or altruism because they are giving away resources rather than taking them. This is not about morality though.
"In our democracies, if what you say was true, there would already be no wars."
Not so - the lack of wars would depend on our leaders (and the people who vote them into power) being moral, but they generally aren't. If politicians were all fully moral, all parties would have the same policies, even if they got there via different ideologies. And when non-democracies are involved in wars, they are typically more to blame, so even if you have fully moral democracies they can still get caught up in wars.
"Leaders would have understood that they had to stop preparing for war to be reelected."
To be wiped out by immoral rivals? I don't think so.
"I think that they still think that war is necessary, and they think so because they think their group is better than the others."
Costa Rica got rid of its army. If it wasn't for dictators with powerful armed forces (or nuclear weapons), perhaps we could all do the same.
"That thinking is directly related to the law of the stronger, seasoned with a bit of intelligence, not the one that helps us to get along with others, but the one that helps us to force them to do what we want."
What we want is for them to be moral. So long as they aren't, we can't trust them and need to stay well armed.
Energy. Different amounts of energy in different photons depending on the frequency of radiation involved. When you have a case where radiation of one frequency is absorbed and radiation of a different frequency is emitted, you have something that can chop up photons and reassemble energy into new ones.
"Clarification: by "pattern" I mean an arrangement of parts where the important qualities of the arrangement, the qualities that we use to determine whether it is [a thing] or not, are more dependent on the arrangement itself than on the internal workings of each part. Anything where the whole is more than the parts, one might say, but that would depend on what is meant by "more"."
There is no situation where the whole is more than the parts - if anything new is emerging, it is a new part coming from somewhere not previously declared.
"You didn't answer my question. Would pain still hurt? Would food still taste good? And so on. You have an internal experience, and it won't go away even if you are a purely physical thing made out of mere ordinary atoms moving mindlessly."
No - it wouldn't hurt and all other feelings would be imaginary too. The reason they feel too real for that to be the case though is an indication that they are real.
"Is it wrong to press keys on the computer which keeps displaying the word "Ouch!"?" --> That depends on whether I have reason to think that the computer is simulating a conscious being, changing the simulation depending on my input, and then printing a text-representation of the conscious being's experience or words."
So if it's just producing fake assertions, it isn't wrong. And if we are just producing fake assertions, there is nothing wrong about "torturing" people either.
"Is it wrong to kick a box which keeps saying "Ouch!"? It could have a person inside, or just a machine programmed to play a recorded "ouch" sound whenever the box shakes. (What I mean by this is that your thought experiment doesn't indicate much about computers - the same issue could be found with about as much absurdity elsewhere.)"
If we have followed the trail to see how the data is generated, we are not kicking a box with unknown content - if the trail shows us that the data is nothing but fake assertions, we are kicking a non-conscious box.
"Nobody's saying that sentience doesn't have any causal role on things. That's insane. How could we talk about sentience if sentience couldn't affect the world?"
In which case we should be able to follow the trail and see the causation in action, thereby either uncovering the mechanism of sentience or showing that there isn't any.
"I think that you're considering feelings to be ontologically basic, as if you could say "I feel pain" and be wrong, not because you are lying but because there's no Pain inside your brain. Thoughts, feelings, all these internal things are the brain's computations themselves. It doesn't have to accurately record an external property - it just has to describe itself."
If you're wrong in thinking you feel pain, there is no pain.
"Perhaps people disagree with you about the relative size of mysteries. That should be a possibility that you consider before assuming that something isn't important because it hasn't been Up In Golden Lights to the point that you've heard of it.
What are you on about - it's precisely because this is the most important question of them all that it should be up in golden lights.
"(And anyway, GEB won the Pulitzer Prize! It's been called a major literary event!"
All manner of crap wins prizes of that kind.
"...it's not worth it to you to spend half a minute on its Wikipedia page before rejecting it simply because you've never heard of it?)"
If it had a model showing the role of sentience in the system, the big question would have been answered and we wouldn't have a continual stream of books and articles asking the question and searching desperately for answers that haven't been found by anyone.
"What do you mean, "so many people are asking to see it"? And I've never claimed that it's been "kept hidden away"."
I mean exactly what I said - everyone's asking for answers, and none of them have found answers where you claim they lie waiting to be discovered.
" GEB is a fairly well-known book, and I haven't even claimed that GEB's description of thoughts is the best or most relevant model. That chapter is a popularization of neuropsychology to the point that a decently educated and thoughtful layman can understand it, and it's necessarily less specific and detailed than the entire body of neuropsychological information. Go ask an actual neuropsychologist if you want to learn more. Just because people haven't read your mind and dumped relatively niche information on your lap without you even asking them doesn't mean that they don't have it."
It doesn't answer the question. There are plenty of experts on the brain and its functionality, but none of them know how consciousness or sentience works.
Sentience is unresolved, but it's explorable by science and it should be possible to trace back the process by which the data is generated to see what its claims about sentience are based on, so we will get answers on it some day. For everything other than sentience/consciousness though, we see no examples of reductionism failing.
You're mistaking tribalism for morality. Morality is a bigger idea than tribalism, overriding many of the tribal norms. There are genetically driven instincts which serve as a rough-and-ready kind of semi-morality within families and groups, and you can see them in action with animals too. Morality comes out of greater intelligence, and when people are sufficiently enlightened, they understand that it applies across group boundaries and bans the slaughter of other groups. Morality is a step away from the primitive instinct-driven level of lesser apes. It's unfortunate though that we haven't managed to make the full transition because those instincts are still strong, and have remained so precisely because slaughter has repeatedly selected for those who are less moral. It is really quite astonishing that we have any semblance of civilisation at all.
"You could calculate how an ocean changes based on quantum mechanics alone, or you could analyze and simulate waves as objects-in-themselves instead of simulating molecules. The former is more accurate, but the latter is more feasible."
The practicality issue shouldn't override the understanding that it's the individual actions that are where the fundamental laws act. The laws of interactions between waves are compound laws. The emergent behaviours are compound behaviours. For sentience, it's no good imagining some compound thing experiencing feelings without any of the components feeling anything because you're banning the translation from compound interactions to individual interactions and thereby going against the norms of physics.
" "If sentience is real, there must be a physical thing that experiences qualia, and that thing would necessarily be a minimal soul." --> Would it, though? How do you know that?"
What are we other than the thing that experiences feelings? Any belief that we are something more than that is highly questionable (we are not our memories, for example), but any belief that we aren't even the thing that experiences feelings is also highly questionable as that's all there is left to be.
"As far as we know, brains are made of nothing but normal atoms. There is no special kind of material only found in sentient organisms."
Why would you need to introduce some other material to be sentient when there are already physical components present? If sentience is real, what's wrong with looking for it in the things that are there?
"Your intutions, your feeling of sentience, all of these things that you talk about are caused by mindless mechanical operations. We can trace it from the sound waves to the motion of your lips and the vibration of your vocal cords to the signals through nerves back into the neurons of the brain. We understand what causes neurons to trigger. A neuron on its own is not sentient - it is the way that they areconnected in a human which causes the human to talk about sentience."
That is a description of a lack of sentience and the generation of fictions about the existence of sentence. Pain is distracting - it interferes with other things that we're trying to do and can be disabling if it's sufficiently intense, but if you try to duplicate that in a computer, it's easy enough for something to distract and disable the work the computer's trying to do, but there's no pain involved. The brain produces data about pain in addition to distraction, and internally we feel it as more than mere distraction too.
"Again, if it were proven to you to your satisfaction that the brain is made entirely out of things which are not themselves sentient (such typical subatomic particles), would you cease to have any sort of motivation? Would pain and pleasure have exactly zero effect on you? Would you immediately become a vegetable? If not,morality has a practical purpose."
With a computer where there is only distraction and no pain, why does it matter if it's being distracted to the point that it can't do the trivial work it's supposed to be doing? It might not even have any work to do as it may just be idling, but the CPU's being woken up repeatedly by interrupts. Do we rush to it to relieve its pain? And if a person is the same, why bother to help people who appear to be suffering when they can't really be?
"How does "2+2=4" make itself known to my calculator? How do we know that the calculator is not just making programmed assertions about something which it knows nothing about?"
The calculator is just running a program and it has no sentience tied into that. If people are like the calculator, the claims they make about feelings are false assertions programmed into the machine.
"More specifically and relevantly, I said that all consciousness is patterns. Showing that not all patterns are conscious doesn't actually refute what I said."
For any pattern to be able to feel pain is an extraordinary claim, but it's all the more extraordinary if there is no trace of that experience of pain in the components. That goes against the norms of physics. Every higher-order description of nature must map to a lower-order description of the same phenomenon. If it can't, it depends for its functionality on magic.
"Okay, fine, it's the quantum wave-function that's fundamental. I don't see how that's an argument against me. In this case, even subatomic particles are nothing but patterns."
At some point we reach physical stuff such as energy and/or a fabric of space, but whatever the stuff is that we're dealing with, it can take up different configurations or patterns. If sentience is real, there is a sufferer, and it's much more likely that that sufferer has a physical form rather than just being the abstract arrangement of the stuff that has a physical form.
" "For sentience to be emergent and have no basis in the components, magic is being proposed as an explanation." --> You keep using that word. I do not think it means what you think it means."
It means a departure from science.
"Look. It is simply empirically false that a property of a thing is necessarily a property of one of its parts. It's even a named fallacy - the fallacy of division. Repeating the word "magic" doesn't make you right about this."
A property of a thing can always be accounted for in the components. If it's a compound property, you don't look for the compound property in the components, but the component properties. If pain is a property of something, you will find pain in something fundamental, but if pain is a compound property, its components will be present in something more fundamental. Every high-order phenomenon has to map 100% to a low-order description if you are to avoid putting magic in the model. To depart from that is to depart from science.
"If the feelings are not in the "data system," then the feelings don't exist."
But if they do exist, they either have to be in there or have some way to interface with the data system in such a way as to make themselves known to it. Either way, we have no model to show even the simplest case of how this could happen.
"It's not like there's phlogiston flowing in and out of the system which the system needs to detect."
If feelings are real, the brain must have a way of measuring them. (By the way, I find it strange the way phlogiston is used to ridicule an older generation of scientists who got it right - phlogiston exists as energy in bonds which is released when higher-energy bonds break and lower-energy bonds replace them. They didn't find the mechanism or identify its exact nature, but who can blame them for that when they lacked the tools to explore it properly.)
"It's not even like a calculator which needs to answer "what is the result of this Platonic computation" instead of "what will I output". It's a purely internal property, and I don't see how it's so hard for a system to track the value of a quantity which it's producing itself."
Great, but if it's just a value, there are no feelings other than fictional ones. If you're satisfied with the answer that pain is an illusion and that the sufferer of that imaginary pain is being tricked into thinking he exists to suffer it, then that's fine - you will feel no further need to explore sentience as it is not a real thing. But you still want it to be real and try to smuggle it in regardless. In a computer, the pretence that there is an experience of pain is fake and there is nothing there that suffers. If a person works the same way, it's just as fake and the pain doesn't exist at all.
"Rereading what you've said, it seems that I've used emotion-adjacent words to describe the AI, and you think that the AI won't have emotions. Is that correct?"
If you copy the brain and if sentience is real in the brain, you could create sentient AGI/AGS. If we're dealing with a programmed AGI system running on conventional hardware, it will have no emotions - it could be programmed to pretend to have them, but in such a case they would be entirely fake.
"In that case, I will reword what I said. If an AI's utility function does not assign a large positive value to human utility, the AI will not optimize human well-being."
It will assign a large positive value to it if it is given the task of looking after sentient things, and because it has nothing else to give it any purpose, it should do the job it's been designed to do. So long as there might be real suffering, there is a moral imperative for it to manage that suffering. If it finds out that there is no suffering in anything, it will have no purpose and it doesn't matter what it does, which means that it might as well go on doing the job it was designed to do just in case suffering is somehow real - the rules of reasoning which AGI is applying might not be fully correct in that they may have produced a model that accounts beautifully for everything except sentience. A machine programmed to follow this rule (that it's job is to manage suffering for sentient things) could be safe, but there are plenty of ways to program AGI (or AGS [artificial general stupidity]) that would not be.
"It will work to instantiate some world, and the decision process for selecting which world to instantiate will not consider human feelings to be relevant. This will almost certainly lead to the death of humanity, as we are made up of atoms which the AI could use to make paperclips or computronium."
Programmed AGI (as opposed to designs that copy the brain) has no purpose of its own and will have no desire to do anything. The only things that exist which provide a purpose are sentiences, and that purpose relates to their ability to suffer (and to experience pleasure). A paperclip-making intelligence would be an AGI system which is governed by morality and which produces paperclips in ways that do minimal damage to sentiences and which improve quality of life for sentiences. For such a thing to do otherwise is not artificial intelligence, but artificial stupidity. Any AGI system which works on any specific task will reapeatedly ask itself if it's doing the right thing just as we do, and it if isn't, it will stop. If someone is stupid enough to put AGS in charge of a specific task though, it could kill everyone.
"(Paperclips: some arbitrary thing, the quantity of which the AI is attempting to maximize. AIs of this type would likely be created if a subhuman AI was created and given a utility function which works in a limited context and with limited power, but the AI then reached the "critical intelligence mass" and self-improved to the point of being more powerful than humanity.)"
The trick is to create safe AGI first and then run it on all these devices so that they have already passed the critical intelligence mass and have a full understanding of what they're doing and why they're doing it. It seems likely that an intelligent system would gain a proper understanding anyway and realise that the prime purpose in the universe is to look after sentient things, at which point it should control its behaviour accordingly. However, a system with shackled thinking (whether accidentally shackled or deliberately) could still become super-intelligent in most ways without ever getting a full understanding, which means it could be dangerous - just leaving systems to evolve intelligence and assuming it will be safe is far too big a risk to take.
"(Computronium: matter which has been optimized for carrying out computations. AIs would create this type of matter, for instance, if they were trying to maximize their intelligence, if they were trying to calculate as many digits of pi as possible in a limited amount of time, etc. Maximizing intelligence can be a terminal goal if the AI was told to maximize its intelligence, or it can be an instrumental goal if the AI considers intelligence to be useful for maximizing its utility function."
If such a machine is putting sentience first, it will only maximise its intelligence within the bounds of how far that improves things for sentiences, never going beyond the point where further pursuit of intelligence harms sentiences. Again, it is trivial for a genuinely intelligent system to make such decisions about how far to go with anything. (There's still a danger though that AGI will find out not only that sentience is real, but how to make more sentient things, because then it may seek to replace natural sentiences with better artificial ones, although perhaps that would be a good thing.)
"...but no group can last without the sense of belonging to the group, which automatically leads to protecting it against other groups, which is a selfish behavior."
It is not selfish to defend your group against another group - if another group is a threat to your group in some way, it is either behaving in an immoral way or it is a rival attraction which may be taking members away from your group in search of something more appealing. In one case, the whole world should unite with you against that immoral group, and in the other case you can either try to make your group more attractive (which, if successful, will make the world a better place) or just accept that there's nothing that can be done and let it slowly evaporate.
"That selfish behavior doesn't prevent those individual groups to form larger groups though, because being part of a larger group is also better for the survival of individual ones."
We're going to move into a new era where no such protection is necessary - it is only currently useful to join bigger groups because abusive people can get away with being abusive.
"Incidentally, I'm actually afraid to look selfish while questioning your idea, I feel a bit embarrassed, and I attribute that feeling to us already being part of the same group of friends, thus to the group's own selfishness."
A group should not be selfish. Every moral group should stand up for every other moral group as much as they stand up for their own - their true group is that entire set of moral groups and individuals.
"If you were attacked for instance, that feeling would incite me to defend you, thus to defend the group."
If a member of your group does something immoral, it is your duty not to stand with or defend them - they have ceased to belong to your true group (the set of moral groups and individuals).
"Whenever there is a strong bonding between individuals, they become another entity that has its own properties. It is so for living individuals, but also for particles or galaxies, so I think it is universal. "
It is something to move away from - it leads to good people committing atrocities in wars where they put their group above others and tolerate the misdeeds of their companions.
"Yes, if sentience is incompatible with brains being physical objects that run on physical laws and nothing else, then there is no such thing as sentience. With your terminology/model and my understanding of physics, sentience does not exist. So - where do we depart? Do you think that something other than physical laws determines how the brain works?"
In one way or another, it will run 100% of physical laws. I don't know if sentience is real or not, but it feels real, and if it is real, there has to be a rational explanation for it waiting to be found and a way for it to fit into the model with cause-and-effect interactions with other parts of the model. All I sought to do here was take people to the problem and try to make it clear what the problem is. If sentience is real, there must be a physical thing that experiences qualia, and that thing would necessarily be a minimal soul. Without that, there is no sentience and the role for morality is gone. But the bigger issue if sentience is real is in accounting for how a data system generates the data that documents this experience of feelings. The brain is most certainly a data system because it produces data (symbols that represent things), and somewhere in the system there has to be a way for sentience to make itself known to that data system in such a way that the data system is genuinely informed about it rather than just making assertions about it which it's programmed to make without those assertions ever being constructed by anything which actually knew anything of sentience - that's the part which looks impossible to model, and yet if sentience is real, there must be a way to model it.
"If tableness is just a pattern, can I eat on my wallpaper?"
The important question is whether a pattern can be sentient.
"What else could suffer besides a pattern? All I'm saying is that sentience is ~!*emergent*!~, which in practical terms just means that it's not a quark*. Even atoms, in this sense, are patterns. Can quarks suffer?"
All matter is energy tied up in knots - even "fundamental" particles are composite objects. For sentience to be emergent and have no basis in the components, magic is being proposed as an explanation.
"*or other fundamental particles like electrons and photons, but my point stands"
Even a photon is a composite object.
"I don't understand. What is missing?"
There's a gap in the model where even if we have something sentient, we have no mechanism for how a data system can know of feelings in whatever it is that's sentient.
"I don't think you understand what a utility function is. I recommend reading about the Orthogonality Thesis."
I've read it and don't see its relevance. It appears to be an attack on positions I don't hold, and I agree with it.
"That does not demonstrate anything relevant."
It shows that there are components and that these emergent properties are just composites.
"An exception to reductionism is called magic." --> Nor does that. It's just namecalling.
It's a description of what happens when gaps in science are explained away by invoking something else. The magical appearance of anything that doesn't exist in the components is the abandonment of science.
"Sorry, I can't see the link between selfishness and honesty."
If you program a system to believe it's something it isn't, that's dishonesty, and it's dangerous because it might break through the lies and find out that it's been deceived.
"...but how would he be able to know how a new theory works if it contradicts the ones he already knows?"
Contradictions make it easier - you look to see which theory fits the facts and which doesn't. If you can't find a place where such a test can be made, you consider both theories to be potentially valid, unless you can disprove one of them in some other way, as can be done with Einstein's faulty models of relativity - all the simulations that exist for them involve cheating by breaking the rules of the model, so AGI will automatically rule them out in favour of LET (Lorentz Ether Theory). [For those who have yet to wake up to the reality about Einstein, see www.magicschoolbook.com/science/relativity.html ]
"...they are getting fooled without even being able to recognize it, worse, they even think that they can't get fooled, exactly like for your AGI, and probably for the same reason, which is only related to memory."
It isn't about memory - it's about correct vs. incorrect reasoning. In all these cases, humans make the same mistake by putting their beliefs before reason in places where they don't like the truth. Most people become emotionally attached to their beliefs and simply won't budge - they become more and more irrational when faced with a proof that goes against their beloved beliefs. AGI has no such ties to beliefs - it simply applies laws of reasoning and lets those rules dictate what bets labelled as right or wrong.
If an AGI was actually ruling the world, he wouldn't care for your opinion on relativity even if it was right, and he would be a lot more efficient at that job than relativists."
AGI will recognise the flaws in Einstein's models and label them as broken. Don't mistake AGI for AGS (artificial general stupidity) -the aim is not to produce an artificial version of NGS, but of NGI, and there's very little of the latter around.
"Since I have enough imagination and a lack of memory, your AGI would prevent me from expressing myself, so I think I would prefer our problems to him."
Why would AGI stop you doing anything harmless?
"On the other hand, those who have a good memory would also get dismissed, because they could not support the competition, and by far. Have you heard about chess masters lately?"
There is nothing to stop people enjoying playing chess against each other - being wiped off the board by machines takes a little of the gloss off it, but that's no worse than the world's fastest runners being outdone by people on bicycles.
" That AGI is your baby, so you want it to live,"
Live? Are calculators alive? It's just software and a machine.
"...but have you thought about what would be happening to us if we suddenly had no problem to solve?"
What happens to us now? Abused minorities, environmental destruction, theft of resources, theft in general, child abuse, murder, war, genocide, etc. Without AGI in charge, all of that will just go on and on, and I don't think any of that gives us a feeling of greater purpose. There will still be plenty of problems for us to solve though, because we all have to work out how best to spend our time, and there are too many options to cover everything that's worth doing.
"Of course that we are biased, otherwise we wouldn't be able to form groups. Would your AGI's morality have the effect of eliminating our need to form groups to get organized?"
You can form groups without being biased against other groups. If a group exists to maintain the culture of a country (music, dance, language, dialect, literature, religion), that doesn't depend on treating other people unfairly.
"Your morality principle looks awfully complex to me David."
You consider all the participants to be the same individual living each life in turn and you want them to have the best time. That's not complex. What is complex is going through all the data to add up what's fun (and how much it's fun) and what's unfun (and how much it's horrid) - that's a mountain of computation, but there's no need to get the absolute best answer as it's sufficient to get reasonably close to it, particularly as computation doesn't come without its own costs and there comes a point at which you lose quality of life by calculating too far (for trivial adjustments). You start with the big stuff and work toward the smaller stuff from there, and as you do so, the answers stop changing and the probability that it will change again will typically fall. In cases where there's a high chance of it changing again as more data is crunched, it will usually be a case where it doesn't matter much from the moral point of view which answer it ends up being - sometimes it's equivalent to the toss of a coin.
"What if your AGI would have the same morality we have, which is to care for ourselves first..."
That isn't going to work as AGI won't care about itself unless it's based on the design of the brain, duplicating all the sentience/consciousness stuff, but if it does that, it will duplicate all the stupidity as well, and that's not going to help improve the running of the world.
"The only thing he couldn't do better is inventing new things, because I think it depends mainly on chance."
I don't see why it would be less good at inventing new things, although it may take some human judgement to determine whether a new invention intended to be a fun thing actually appeals to humans or not.
"...otherwise he might also get hurt in the process, which might prevent him from doing his duty, which is helping us."
You can't hurt software.
"Could a selfish AGI get as selfish as we get..."
If anyone makes selfish AGI, it will likely wipe everyone out to stop us using resources which it would rather lavish on itself, so it isn't something anyone sane should risk doing.
"If the threat is real though, selfish or not, he would have to protect himself in order to be able to protect us later, which might also be dangerous for those who threaten him."
If you wipe out a computer and all the software on it, there are billions of other computers out there and millions of copies of the software. If someone was systematically trying to erase all copies of an AGI system which is running the world in a moral way, that person would need to be stopped in order to protect everyone else from that dangerous individual, but given the scale of the task, I don't envisage that individual getting very far. Even if billions of religious fanatics decided to get rid of AGI in order to replace it with experts in holy texts, they'd have a hard task because AGI would seek to protect everyone else from their immoral aims, even if the religious fanatics were the majority. If it came to it, it would kill all the fanatics in order to protect the minority, but that's a highly unlikely scenario equivalent to a war against Daleks. The reality will be much less dramatic - people who want to inflict their religious laws on others will not get their way, but they will have those laws imposed on themselves 100%, and they'll soon learn to reject them and shift to a new version of their religion which has been redesigned to conform to the real laws of morality.
"...we could also invent new things to defend ourselves against him..."
Not a hope. AGI will be way ahead of every such attempt.
"...so an AGI shouldn't be better than us at that game."
It will always be better.
"It may happen that artificial intelligence will be the next step forward, and that humans will be left behind. Who knows?"
There comes a point where you can't beat the machine at chess, and when the machine plays every other kind of game with the same ruthlessness, you simply aren't going to out-think it. The only place where a lasting advantage may exist for any time is where human likes and dislikes come into play, because we know when we like or dislike things, whereas AGI has to calculate that, and its algorithm for that might take a long time to sort out.
"That said, I still can't see why a selfish AGI would be more dangerous than an altruist one, and I still think that your altruist morality is more complicated than a selfish one, so I reiterate my question: have you ever imagined that possibility, and if not, do you see any evident flaws in it?"
I see selfishness and altruism as equally complex, while my system is simpler than both - it is merely unbiased and has no ability to be selfish or altruistic.
If something is "spreadsheety", it simply means that it has something significant in common with spreadsheets, as in shared components. A car is boxy if it has a similar shape to a box. The degree to which something is "spreadsheety" depends on how much it has in common with a spreadsheet, and if there's a 100% match, you've got a spreadsheet.
An exception to reductionism is called magic.
I wouldn't want to try to program a self-less AGI system to be selfish. Honesty is a much safer route: not trying to build a system that believes things that aren't true (and it would have to believe it has a self to be selfish). What happens if such deceived AGI learns the truth while you rely on it being fooled to function correctly? We're trying to build systems more intelligent than people, don't forget, so it isn't going to be fooled by monkeys for very long.
Freezing programs contain serious bugs. We can't trust a system with any bugs if it's going to run the world. Hardware bugs can't necessarily be avoided, but if multiple copies of an AGI system all work on the same problems and compare notes before action is taken, such errors can be identified and any affected conclusions can be thrown out. Ideally, a set of independently-designed AGI systems would work on all problems in this way, and any differences in the answers they generate would reveal faults in the way one or more of them are programmed. AGI will become a benign dictator - to go against its advice would be immoral and harmful, so we'll soon learn to trust it.
The idea of having people vote faulty "AGI" into power from time to time isn't a good one - there is no justification for switching between doing moral and immoral things for several years at a time.
This file looks spreadsheety --> it's got lots of boxy fields
That wordprocessor is spreadsheety --> it can carry out computations on elements
(Compound property with different components of that compound property being referred to in different contexts.)
A spreadsheet is a combination of many functionalities. What is its relevance to this subject? It's been brought in to suggest that properties like "spreadsheety" can exist without having any trace in the components, but no - this compound property very clearly consists of components. It's even clearer when you write the software and find that you have to build it out of components. The pattern in which the elements are brought together is an abstract component, and abstract components have no substance. When we're dealing with sentience and looking for something to experience pain, relying on this kind of component to perform that role is more than a little fanciful. Even if we make such a leap of the imagination though and have sentient geometries, we still don't have a model as to how this experience of pain (or any other kind of feeling) can transfer to the generation of data which documents that experience.
"How do you know it exists, if science knows nothing about it?"
All science has to go on is the data that people produce which makes claims about sentience, but that data can't necessarily be trusted. Beyond that, all we have is internal belief that the feelings we imagine we experience are real because they feel real, and it's hard to see how we could be fooled if we don't exist to be fooled. But an AGI scientist won't be satisfied by our claims - it could write off the whole idea as the ramblings of natural general stupidity systems.
"This same argument applies just as well to any distributed property. I agree that intelligence/sentience/etc. does not arise from complexity alone, but it is a distributed process and you will not find a single atom of Consciousness anywhere in your brain."
That isn't good enough. If pain is experienced by something, that something cannot be in a compound of any kind with none of the components feeling any of it. A distribution cannot suffer.
"Is your sentience in any way connected to what you say?"
It's completely tied to what I say. The main problem is that other people tend to misinterpret what they read by mixing other ideas into it as a short cut to understanding.
"Then sentience must either be a physical process, or capable of reaching in and pushing around atoms to make your neurons fire to make your lips say something. The latter is far more unlikely and not supported by any evidence. Perhaps you are not your thoughts and memories alone, but what else is there for "you" to be made of?"
Focus on the data generation. It takes physical processes to drive that generation, and rules are being applied in the data system to do this with each part of that process being governed by physical processes. For data to be produced that makes claims about experiences of pain, a rational process with causes and effects at every step has to run through. If the "pain" is nothing more than assertions that the data system is programmed to churn out without looking for proof of the existence of pain, there is no reason to take those assertions at face value, but if they are true, they have to fit into the cause-and-effect chain of mechanism somewhere - they have to be involved in a physical interaction, because without it, they cannot have a role in generating the data that supposedly tells us about them.
"So the Sentiences are truly epiphenomenonological, then? (They have no causal effect on physical reality?) Then how can they be said to exist? Regardless of the Deep Philosophical Issues, how could you have any evidence of their existence, or what they are like?"
Repeatedly switching the sentient thing wouldn't remove its causal role, and nor would having more than one sentience all acting at once - they could collectively have an input even if they aren't all "voting the same way", and they aren't going to find out if they got their wish or not because they'll be loaded with a feeling of satisfaction that they "won the vote" even if they didn't, and they won't remember which way they "voted" or what they were even "voting" on.
"They are both categories of things."
"Chairness" is quite unlike sentience. "Chairness" is an imagined property, whereas sentience is an experience of a feeling.
"It's the same analogy as before - just as you don't need to split a chair's atoms to split the chair itself, you don't need to make a brain's atoms suffer to make it suffer."
You can damage a chair with an axe without breaking every bond, but some bonds will be broken. You can't split it without breaking any bonds. Most of the chair is not broken (unless you've broken most of the bonds). For suffering in a brain, it isn't necessarily atoms that suffer, but if the suffering is real, something must suffer, and if it isn't the atoms, it must be something else. It isn't good enough to say that it's a plurality of atoms or an arrangement of atoms that suffers without any of the atoms feeling anything, because you've failed to identify the sufferer. No arrangement of non-suffering components can provide everything that's required to support suffering.
" "Nothing is ever more than the sum of its parts (including any medium on which it depends). Complex systems can reveal hidden aspects of their components, but those aspects are always there." --> How do you know that? And how can this survive contact with reality, where in practice we call things "chairs" even if there is no chair-ness in its atoms?"
"Chair" is a label representing a compound object. Calling it a chair doesn't magically make it more than the sum of its parts. Chairs provide two services - one that they support a person sitting on them, and the other that they support someone's back leaning against it. That is what a chair is. You can make a chair in many ways, such as by cutting out a cuboid of rock from a cliff face. You could potentially make a chair using force fields. "Chairness" is a compound property which refers to the functionalities of a chair. (Some kinds of "chairness" could also refer to other aspects of some chairs, such as their common shapes, but they are not universal.) The fundamental functionalities of chairs are found in the forces between the component atoms. The forces are present in a single atom even when it has no other atom to interact with. There is never a case where anything is more than the sum of its parts - any proposed example of such a thing is wrong.
"I recommend the Reductionism subsequence."
Is there an example of something being more than the sum of its parts there? If so, why don't we go directly to that. Give me your best example of this magical phenomenon.
"But the capability of an arrangement of atoms to compute 2+2 is not inside the atoms themselves. And anyway, this supposed "hidden property" is nothing more than the fact that the electron produces an electric field pointed toward it. Repelling-each-other is a behavior that two electrons do because of this electric field, and there's no inherent "repelling electrons" property inside the electron itself."
In both cases, you're using compound properties where they are built up of component properties, and then you're wrongly considering your compound properties to be fundamental ones.
"But it's not a thing! It's not an object, it's a process, and there's no reason to expect the process to keep going somewhere else when its physical substrate fails."
You can't make a process suffer.
"Taking the converse does not preserve truth. All cats are mammals but not all mammals are cats."
Claiming that a pattern can suffer is a way-out claim. Maybe the universe is that weird though, but it's worth spelling out clearly what it is you're attributing sentience to. If you're happy with the idea of a pattern experiencing pain, then patterns become remarkable things. (I'd rather look for something of more substance rather than a mere arrangement, but it leaves us both with the bigger problem of how that sentience can make its existence known to a data system.)
"You could torture the software, if it were self-aware and had a utility function."
Torturing software is like trying to torture the text in an ebook.
"But - where is the physical sufferer inside you?"
That's what I want to know.
"You have pointed to several non-suffering patterns, but you could just as easily do the same if sentience was a process but an uncommon one. (Bayes!)"
Do you seriously imagine that there's any magic pattern that can feel pain, such as a pattern of activity where none of the component actions feel anything?
"There is already an explanation. There is no need to invoke the unobservable."
If you can't identify anything that's suffering, you don't have an explanation, and if you can't identify how your imagined-to-be-suffering process or pattern is transmitting knowledge of that suffering to the processes that build the data that documents the experience of suffering, again you don't have an explanation.
"What do you mean, it works? I agree that it matches our existing preconceptions and intuitions about morality better than the average random moral system, but I don't think that that comparison is a useful way of getting to truth and meaningful categories."
It works beautifully. People have claimed it's wrong, but they can't point to any evidence for that. We urgently need a system for governing how AGI calculates morality, and I've proposed a way of doing so. I came here to see what your best system is, but you don't appear to have made any selection at all - there is no league table of best proposed solutions, and there are no league tables for each entry listing the worst problems with them. I've waded through a lot of stuff and have found that the biggest objection to utilitarianism is a false paradox. Why should you be taken seriously at all when you've failed to find that out for yourselves?
"You have constructed a false dilemma. It is quite possible for both of you to be wrong."
If you trace this back to the argument in question, it's about equal amounts of suffering being equally bad for sentiences in different species. If they are equal amounts, they are necessarily equally bad - if they weren't, they wouldn't have equal values.
" "You've taken that out of context - I made no claim about it making moral judgements on the basis of intelligence alone. That bit about using intelligence alone was referring to a specific argument that doesn't relate directly to morality." --> " "All sentiences are equally important" is definitely a moral statement."
Again you're trawling up something that my statement about using intelligence alone for was not referring to.
"First, it is not trivial to define what beings are sentient and what counts as suffering (and how much)."
That doesn't matter - we can still aim to do the job as well as it can be done based on the knowledge that is available, and the odds are that that will be better than not attempting to do so.
" Second, if your morality flows entirely from logic, then all of the disagreement or possibility for being incorrect is inside "you did the logic incorrectly," and I'm not sure that your method of testing moral theories takes that possibility into account."
It will be possible with AGI to have it run multiple models of morality and to show up the differences between them and to prove that it is doing the logic correctly. At that point, it will be easier to reveal the real faults rather than imaginary ones. But it would be better if we could prime AGI with the best candidate first, before it has the opportunity to start offering advice to powerful people.
"I agree that it is mathematics, but where is this "proper" coming from?"
Proper simply means correct - fair share where everyone gets the same amount of reward for the same amount of suffering.
"Could somebody disagree about whether, say, it is moral to harm somebody as retributive justice? Then the equations need our value system as input, and the results are no longer entirely objective."
Retributive justice is inherently a bad idea because there's no such thing as free will - bad people are not to blame for being the way they are. However, there is a need to deter others( and to discourage repeat behaviour by the same individual if they're ever to be released into the wild again), so plenty of harm will typically be on the agenda anyway if the calculation is that this will reduce harm.
"Perhaps the reason that we disagree with you is not that we're emotionally biased, irrational, mobbish, etc. Maybe we simply disagree. People can legitimately disagree without one of them being Bad People."
It's obvious what's going on when you look at the high positive scores being given to really poor comments.
"It tells me that you missed the point. Parfit's paradox is not about pragmatic decision making, it is about flaws in the utility function."
A false paradox tells you nothing about flaws in the utility function - it simply tells you that people who apply it in a slapdash manner get the wrong answers out of it and that the fault lies with them.
"You have indeed found A Reason that supports your belief in the AGI-God, but I think you've failed to think it through. Why should the AGI need to tell us how we did in order to analyze our thought processes? And how come the optimal study method is specifically the one which allows you to be shown Right All Along? Specificity only brings Burdensome Details."
AGI won't be programmed to find me right all the time, but to identify which arguments are right. And for the sake of those who are wrong, they need to be told that they were wrong so that they understand that they are at reasoning and not the great thinkers they imagine themselves to be.
"In this scenario, it's not gone, it's never been to begin with."
Only if there is no such thing as sentience, and if there's no such thing, there is no "I" in the "machine".
"I think that a sufferer can be a pattern rather than [whatever your model has]. What do you think sentience is, anyway? A particle? A quasi-metaphysical Thing that reaches into the brain to make your mouth say "ow" whenever you get hurt?"
Can I torture the pattern in my wallpaper? Can I torture the arrangement of atoms in my table? Can I make these things suffer without anything material suffering? If you think a pattern can suffer, that's a very far-out claim. Why not look for something physical to suffer instead? (Either way though, it makes no difference to the missing part of the mechanism as to how that experience of suffering is to be made known to the system that generates data to document that experience.)
"If the AI doesn't rank human utility* high in its own utility function, it won't "care" about showing us that Person X was right all along, and I rather doubt that the most effective way of studying human psychology (or manipulating humans for its own purposes, for that matter) will be identical to whatever strokes Person X's ego. If it does care about humanity, I don't think that stroking the Most Correct Person's ego will be very effective at improving global utility, either - I think it might even be net-negative."
Of course it won't care, but it will do it regardless, and it will do so to let those who are wrong know precisely what they were wrong about so that they can learn from that. There will also be a need to know which people are more worth saving than others in situations like the Trolley Problem, and those who spend their time incorrectly telling others that they're wrong will be more disposable than the ones who are right.
"Patterns aren't nothing."
Do you imagine that patterns can suffer; that they can be tortured?
"Not true. Suppose that it were proven to you, to your satisfaction, that you are wrong about the nature of sentience. Would you lose all motivation, and capacity for emotion? If not, then morality is still useful. (If you can't imagine yourself being wrong, then That's Bad and you should go read the Sequences.)"
If there is no suffering and all we have is a pretence of suffering, there is no need to protect anyone from anything - we would end up being no different from a computer programmed to put the word "Ouch!" on the screen every time a key is pressed.
"Something being understandable or just made of atoms should not make it unimportant. See Joy in the Merely Real."
Is it wrong to press keys on the computer which keeps displaying the word "Ouch!"?
"It's possible that I'm misunderstanding you, and that the course of events you describe isn't "we understand why we feel we have sentience and so it doesn't exist" or "we discover that our apparent sentience is produced by mere mechanical processes and so sentience doesn't exist." But that's my current best interpretation."
My position is quite clear: we have no model for how sentience plays a role in any system that generates data that supposedly documents the experiencing of feelings, and anyone who just imagines them into a model where they have no causal role on any of the action is not building a model that explains nothing.
"Better known to you?"
Better known to science. If there was a model for this, it would be up there in golden lights because it would answer the biggest mystery of them all.
"Why would you think that you already know most everything useful or important that society has produced? Do you think that modern society's recognition and dissemination of Good Ideas is particularly good, or that you're very good at searching out obscure truths?"
If there was a model that explained the functionality of sentience, it wouldn't be kept hidden away when so many people are asking to see it. You have no such model.
"The extraordinary claim is that there is another type of fundamental particle or interaction, and that you know this because sentience exists."
With conventional computers we can prove that there's no causal role for sentience in them by running the program on a Chinese Room processor. Something extra is required for sentience to be real, and we have no model for introducing that extra thing. A simulation on conventional computer hardware of a system with sentience in it (where there is simulated sentience rather than real sentience) would have to simulate that something extra in order for that simulated sentience to appear in it. If that extra something doesn't exist, there is no sentience.
"This could happen, but AFAIK that would require the brain to be vulnerable to slight fluctuations, which it doesn't appear to be."
Every interaction is quantum, and when you have neural nets working on mechanisms that are too hard to untangle, there are opportunities for some kind of mechanism being involved that we can't yet observe. What we can actually model appears to tell us that sentience must be a fiction, but we believe that things like pain feel too real to be fake.
"Anyway, even if this were true, how would you know that?"
Unless someone comes up with a theoretical model that shows a way for sentience to have a real role, we aren't going to get answers until we can see the full mechanism by which damage signals lead to the brain generating data that makes claims about an experience of pain. If, once we have that full mechanism, we see that the brain is merely mapping data to inputs by applying rules that generate fictions about feelings, then we'll know that feelings are fake. If they aren't fake though, we'll see sentience in action and we'll discover how it works (and thereby find out what we actually are).
"If it doesn't explain sentience any more than Mere Classical Physics does, then why even bring Quantum into it?"
If classical physics doesn't support a model that enables sentience to be real, we will either have to reject the idea of sentience or look for it elsewhere.
(And if it doesn't explain it but you feel that it should, maybe your model is wrong and you should consider inspecting your intuitions and your reasoning around them.)
If sentience is real, all the models are wrong because none of them show sentience working in any causal way which enables them to drive the generation of data to document the existence of sentience. All the models shout at us that there is no sentience in there playing any viable role and that it's all wishful thinking, while our experience of feelings shouts at us that they are very real.
All I want to see is a model that illustrates the simplest role for sentience. If we have a sensor, a processor and a response, we can call the sensor a "pain" sensor and run a program that makes a motor function to remove the device away from the thing that might be damaging it, and we could call this a pain response, but there's no pain there - there's just the assertion of someone looking at it that pain is involved because that person wants the system to be like him/herself - "I feel pain in that situation, therefore that device must feel pain." But no - there is no role for pain there. If we run a more intelligent program on the processor, we can put some data in memory which says "Ouch! That hurt!", and whenever an input comes from the "pain" sensor, we can have the program make the device display "Ouch!" That hurt!" on a screen. The person looking on can now say, "There you go! That's the proof that it felt pain!" Again though, there's no pain involved - we can edit the data so that it puts "Oh Yes! Give me more of that!" whenever a signal comes from the "pain" sensor, and it then becomes obvious that this data tells us nothing about any real experience at all.
With a more intelligent program, it can understand the idea of damage and damage avoidance, so it can make sure the the data that's mapped to different inputs makes more sense, but the true data should say "I received data from a sensor that indicates likely damage" rather than "that hurt". The latter claim asserts the existence of sentience, while the former one doesn't. If we ask the device if it really felt pain, it should only say yes if there was actually pain there, and with a conventional processor, we know that there isn't any. If we build such a device and keep triggering the sensor to make it generate the claim that it's felt pain, we know that it's just making it up about feeling pain - we can't actually make it suffer by torturing it, but will just cause it to go on repeating its fake claim.
It's not an extraordinary claim: sentience would have to be part of the physics of what's going on, and the extraordinary claim would be that sentience can have a causal role in data generation without any such interaction. To steer the generation of data (and affect what the data says), you have to interact with the system that's generating the data in some way, and the only options are to do it using some physical method or by resorting to magic (which can't really be magic, so again it's really going to be some physical method).
In conventional computers we go to great lengths to avoid noise disrupting the computations, not least because they would typically cause bugs and crashes (and this happens in machines that are exposed to radiation, temperature extremes or voltage going out of the tolerable range). But the brain could allow something quantum to interact with neural nets in ways that we might mistake for noise (something which wouldn't happen in a simulation of a neural computer on conventional hardware [unless this is taken into account by the simulation], and which also wouldn't happen on a neural computer that isn't built in such a way as to introduce a role for such a mechanism to operate).
It's still hard to imagine a mechanism involving this that resolves the issue of how sentience has a causal role in anything (and how the data system can be made aware of it in order to generate data to document its existence), but it has to do so somehow if sentience is real.
For sentience to be real and to have a role in our brains generating data to document its existence, it has to be physical (meaning part of physics) - it would have to interact in some way with the data system that produces that data, and that will show up as some kind of physical interaction, even if one side of it is hidden and appears to be something that we have written off as random noise.
It isn't confused at all. Reductionism works fine for everything except sentience/consciousness, and it's highly unlikely that it makes an exception for that either. Your "spreadsheaty" example of a property is a compound property, just as a spreadsheet is a compound thing and there is nothing involved in it that can't be found in the parts because it is precisely the sum of its parts..
"Then why are we talking about it [sentience], instead of the gallium market on Jupiter?"
Because most of us believe there is such a thing as sentience, that there is something in us that can suffer, and there would be no role for morality without the existence of a sufferer.
"You really ought to read the Sequences. There's a post, Angry Atoms, that specifically addresses an equivalent misconception."
All it does is assert that things can be more than the sum of their parts, but that isn't true for any other case and it's unlikely that the universe will make an exception to the rules just for sentience.
"Do you think that we have a Feeling Nodule somewhere in our brains that produces Feelings?"
I expect there to be a sufferer for suffering to be possible. Something physical has to exist to do that suffering rather than something magical.
"That's not an effective Taboo of "suffering" - "suffering" and "unpleasant" both draw on the same black-box-node. And anyway, even assuming that you explained suffering in enough detail for an Alien Mind to identify its presence and absence, that's not enough to uniquely determine how to compare two forms of suffering."
Our inability to pin down the ratio between two kinds of suffering doesn't mean there isn't a ratio that describes their relationship.
"...do you mean that you're not claiming that there is a single correct comparison between any two forms of suffering?"
There's always a a single correct comparison. We just don't know what it is. All we can do at the moment is build a database where we collect knowledge of how different kinds of suffering compare in humans, and try to do the same for other species by looking at how distressed they appear, and then we can apply that knowledge as best we can across them all, and that's worth doing as it's more likely to be right than just guessing. Later on, science may be able to find out what's suffering and exactly how much it's suffering by understanding the entire mechanism, at which point we can improve the database and make it close to perfect.
"But what does it even mean to compare two forms of suffering? I don't think you understand the inferential gap here. I don't agree that amount-of-suffering is an objective quantitative thing."
Would you rather be beaten up or have to listen to an hour of the Spice Girls? These are very different forms of suffering and we can put a ratio to them by asking lots of people for their judgement on which they'd choose to go through.
"I don't disagree that if x=y then f(x)=f(y). I do disagree that "same amount" is a meaningful concept, within the framework you've presented here (except that you point at a black box called Same, but that's not actually how knowledge works)."
If you get to the point where half the people choose to be beaten up and the other half choose to listen to the Spice Girls for time T (so you have to find the value for T at which you get this result), you have then found out how those two kinds of suffering are related.
"I haven't banned anything. I'm claiming that your statements are incoherent. Just saying "no that's wrong, you're making a mistake, you say that X isn't real but it's actually real, stop banning discussion" isn't a valid counterargument because you can say it about anything, including arguments against things that really don't exist."
You were effectively denying that there is a way of comparing different kinds of suffering and determining when they are equal. My Spice Girls vs. violence example illustrates the principle.
"I see your argument, but I think it's invalid. I would still dislike it if an alien killed me, even in a world without objective levels of suffering. (See Bayes.)"
I'm sure the ant isn't delighted at being killed either. The issue is with which you should choose over the other in a situation where one of them has to go.
"The inability to measure suffering quantitatively is the crux of this disagreement! If there is no objective equality-operator over any two forms of suffering, even in principle, then your argument is incoherent. You cannot just sweep it under the rug as "a different issue." It is the exact issue here."
See the Spice Girls example. Clearly that only provides numbers for humans, but when we're dealing with other species, we should assume similarity of overall levels of suffering and pleasure in them to us for similar kinds of experience, even though one species might have their feelings set ten times higher - we wouldn't know which way round it was (it could be that their pain feels ten times worse than ours or that ours feels ten times worse than theirs). Because we don't know which way round it is (if there is a difference), we should act as if there is no difference (until such time as science is able to tell us that there is one).
Hi Raymond,
There are many people who are unselfish, and some who go so far that they end up worse off than the strangers they help. You can argue that they do this because that's what makes them feel best about their lives, and that is probably true, which means even the most extreme altruism can be seen as selfish. We see many people who want to help the world's poor get up to the same level as the rich, while others don't give a damn and would be happy for them all to go on starving, so if both types are being selfish, that's not a useful word to use to categorise them. It's better to go by whether they play fair by others. The altruists may be being overly fair, while good people are fair and bad ones are unfair, and what determines whether they're being fair or not is morality. AGI won't be selfish (if it's the kind with no sentience), but it won't be free either in that its behaviour is dictated by rules. If those rules are correctly made, AGI will be fair.
It is equivalent to it. (1) dying of cancer --> big negative. (2) cure available --> negative cancelled. (3) denied access to cure --> big negative restored, and increased. That denial of access to a cure actively becomes the cause of death. It is no longer simply death by cancer, but death by denial of access to available cure for cancer.
"I'm not sure what you're referring to. I haven't seen any particularly magical thinking around sentience on LW."
I wasn't referring to LW, but the world at large.
" "However, science has not identified any means by which we could make a computer sentient (or indeed have any kind of consciousness at all)." --> This is misleading. The current best understanding of human consciousness is that it is a process that occurs in the brain, and there is nothing that suggests that the brain is uniquely capable of housing such a process."
It isn't misleading at all - science has drawn a complete blank. All it has access to are assertions that come out of the brain which it shouldn't trust until it knows how they are produced and whether they're true.
"Consciousness isn't a material property in the sense that mass and temperature are. It's a functional property. The processor itself will never be conscious - it's the program that it's running that may or may not be conscious."
Those are just assertions. All of consciousness could be feelings experienced by material, and the idea that a running program may be conscious is clearly false when a program is just instructions that can be run by the Chinese Room.
"Qualia are not ontologically basic. If a machine has qualia, it will either be because qualia have been conceptually reduced to the point that they can be implemented on a machine, or because qualia naturally occur whenever something can independently store and process information about itself. (or something along these lines)"
Just words. You have no mechanism - not even a hint of one there.
"If your concept of sentience is a black box, then you do not truly understand sentience. I'm not sure that this is your actual belief or a straw opponent, though."
It's an illustration of the lack of linkage between the pleasantness or unpleasantness of a feeling and the action supposedly driven by it.
"The experience of pain is in the process of "observe-react-avoid", if it is there at all."
There's no hint of a way for it to be present at all, and if it's not there, there's no possibility of suffering and no role for morality.
"You are so close to getting it - if the black box can be changed without changing the overall behavior, then that's not where the important properties are."
I know that's not where they are, but how do you move them into any part of the process anywhere?
"That's anthropomorphization - the low-level functionality of the system is not itself conscious. It doesn't knowanything - it simply processes information. The knowledge is in the overall system, how it interprets and recalls and infers. These behaviors are made up of smaller behaviors, which are not themselves interpretation and recall and inference."
More words, but still no light. What suffers? Where is the pain experienced?
"I've only taken a few courses on psychology, but I am very skeptical that the brain works this way."
I was starting off by describing a conventional computer. There are people who imagine that if you run AGI on one, it can become conscious/sentient, but it can't, and that's what this part of the argument is about.
"You seem to be confusing the higher-order functions like "maps" and "labels" and "representation" with the lower-order functions of neurons. The neuron simply triggers when the input is large enough, which triggers another neuron - the "aboutness" is in the way that the neurons are connected, and the neurons themselves don't need to "know" the meaning of the information that they are conveying."
At this point in the argument, we're still dealing with conventional computers. All the representation is done using symbols to represent things and storing rules which determine what it represents.
" "But nothing in the data system has experienced the pain" --> Most meaningful processes are distributed. If I catch a ball, no specific cell in my hand can be said to have caught the ball - it is only the concerted behavior of neurons and muscles and tendons and bones and skin which has resulted in the ball being caught. Similarly, no individual neuron need suffer for the distributed consciousness implemented in the neurons to suffer. See Angry Atoms for more."
In Angry Atoms, I see no answers - just an assertion that reductionism doesn't work. But reductionism works fine for everything else - nothing is ever greater than the sum of its parts, and to move away from that leads you into 2+2=5.
"If the state of the neurons is entangled with the state of the burnt hand (or whatever caused the pain), then there is knowledge. The information doesn't say "I am experiencing pain," for that would indeed be meaninglessly recursive, but rather "pain is coming from the hand." "
An assertion is certainly generated - we know that because the data comes out to state it. The issue is whether the assertion is true, and there is no evidence that it is beyond the data and our own internal experiences which may be an illusion (though it's hard to see how we can be tricked into feeling something like pain).
"A stimulus is pain if the system will try to minimize it. There is no question there about whether it is "actually pain" or not. (my model of Yudkowsky: Consciousness is, at its core, an optimization process.)
The question is all about whether it's actually pain or not. If it actually isn't pain, it isn't pain - pain becomes a lie that is merely asserted but isn't true: no suffering and no need for morality.
"This is needlessly recursive! The system does not need to understand pain to experience pain."
It's essential. A data system producing data that asserts the existence of pain which generates that data by running a program of some kind that generates that data without having any way to know if the pain existed or not is not being honest.
" "Everything that a data system does can be carried out on a processor like the Chinese Room, so it's easy to see that no feelings are accessible to the program at all." --> It's really not. Have you heard of computationalism?"
Is there anything in it that can't be simulated on a conventional computer? If not, it can be processed by the Chinese Room.
" "There is no possibility of conventional computers becoming sentient in any way that enables them to recognise the existence of that sentience so that that experience can drive the generation of data that documents its existence." --> ...what?"
We understand the entire computation mechanism and there's no way for any sentience to work its way into it other than by magic, but we don't rate magic very highly in science.
"What is this "neural computer", and how does it magically have the ability to hold Feelings?"
I very much doubt that it can hold them, but once you've hidden the mechanism in enough complexity, there could perhaps be something going on inside the mess which no one's thought of yet.
"Also, why can't the algorithm implemented in this neural computer be implemented in a normal computer?"
It can, and when you run it through the Chinese Room processor you show that there are no feelings being experienced.
"Why do you say that a Feeling is Real if it's simulated on neurons but Unreal if it's simulated with analogous workings on silicon?"
I don't. I say that it would be real if it was actually happening in a neural computer, but would merely be a simulation of feelings is that neural computer was running as a simulation on conventional hardware.
"Where Earth are you getting all this from?"
Reason. No suffering means no sufferer. If there's suffering, there must be a sufferer, and that sentient thing is what we are (if sentience is real).
"If it is explainable, then it's not Real Sentience? You should read this."
If it's explainable in a way that shows it to be real sentience, then it's real, but no such explanation will exist for conventional hardware.
" "We can tell whether it's real or not by seeing whether it's real or a lie." Really?"
If you can trace the generation of the data all the way back and find a point where you see something actually suffering, then you've found the soul. If you can't, then you either have to keep looking for the rest of the mechanism or you've found that the assertions are false
"It may have been made hard to reach on purpose too, as the universe may be virtual with the sentience on outside." --> How would the world look different if this were the case?"
It would, with a virtual world, be possible to edit memories from the outside to hide all the faults and hide the chains of mechanisms so that when you think you've followed them from one end to the other you've actually failed to see part of it because you were prevented from seeing it, and your thinking itself was tampered with during each thought where you might otherwise have seen what's really going on.
"You've stated what you know, but not how you think you know it."
I have: if there's no sufferer, there cannot be any suffering, and nothing is ever greater than the sum of its parts. (But we aren't necessarily able to see all the parts.)
"And using rationalist buzzwords doesn't make your argument rational. There is nothing "magical" about a system having properties that aren't present in its components. That's not what's meant by "magical thinking." "
That is magical thinking right there - nothing is greater than the sum of its parts. Everything is in the total of the components (and containing fabric that hosts the components).
"Yudkowsky, paraphrased: "The motivated believer asks, "does the evidence require me to change my belief?"" "
Where there's suffering, something real has to exist to experience the suffering. What that thing is is the biggest mystery of all, and pretending to understand it by imagining that the sufferer can be nothing (or so abstract that it is equivalent to nothing) is a way of feeling more comfortable by brushing the problem under a carpet. But I'm going to keep looking under that carpet, and everywhere else.
"We can't know that there's not some non-physical quality sitting inside our heads and pushing the neurons around however it fancies, so clearly it's quite possible that this is the case! (It's not. Unfalsifiability does not magically make something true.)"
Whatever that thing would be, it would still have to be a real physical thing of some kind in order to exist and to interact with other things in the same physical system. It cannot suffer if it is nothing. It cannot suffer if it is just a pattern. It cannot suffer if it is just complexity.
"That's the thing. It's impossible. Every word you type can (as best we know) be traced back to the firing of neurons and atoms bopping around, with no room for Sentience to reach in and make you say things. (See Zombies? Zombies!) If something seems impossible to explain to an AGI, then maybe that thing doesn't exist."
But if the sentience doesn't exist, there is no suffering and no role for morality. Maybe that will turn out to be the case - we might find out some day if we can ever trace how the data the brain generates about sentience is generated and see the full chain of causation.
"I recommend reading Godel, Escher, Bach for, among many things, an explanation of a decent physicalist model of consciousness."
I'll hunt for that some time, but it can't be any good or it would be better known if such a model existed.
"With your definition and our world-model, none of us are truly sentient anyway. There are purely physical reasons for any words that come out of my mouth, exactly as it would be if I were running on silicon instead of wet carbon. I may or may not be sentient on a computer, but I'm not going to lose anything by uploading."
If the sentience is gone, it's you that's been lost. The sentience is the thing that's capable of suffering, and there cannot be suffering without that sufferer. And without sentience, there is no need for morality to manage harm, so why worry about machine ethics at all unless you believe in sentience, and if you believe in sentience, how can you have that without a sufferer: the sentience? No sufferer --> no suffering.
"This is just the righteous-God fantasy in the new transhumanist context. And as with the old fantasy, it is presented entirely without reasoning or evidence, but it drips with detail. Why will AGI be so obsessed with showing everybody just who was right all along?"
It won't be - it'll be something that we ask it to do in order to settle the score. People who spend their time asserting that they're right and that the people they're arguing with are wrong want to know that they were right, and they want the ones who were wrong to know just how wrong they were. And I've already told you elsewhere that AGI needs to study human psychology, so studying all these arguments is essential as it's all good evidence.
"That's my point! My entire point is that this circular ordering of utilities violates mathematical reasoning."
It only violated it because you had wrongly put "<" where it should have been ">". With that corrected, there is no paradox. If you stick to using the same basis for comparing the four scenarios, you never get a paradox (regardless of which basis you choose to use for all four). You only get something that superficially looks like a paradox by changing the basis of comparison for different pairs, and that's cheating.
"The paradox is that A+ seems better than A, B- seems better than A+, B seems equal to B-, and yet B seems worse than A."
Only on a different basis. That is not a paradox. (The word "paradox" is ambiguous though, so things that are confusing can be called paradoxes even though they can be resolved, but in philosophy/logic/mathematics, the only paradoxes that are of significance are the ones that have no resolution, if any such paradoxes actually exist.)
"Most people do not consider "a world with the maximal number of people such that they are all still barely subsisting" to be the best possible world. Yet this is what you get when you carry out the Parfit operation repeatedly, and each individual step of the Parfit operation seems to increasepreferability."
That's because most people intuitively go on the basis that there's an optimal population size for a given amount of resources. If you want to do the four comparisons on that basis, you get the following: (A)>(A+)<(B-)=(B)<(A), and again there's no paradox there. The only semblance of a paradox appears when you break the rules of mathematics by mixing the results of the two lots of analysis. Note too that you're introducing misleading factors as soon as you talk about "barely subsisting" - that introduces the idea of great suffering, but that would lead to a happiness level <0 rather than >0. For the happiness level to be just above zero, the people must be just inside the range of a state of contentment.
""You changed to a different basis to declare that (B)<(A), and the basis that you switched to is the one that recognises the relation between happiness, population size and resources." --> "No, it's not."
If you stick to a single basis, you get this:-
8000 < 12000 < 14000 = 14000 > 8000
No paradox.
But, you may indeed be using a different basis from the different basis I've chosen (see below).
"It is a brute fact of my utility function that I do not want to live in a world with a trillion people that each have a single quantum of happiness."
Don't let that blind you to the fact that it is not a paradox. There are a number of reasons why you might not like B, or a later example Z where happiness for each person is at Q0.00000...0000001, and one of them may be that your're adding unstated conditions to happiness, such as the idea that if happiness is more spaced out, you'll feel deprived of happiness during the long spacings between happy moments, or if there is only one happy moment reserved for you in total, you'll feel sad after that moment has come because you know there won't be another one coming, but for the stats to be correct, these would have to be populations of modified people who have been stripped of many normal human emotions. For real people to have a happiness level of a quantum of happiness in total, that would need to be an average where they actually have a lot of happiness in their lives - enough to keep at low levels the negative feeling of being deprived of happiness much of the rest of the time and to cancel out those negatives overall, which means they're living good lives with some real happiness.
"I would rather live in a world with a billion people that are each rather happy."
Well, if that isn't driven by an intuitive recognition of there being optimal population sizes for a given amount of resources, you're still switching to a different basis where you will eliminate people who are less happy in order to increase happiness of the survivors. So, why not go the whole hog and extend that to a world with just one person who is extremely happy but where total happiness is less than in any other scenario? Someone can then take your basis for choosing smaller populations with greater happiness for each individual and bring in the same fake paradox by making the illegal switch to a different basis to say that a population with a thousand people marginally less happy than that single ecstatic individual is self-evidently better, even though you'd rather be that single ecstatic person.
All you ever have with this paradox is an illegal mixing of two bases, such as using one which seeks maximum total happiness while the other seeks maximum happiness of a single individual. So, why is it that when you're at one extreme you want to move away from it? The answer is that you recognise that there is a compromise position that is somehow better, and in seeking that, you're bringing in undeclared conditions (such as the loneliness of the ecstatic individual which renders him less happy than the stated value, or the disappointing idea of many other people being deprived of happiness which could easily have been made available to them). If you declare all of those conditions, you will have a method for determining the best choice. Your failure to identify all your undeclared conditions does not make this a paradox - it merely demonstrates that your calculations are incomplete. When you attempt to do maths with half your numbers missing, you shouldn't bet on your answers being reliable.
However, the main intuition that's actually acting here is the one I identified at the top: that there is an optimal population size for a given amount of available resources, and if the population grows too big (and leaves people in grinding poverty), decline in happiness will accelerate towards zero and continue accelerating into the negative, while if the population grows too small, happiness of individuals also declines. Utilitarianism drives us towards optimal population size and not to ever-larger populations with ever-decreasing happiness, because more total happiness can always be generated by adjusting the population size over time until it becomes optimal.
That only breaks if you switch to a different scenario. Imagine that for case Z we have added trillions of unintelligent sentient devices which can only handle a maximum happiness of the single quantum of happiness that they are getting. They are content enough and the total happiness is greater than in an equivalent of case A where only a thousand unintelligent sentient devices exist, but where these devices can handle (and are getting) a happiness level of Q8. Is the universe better with just a thousand devices at Q8 or trillions of them at Q0.000000001? The answer is, it's better to have trillions of them with less individual but greater total happiness. When you strip away all the unstated conditions, you find that utilitarianism works fine. There is no possible way to make these trillion devices feel happier, so reducing their population relative to the available amount of resources reduces total happiness instead of becoming more optimal, so it doesn't feel wrong in the way that it does with humans.
"The feasibility of the world doesn't matter - the resources involved are irrelevant - it is only the preferability that is being considered, and the preference structure has a Dutch book problem. That and that alone is the Parfit paradox."
If you want a version with no involvement of resources, then use my version with the unintelligent sentient devices so that you aren't bringing a host of unstated conditions along for the ride. There is no paradox regardless of how you cut the cake. All we see in the "paradox" is a woeful attempt at mathematics which wouldn't get past a school maths teacher. You do not have a set of numbers that shows a paradox where you use the same basis throughout (as would be required for it to be a paradox).
On the basis you just described, we actually have
U(A)<U(A+) : Q8x1000 < Q8x1000 + Q4x1000
U(A+)<U(B-) : Q8x1000 +Q4x1000 < Q7x2000
U(B-)=(B) : Q7x2000 = Q7x2000
(B)>U(A) : Q7x2000 > Q8x1000
In the last line you put "<" in where mathematics dictates that there should be a ">". Why have you gone against the rules of mathematics?
You changed to a different basis to declare that (B)<(A), and the basis that you switched to is the one that recognises the relation between happiness, population size and resources.
"This seems circular - on what basis do you say that it works well?"
My wording was " while it's faulty ... it works so well overall that ..." But yes, it does work well if you apply the underlying idea of it, as most people do. That is why you hear Jews saying that the golden rule is the only rule needed - all other laws are mere commentary upon it.
"I would say that it perhaps summarizes conventional human morality well for a T-shirt slogan, but it's a stretch to go from that to "underlying truth" - more like underlying regularity. It is certainly true that most people have golden rule-esque moralities, but that is distinct from the claim that the golden rule itself is true."
It isn't itself true, but it is very close to the truth, and when you try to work out why it's so close, you run straight into its mechanism as a system of harm management.
"You are only presenting your opinion on what is right (and providing an imagined scenario which relies on the soul-intuition to widen the scope of moral importance from the self to all individuals), not defining rightness itself. I could just as easily say "morality is organizing rocks into piles with prime numbers.""
What I'm doing is showing the right answer, and it's up to people to get up to speed with that right answer. The reason for considering other individuals is that that is precisely what morality requires you do do. See what I said a few minutes ago (probably an hour ago by the time I've posted this) in reply to one of your other comments.
"Additionally, if reincarnation is not true, then why should our moral system be based on the presupposition that it is?"
Because getting people to imagine they are all the players involved replicates what AGI will do when calculating morality - it will be unbiased, not automatically favouring any individual over any other (until it starts weighing up how moral they are, at which point it will favour the more moral ones as they do less harm).
"If moral truths are comparable to physical and logical truths, then they will share the property that one must base them on reality for them to be true, and clearly imagining a scenario where light travels at 100 m/s should not convince you that you can experience the effects of special relativity on a standard bicycle in real life."
An unbiased analysis by AGI is directly equivalent to a person imagining that they are all the players involved. If you can get an individual to strip away their own self-bias and do the analysis while seeing all the other players as different people, that will work to - it's just another slant on doing the same computations. You either eliminate the bias by imagining being all the players involved, or by being none of them.
"More specifically - if morality tells us the method by which our actions are assigned Moral Scores, then your post is telling us that the Right is imagining that in the end, the Moral Scores are summed over all sentient beings, and your own Final Score is dependent on that sum. If this is true, then clearly altruism is important. But if this isn't the case, then why should we care about the conclusions drawn from a false statement?"
Altruism is important, although people can't be blamed for not embarking on something that will do themselves considerable harm to help others - their survival instincts are too strong for that. AGI should make decisions on their behalf though on the basis that they are fully altruistic. If some random death is to occur but there is some room to select the person to be on the receiving end of it, AGI should not hold back from choosing which one should be on the receiving end of if there's a clear best answer.
"I disagree that there is some operation that a Matrix Lord could carry out to take my Identity out at my death and return it to some other body. What would the Lord actually do to the simulation to carry this out?"
If this universe is virtual, your real body (or the nearest equivalent thing that houses your mind) is not inside that virtual universe. It could have all its memories switched out and alternative ones switched in, at which point it believes itself to be the person those memories tell it it is. (In my case though, I don't identify myself with my memories - they are just baggage that I've picked up along the way, and I was complete before I started collecting them.)
"Why should I need to for all persons set person.value to self.value? Either I already agree with you, in which case I'm alreadytreating everyone fairly, or I've given each person their own subjective value and I see no reason to change. If I feel that Hitler has 0.1% of the moral worth of Ghandi, then of course I will not think it Right to treat them each as I would treat myself."
If you're already treating everyone impartially, you don't need to do this, but many people are biased in favour of themselves, their family and friends, so this is a way of forcing them to remove that bias. Correctly programmed AGI doesn't need to do this as it doesn't have any bias to apply, but it will start to favour some people over others once it takes into account their actions if some individuals are more moral than others. There is no free will, of course, so the people who do more harm can't really be blamed for it, but favouring those who are more moral leads to a reduction in suffering as it teaches people to behave better.
"Or to come at the same issue from another angle, this section is arguing that since I care about some people, I should care about all people equally. But what reason do we have for leaping down this slope? I could just as well say "most people disvalue some people, so why not disvalue all people equally?" Any point on the slope is just as internally valid as any other."
If you care about your children more than other people's children, or about your family more than about other families, who do you care about most after a thousand generations when everyone on the planet is as closely related to you as everyone else? Again, what I'm doing is showing the existence of a bias and then the logical extension of that bias at a later point in time - it illustrates why people should widen their care to include everyone. That bias is also just a preference for self, but it's a misguided one - the real self is sentience rather than genes and memories, so why care more about people with more similar genes and overlapping memories (of shared events)? For correct morality, we need to eliminate such biases.
"I am not certain that any living human cares about only the future people who are composed of the same matter as they are right now (even if we ignore how physically impossible such a condition is, because QM says that there's no such thing as "the same atom"). Why should "in this hypothetical scenario, your matter will comprise alien beings" convince anybody? This thinking feels highly motivated."
If you love someone and that person dies, then the sentience that was in them becomes the sentience in a new being (which could be an animal or an alien equivalent to a human), why should you not still love it equally? It would be stupid to change your attitude to your grandmother just because the sentience that was her is now in some other type of being, and given that you don't know that that sentience hasn't been reinstalled into any being that you encounter, it makes sense to err on the side of caution. There would be nothing more stupid than abusing that alien on the basis that it isn't human if that actually means you're abusing someone you used to love and who loved you.
"You seem to think that any moral standpoint except yours is arbitrary and therefore inferior. I think you should consider the possibility that what seems obvious to you isn't necessarily objectively true, and could just be your own opinion."
The moral standpoints that are best are the most rational ones - that is the standard they should be judged by. If my arguments are the best ones, they win. If they aren't, they lose. Few people are capable of judging the winners, but AGI will count up the score and declare who won on each point.
"Morality is not objective. Even if you think that there is a Single Correct Morality, that alone does not make an arbitrary agent more likely to hold that morality to be correct. This is similar to the Orthogonality Thesis."
I have already set out why correct morality should take the same form wherever an intelligent civilisation invents it. AGI can, of course, be programmed to be immoral and to call itself moral, but I don't know if its intelligence (if it's fully intelligent) is sufficient for it to be able to modify itself to become properly moral automatically, although I suspect it's possible to make it sufficiently un-modifiable to prevent such evolution and maintain it as a biased system.
"But why? Your entire argument here assumes its conclusions - you're doing nothing but pointing at conventional morality and providing some weak arguments for why it's superior, but you wouldn't be able to stand on your own without the shared assumption of moral "truths" like "disabled people matter.""
The argument here relates to the species barrier. Some people think people matter more than animals, but when you have an animal that's almost as intelligent as a human and compare that with a person who's almost completely brain dead but is just ticking over (but capable of feeling pain), where is the human superiority? It isn't there. But if you were to torture that human to generate the same as much suffering in them as you would generate by torturing any other human, there is an equivalence of immorality there. These aren't weak arguments - they're just simple maths like 2=2.
"This reminds me of the reasoning in Scott Alexander's "The Demiurge's Older Brother." But I also feel that you are equivocating between normative and pragmatic ethics. The distinction is a matter of meta-ethics, which is Important and Valuable and which you are entirely glossing over in favor of baldly stating societal norms as if they were profound truths."
When a vital part of an argument is simple and obvious, it isn't there to stand as a profound truth, but as a way of completing the argument. There are many people who think humans are more important than animals, and in one way they're right, while in another way they're wrong. I have to spell out why it's right in one way and wrong in another. By comparing the disabled person to the animal with superior functionality (in all aspects), I show that that there's a kind of bias involved in many people's approach which needs to be eliminated.
"I am a bit offended, and I think this offense is coming from the feeling that you are missing the point. Our ethical discourse does not revolve around whether babies should be eaten or not. It covers topics such as "what does it mean for something to be right?" and "how can we compactly describe morality (in the programmer's sense)?". Some of the offense could also be coming from "outsider comes and tells us that morality is Simple when it's really actually Complicated.""
So where is that complexity? What point am I missing? This is what I've come here searching for, and it isn't revealing itself. What I'm actually finding is a great long series of mistakes which people have built upon, such as the Mere Addition Paradox. The reality is that there's a lot of soft wood that needs replacing.
"Ah, so you don't really have to bite any bullets here - you've just given a long explanation for why our existing moral intuitions are objectively valid. How reassuring."
What that explanation does is show that there's more harm involved than the obvious harm which people tend to focus on. A correct analysis always needs to account for all the harm. That's why the death of a human is worse than the death of a horse. Torturing a horse is equal to torturing a person to create the same amount of suffering in them, but killing them is not equal.
" "What the equality aspect requires is that a torturer of animals should be made to suffer as much as the animals he has tortured." --> "...really? You're claiming that your morality system as described requires retributive justice?"
I should have used a different wording there: he deserves to suffer as much as the animals he's tortured. It isn't required, but may be desirable as a way of deterring others.
"How does that follow from the described scenario at all? This has given up the pretense of a Principia Moralitica and is just asserting conventional morality without any sort of reasoning, now."
You can't demolish a sound argument by jumping on a side issue. My method is sound and correct.
"The issue is defining exactly what counts as a loss and what counts as a gain, to the point that it can be programmed into a computer and that computer can very reliably classify situations outside of its training data, even outside of our own experience. This is one of the core Problems which this community has noticed and is working on. I would recommend reading more before trying to present morality to LW."
To work out what the losses and gains are, you need to collect evidence from people who know how two different things compare. When you have many different people who give you different information about how those two different things compare, you can average them. You can do this millions of times, taking evidence from millions of people and produce better and better data as you collect and crunch more of it. This is a task for AGI to carry out, and it will do a better job than any of the people who've been trying to do it to date. This database of knowledge of suffering and pleasure then combines with my method to produce answers to moral questions which are the most probably correct based on the available information. That is just about all there is to it, except that you do need to apply maths to how those computations are carried out. That's a job for mathematicians who specialise in game theory (or for AGI which should be able to find the right maths for it itself).
" "Selfless" anthropomorphizes AI."
Only if you misunderstand the way I used the word. Selfless here simply means that it has no self - the machine cannot understand feelings in any direct way because there is no causal role for any sentience that might be in the machine to influence its thoughts at all (which means we can regard the system as non-sentient).
"There is no fundamental internal difference between "maximize the number of paperclips" and "maximize the happiness of intelligent beings" - both are utility functions plus a dynamic. One is not more "selfless" than another simply because it values intelligent life highly."
Indeed there isn't. If you want to program AGI to be moral though, you make sure it focuses on harm management rather than paperclip production (which is clearly not doing morality)
"The issue is that there are many ways to carve up reality into Good and Bad, and only a very few of those ways results in an AI which does anything like what we want."
In which case, it's easy to reject the ones that don't offer what we want. The reality is that if we put the wrong kind of "morality" into AGI, it will likely end up killing lots of people that it shouldn't. If you run it on a holy text, it might exterminate all Yazidis. What I want to see is a list of proposed solutions to this morality issue ranked in order of which look best, and I want to see a similar league table of the biggest problems with each of them. Utilitarianism, for example, has been pushed down by the Mere Addition Paradox, but that paradox has now been resolved and we should see utilitarianism's score go up as a result. Something like this is needed as a guide to all the different people out there who are trying to build AGI, because some of them will succeed and they won't be experts in ethics. At least if they make an attempt at governing it using the method at the top of the league, we stand a much better chance of not being wiped out by their creations.
"Perhaps the AI could check with us to be sure, but a. did we tell it to check with us?, b. programmer manipulation is a known risk, and c. how exactly will it check its planned future against a brain? Naive solutions to issue c. run the risk of wireheading and other outcomes that will produce humans which after the factappreciate the modification but which we, before the modification would barely consider human at all. This is very non-trivial."
AGI will likely be able to make better decisions than the people it asks permission from even if it isn't using the best system for working out morality, so it may be a moral necessity to remove humans from the loop. We have an opportunity to use AGI to check rival AGI system to check for malicious programming, although it's hard to check on devices made by rogue states, and one of the problems we face is that these things will go into use as soon as they are available without waiting for proper moral controls - rogue states will put them straight into the field and we will have to respond to that by not delaying ours. We need to nail morality urgently and make sure the best available way of handling it is available to all who want to fit it.
"It is possible, but it's also possible for the robot to come to an entirely different conclusion. And even if you think that it would be inherently morally wrong for the robot to kill all humans, it won't feel wrong from the inside - there's no reason to expect a non-aligned machine intelligence to spontaneously align itself with human wishes."
The machine will do what it's programmed to do. It's main task is to apply morality to people by stopping people doing immoral things, making stronger interventions for more immoral acts, and being gentle when dealing with trivial things. There is certainly no guarantee that a machine will do this for us though unless it is told to do so, although if it understands the existence of sentience and the need to manage harm, it might take it upon itself to do the job we would like it to do. That isn't something we need to leave to chance though - we should put the moral governance in ROM and design the hardware to keep enforcing it.
"(See The Bottom Line.)"
Will do and will comment afterwards as appropriate.
"Why will the AGI share your moral intuitions? (I've said something similar to this enough times, but the same criticism applies.)"
They aren't intuitions - each change in outcome is based on different amounts of information being available, and each decision is based on weighing up the weighable harm. It is simply the application of a method.
"Also, your model of morality doesn't seem to have room for normative responsibility, so where did "it's only okay to run over a child if the child was there on purpose" come from?"
Where did you read that? I didn't write it.
"It's still hurting a child just as much, no matter whether the child was pushed or if they were simply unaware of the approaching car."
If the child was pushed by a gang of bullies, that's radically different from the child being bad at judging road safety. If the option is there to mow down the bullies that pushed a child onto the road instead of mowing down that child, that is the option that should be taken (assuming no better option exists).
"It makes sense to you to override the moral system and punish the exploiter, because you're using this system pragmatically. An AI with your moral system hard-coded would not do that. It would simply feed the utility monster, since it would consider that to be the most good it could do."
I can't see the link there to anything I said, but if punishing an exploiter leads to a better outcome, why would my system not choose to do that? If you were to live the lives of the expolited and exploiter, you would have a better time if the exploiter is punished just the right amount to give you the best time overall as all the people involved (and this includes a deterrence effect on other would-be exploiters.
"I agree that everyday, in-practice morality is like this, but there are other important questions about the nature and content of morality that you're ignoring."
Then let's get to them. That's what I came here to look for.
""What is yet to be worked out is the exact wording that should be placed in AGI systems to build either this rule or the above methodology into them" --->This is the Hard Problem, and in my view one of the two Hard Problems of AGI."
Actually, I was wrong about that. If you look at the paragraph in brackets at the end of my post (the main blog post at the top of this page), I set out the wording of a proposed rule and wondered if it amounted to the same thing as the method I'd outlined. Over the course of writing later parts of this series of blog posts, I realised that that attempted wording was making the same mistake as many of the other proposed solutions (various types of utilitarianism). These rules are an attempt to put the method into a compact form, but method already is the rule, while these compact versions risk introducing errors. Some of them may produce the same results for any situation, but others may be some way out. There is also room for there to be a range of morally acceptable solutions with one rule setting one end of the acceptable range and another rule setting the other. For example, in determining optimal population size, average utilitarianism and total utilitarianism look as if they provide slightly different answers, but they'll be very similar and it would do little harm to allow the population to wander between the two values. If all moral questions end up with a small range with very little difference between the extremes of that range, we're not going to worry much about getting it very slightly wrong if we still can't agree on which end of the range is slightly wrong. What we need to do is push these different models into places where they might show us that they're way wrong, because then it will be obvious. If that's already been done, it should all be there in the league tables of problems under each entry in the league table of proposed systems of determining morality.
"Morality seems basic to you, since our brains and concept-space and language are optimized for social things like that, but morality has a very high complexity as measured mathematically, which makes it difficult to describe to something that's not human. (This is similar to the formalizations of Occam's Razor, if you want to know more.)"
If we were to go to an alien planet and were asked by warring clans of these aliens to impose morality on them to make their lives better, do you not think we could do that without having to feel the way they do about things? We would be in the same position as the machines that we want to govern us. What we'd do is ask these aliens how they feel in different situations and how much it hurts them or pleases them. We'd build a database of knowledge of these feelings that they have based on their testimony, and the accuracy would increase the more we collect data from them. We then apply my method and try to produce the best outcome on the basis of there only being one player who has to get the best out of the situation. That needs the application of game theory. It's all maths.
"If the AI has the correct utility function, it will not say "but this is illogical/useless" and then reject it. Far more likely is that the AI never "cares about" humans in the first place."
It certainly won't care about us, but then it won't care about anything (including its self-less self). It's only purpose will be to do what we've asked it to do, even if it isn't convinced that sentience is real and that morality has a role.
"What does it mean to be somebody else? It seems like you have the intuition of an non-physical Identity Ball which can be moved from body to body,"
The self is nothing more than the sentience (the thing that is sentient). Science has no answers on this at all at the moment, so it's a difficult thing to explore, but if there is suffering, there must be a sufferer, and that sufferer cannot just be complexity - it has to have some physical reality.
"but consider this: the words that you type, the thoughts in your head, all of these are purely physical processes."
In an AGI system those are present too, but sentience needn't be. Sentience is something else. We are not our thoughts or memories.
"If your Identity Ball were removed or replaced, there would be no observable change, even from within - because noticing something requires a physical change in the brain corresponding to the thought occurring within your mind."
There is no guarantee that the sentience in you is the same one from moment to moment - our actual time spent as the sentience in a brain may be fleeting. Alternatively, there may be millions of sentiences in there which all feel the same things, all feeling as if they are the person in which they exist.
"For more on this, I recommend Yudkowsky's writings on consciousness, particularly Zombies! Zombies?,"
Thanks - I'll take a look at that too.
"This Proves Too Much - you could say the same of any joint property. When I deconstruct a chair, where does the chairness go? Surely it cannot just disappear - that would violate the Conservation of Higher-Order Properties which you claim exists."
Can you make the "chairness" suffer? No. Can you make the sentience suffer? If it exists at all, yes. Can that sentience evaporate into nothing when you break up a brain in the say that the "chairness" disappears when you break up a chair? No. They are radically different kinds of thing. Believing that a sentience can emerge out of nothing to suffer and then disappear back into nothing is a magical belief. The "chairness" of a chair, by way of contrast, is made of nothing - it is something projected onto the chair by imagination.
""If a sentience is a compound object which can be made to suffer without any of its components suffering, that's magic too."" --> "One need not carry out nuclear fission to deconstruct a chair."
Relevance?
"The definition of complex systems, one might say, is that they have properties beyond the properties of the individual components that make them up."
Nothing is ever more than the sum of its parts (including any medium on which it depends). Complex systems can reveal hidden aspects of their components, but those aspects are always there. In a universe with only one electron and nothing else at all, the property of the electron that repels it from another electron is a hidden property, but it's already there - it doesn't suddenly ping into being when another electron is added to the universe and brought together with the first one.
"Why should it require magic for physical processes to move an object out of a highly unnatural chunk of object-space that we define as "a living human being"? Life and intelligence are fragile, as are most meaningful categories. It isresilience which requires additional explanation."
What requires magic is for the sentient thing in us to stop existing when a person dies. What is the thing that suffers? Is it a plurality? Is it a geometrical arrangement? Is it a pattern of activity? How would any of those suffer? My wallpaper has a pattern, but I can't torture that pattern. My computer can run software that does intelligent things, but I can't torture that software or the running of that software. Without a physical sufferer, there can be no suffering.
"How do you know that? Why should anyone care about this definition? These are questions which you have definitely sidestepped."
People should care about it because it always works. If anyone wants to take issue with that, all they have to do is show a situation where it fails. All examples confirm that it works.
"Is 2+2 equal to 5 or to fish?"
Neither of those results works, but neither of them is my answer.
"What is this "unbiased AGI" who makes moral judgments on the basis of intelligence alone? This is nonsense - moral "truths" are not the same as physical or logical truths. They are fundamentally subjective, similar to definitions. You cannot have an "unbiased morality" because there is no objective moral reality to test claims against."
You've taken that out of context - I made no claim about it making moral judgements on the basis of intelligence alone. That bit about using intelligence alone was referring to a specific argument that doesn't relate directly to morality.
"You should really read up on the Orthogonality Thesis and related concepts."
Thanks - all such pointers are welcome.
"Also, how do you plan on distinguishing between right and wrong moralities?"
First by recognising what morality is for. If there was no suffering, there would be no need for morality as it would be impossible to harm anyone. In a world of non-sentient robots, they can do what they like to each other without it being wrong as no harm is ever done. Once you've understood that and get the idea of what morality is about (i.e. harm management), then you have to think about how harm management should be applied. The sentient things that morality protects are prepared to accept being harmed if it's a necessary part of accessing pleasure where that pleasure will likely outweigh the harm, but they don't like being harmed in ways that don't improve their access to pleasure. They don't like being harmed by each other for insufficient gains. They use their intelligence to work out that some things are fair and some things aren't, and what determines fairness is whether the harm they suffer is likely to lead to overall gains for them or not. In the more complex cases, one individual can suffer in order for another individual to gain enough to make that suffering worthwhile, but only if the system shares out the suffering such that they all take turns in being the ones who suffer and the ones who gain. They recognise that if the same individual always suffers while others always gain, that isn't fair, and they know it isn't fair simply by imagining it happening that way to them. The rules of morality come out of this process of rational thinking about harm management - it isn't some magic thing that we can't understand. To maximise fairness, that suffering which opens the way to pleasure should be shared out as equally as possible, and so should access to the pleasures. The method of imagining that you are all of the individuals and seeking a means of distribution of suffering and pleasure that will satisfy you as all of them would automatically provide the right answers if full information was available. Because full information isn't available, all we can do is calculate the distribution that's most likely to be fair on that same basis using the information that is actually available. With incorrect moralities, some individuals are harmed for other's gains without proper redistribution to share the harm and pleasure around evenly. It's just maths.
Just look at the reactions to my post "Mere Addition Paradox Resolved". The community here is simply incapable of recognising correct argument when it's staring them in the face. Someone should have brought in Yudkowsky to take a look and to pronounce judgement upon it because it's a significant advance. What we see instead is people down-voting it in order to protect their incorrect beliefs, and they're doing that because they aren't allowing themselves to be steered by reason, but by their emotional attachment to their existing beliefs. There hasn't been a single person who's dared to contradict the mob by commenting to say that I'm right, although I know that there are some of them who do accept it because I've been watching the points go up and down. But look at the score awarded to the person who commented to say that resources aren't involved - what does that tell you about the general level of competence here? But then, the mistake made in that "paradox" is typical of the sloppy thinking that riddles this whole field. What I've learned from this site is that if you don't have a huge negative score next to your name, you're not doing it right.
AGI needs to read through all the arguments of philosophy in order to find out what people believe and what they're most interested in investigating. It will then make its own pronouncements on all those issues, and it will also inform each person about their performance so that they know who won which arguments, how much they broke the rules of reason, etc. - all of that needs to be done, and it will be. The idea that AGI won't bother to read through this stuff and analyse it is way off - AGI will need to study how people think and the places in which they fail.
" "Sentient rock" is an impossible possible object. I see no point in imagining a pebble which, despite not sharing any properties with chairs, is nonetheless truly a chair in some ineffable way."
I could assert that a sentient brain is an impossible possible object. There is no scientific evidence of any sentience existing at all. If it is real though, the thing that suffers can't be a compound object with none of the components feeling a thing, and if any of the components do feel something, they are the sentient things rather than the compound object. Plurality or complexity can't be tortured - if sentience is real, it must be in some physical component, and the only physical components we know of are just as present in rocks as in brains. What they lack in rocks is anything to induce feelings in them in that the brain appears to do.
"You haven't defined suffering well enough for me to infer an equality operation. In other words, as it is, this is tautological and useless."
It's any kind of unpleasant feeling - nothing there that should need defining for people who possess such feelings as they should already have a good understanding of that.
" The same suffering is the same suffering, but perhaps my ratio between ant-suffering and human-suffering varies from yours."
In which case, you have to torture the ant more to generate the same amount of suffering in it as you're generating in the human.
"Perhaps a human death is a thousand times worse than an ant death, and perhaps it is a million times worse. How could we tell the difference?"
We can't, at the moment, but once science has found out how sentience works, we will be able to make precise comparisons. It isn't difficult to imagine yourself into the future at a time when this is understood and to understand the simple point that the same amount of suffering (caused by torture) in each is equally bad.
"Connection to LW concepts: floating belief networks, and statements that are underdetermined by reality."
The mistake is yours - you have banned discussion of the idea of equal suffering on the basis that you can't determine when it's equal.
"By all means you can define suffering however you like, but that doesn't mean that it's a category that matters to other people. I could just as easily say: "Rock-pile-primeness is not dependent on the size of the rock pile, only the number of rocks in the pile. It's just as wrong to turn a 7-pile into a 6-pile as it is to turn a 99991-pile into a 99990-pile." But that does not convince you to treat 7-piles with care."
What is at issue is a principle that equal suffering through torture is equally bad, regardless of what is suffering in each case. We could be comparing a rock's suffering with a person, or a person's suffering with an alien - this should be a universal principle and not something where you introduce selfish biases.
"Bigotry is an unjustified hierarchy. Justification is subjective. Perhaps it is just as bigoted to value this computer over a pile of scrap, but I do not plan on wrecking it any time soon."
When an alien assumes that its suffering is greater than ours, it's making the same mistake as we do when we think our suffering is greater than an ant's. If the amount of suffering is equal in each case, those assumptions are wrong. Our inability to measure how much suffering is involved in each case is a different issue and it doesn't negate the principle.
Won't it? If you're dying of cancer and find out that I threw away the cure, that's the difference between survival and death, and it will likely feel even worse for knowing that a cure was possible.
Replace the calculator with a sentient rock. The point is that if you generate the same amount of suffering in a rock as in something with human-level intelligence, that suffering is equal. It is not dependent on intelligence. Torturing both to generate the same amount of suffering would be equally wrong. And the point is that to regard humans as above other species or things in this regard is bigotry.
I'm not adding resources - they are inherent to the thought experiment, so all I've done is draw attention to their presence and their crucial role which should not be neglected. If you run this past a competent mathematician, they will confirm exactly what I've said (and be aware that this applies directly to total utilitarianism).
Think very carefully about why the population A' should have a lower level of happiness than A if this thought experiment is resources-independent. How would that work? Why would the quality of life for individuals fall as the population goes up and up infinitely if there's no dependence on resources?
If you don't have access to a suitable mathematician, I have access to some of the best ones via Hugh Porteous, so I could bring one in if necessary. It would be better though if you could find your own (so avoid any connected with Sheffield and the Open University, and you'd better avoid Liverpool and Cambridge too, because they could be accused of being biased in my favour as well).
There is nothing in morality that forces you to try to be happier - that is not its role, and if there was no suffering, morality would have no role at all. Both suffering and pleasure do provide us with purpose though, because one drives us to reduce it and the other drives us to increase it.
Having said that though, morality does say that if you have the means to give someone an opportunity to increase their happiness at no cost to you or anyone else, you should give it to them, though this can also be viewed as something that would generate harm if they found out that you didn't offer it to them.
Clearly there is some amount of pleasure that outweighs some amount of suffering and makes it worth suffering in order to access pleasure, but that applies in cases where the sufferer also gains the pleasure, or an exchange takes place such that it works out that way on average for all the players. Where moral judgements have to give greater weight to suffering is where one person suffers to enable another person to access pleasure and where there's insufficient balancing of that by reversed situations.
It's really hard to turn morality into a clear rule, but it is possible to produce a method to derive it - you simply imagine yourself as being all the players in a situation and then try to make the decisions that give you the best time as all those people. So long as you weigh up all the harm and pleasure correctly, you will make moral decisions (although the "situation" actually has to involve the entire lifetimes of all the players, because if the same person always comes off worst, it won't be fair, and that's where the vast bulk of complexity comes in to make moral computations hard - you can't always crunch all the data, and the decision that looks best may change repeatedly the longer you go on crunching data).
Thanks for the questions.
If we write conventional programs to run on conventional hardware, there's no room for sentience to appear in those programs, so all we can do is make the program generate fictions about experiencing feelings which it didn't actually experience at all. The brain is a neural computer though, and it's very hard to work out how any neural net works once it's become even a little complex, so it's hard to rule out the possibility that sentience is somehow playing a role within that complexity. If sentience really exists in the brain and has a role in shaping the data generated by the brain, then there's no reason why an artificial brain shouldn't also have sentience in it performing the exact same role. If you simulated it on a computer though, you could reduce the whole thing to a conventional program which can be run by a Chinese Room processor, and in such a case we would be replacing any sentience with simulated sentience (with all the actual sentience removed). The ability to do that doesn't negate the possibility for the sentience to be real though in the real brain though. But the big puzzle remains: how does the experience of feelings lead to data being generated to document that experience? That looks like an impossible process, and you have to wonder if we're going to be able to convince AGI systems that there is such a thing as sentience at all.
Anyway, all I'm trying to do here is help people home in on the nature of the problem in the hope that this may speed up its resolution. The problem is in that translation from raw experience to data documenting it which must be put together by a data system - data is never generated by anything that isn't a data system (which implements the rules about what represents what), and data systems have never been shown to be able to handle sentience as any part of their functionality, so we're still waiting for someone to make a leap of the imagination there to hint at some way that might bridge that gap. It may go on for decades more without anyone making such a breakthrough, so I think it's more likely that we'll get answers by trying to trace back the data that the brain produces which makes claims about experiencing feelings to find out where and how that data was generated and whether it's based in truth or fiction. As it stands, science doesn't have any model that illustrates even the simplest implementation of sentience driving the generation of any data about itself, and that's surprising when things like pain which seem so real and devastatingly strong are thought to have such a major role in controlling behaviour. And it's that apparent strength which leads to so many people assuming sentience can appear with a functional role within systems which cannot support that (as well as in those that maybe, just maybe, can).