A brief theory of why we think things are good or bad

post by David Johnston (david-johnston) · 2024-10-20T20:31:26.309Z · LW · GW · 10 comments

Contents

10 comments

10 comments

Comments sorted by top scores.

comment by Seth Herd · 2024-10-21T03:09:45.566Z · LW(p) · GW(p)

You just have to explain what you mean by "good" or "bad"'. They're very vague terms. Often they mean "things I like" or "things I and those I like like" . "Those I like" could be anywhere between one other person I like a little, and every thing that can think even a little getting equal weight to myself. People mean all of those things by "good". You can guess by context what someone might mean, but if you want to have a clear discussion, it's best to specify.

As for whether those other information-processing systems (like LLMs and bugs) really have opinions about what is good and bad for them in the same rich way humans seem to, that is a separate question.

Replies from: david-johnston
↑ comment by David Johnston (david-johnston) · 2024-10-21T03:47:56.429Z · LW(p) · GW(p)

I think precisely defining "good" and "bad" is a bit beside the point - it's a theory about how people come to believe things are good and bad, and we're perfectly capable of having vague beliefs about goodness and badness. That said, the theory is lacking a precise account of what kind of beliefs it is meant to explain.

The LLM section isn't meant as support for the theory, but speculation about what it would say about the status of "experiences" that language models can have. Compared to my pre-existing notions, the theory seems quite willing to accommodate LLMs having good and bad experiences on par with those that people have.

comment by cubefox · 2024-10-20T23:21:59.634Z · LW(p) · GW(p)

We believe many things because we are somewhat rational; we consider hypotheses, compare them with observed evidence, and emphasise those that are more compatible. Goodness and badness defy this practice; normal reasoning does not produce hypotheses of the form "if X is morally good then I will observe Y".

I disagree. If X is an action, it is usually considered good if it increases welfare, and bad if it decreases welfare. And we can find evidence for or against something being conducive to welfare. So we can find evidence for or against something being good.

Replies from: david-johnston
↑ comment by David Johnston (david-johnston) · 2024-10-21T00:04:36.116Z · LW(p) · GW(p)

I have a pedantic and a non-pedantic answer to this. Pedantic: you say X is "usually considered good" if it increases welfare. Perhaps you mean to imply that if X is usually considered good then it is good. In this case, I refer you to the rest of the paragraph you quote.

Non-pedantic: yes, it's true that once you accept some fundamental assumptions about goodness and badness you can go about theorising and looking for evidence. I'm suggesting that motivated reasoning is the mechanism that makes those fundamental assumptions believable.

I added a paragraph mentioning this, because I think your reaction is probably common.

Replies from: cubefox
↑ comment by cubefox · 2024-10-21T11:49:57.503Z · LW(p) · GW(p)

If I believe eating meat is not bad because I engage in motivated reasoning, then this is, like all forms of motivated reasoning, just an irrational belief. But if I believe eating meat is not bad because I believe it doesn't create a significant amount of additional suffering, there is nothing irrational about that belief. So motivated reasoning can only explain (some) irrational beliefs. Not all beliefs about things being good or bad.

However, when something being bad means that it decreases some sort of welfare in some general way, then we don't have this problem. Now, what exactly does "welfare" etc mean? That's a question that normative ethicists try to figure out. For example via various proposed theories of utilitarianism. If philosophers are analyzing a subject matter, it's safe to assume they are analyzing some concept. Now, what's a concept? It's a meaning of a word. Like "good" or "bad".

Replies from: david-johnston
↑ comment by David Johnston (david-johnston) · 2024-10-21T21:49:37.695Z · LW(p) · GW(p)

Thanks for your continued engagement.

I’m interested in explaining foundational moral beliefs like suffering is bad, not beliefs like “animals do/don’t suffer”, which is about badness only because we accept the foundational assumption that suffering is bad. Is that clear in the updated text?

Now, I don’t think these beliefs come from playing axiomatic games like “define good as that which increases welfare”. There are many lines of evidence for this. First: “define bad as that which increases suffering” is not equally as plausible as “define good as that which increases suffering”. We have pre-existing beliefs about this.

Second: you talk about philosophers analysing welfare. However, the method that philosophers use to do this usually involves analysing a bunch of fundamental moral assumptions. For example, from the Stanford encyclopaedia of philosophy:

Correspondingly, no amount of empirical investigation seems by itself, without some moral assumption(s) in play, sufficient to settle a moral question https://plato.stanford.edu/entries/metaethics/

I am suggesting that the source of these fundamental moral assumptions may not be mysterious - we have a known ability to form beliefs based on what we want, and fundamental moral beliefs often align with what we want.

Replies from: cubefox
↑ comment by cubefox · 2024-10-21T23:58:45.494Z · LW(p) · GW(p)

Problem is, motivated reasoning can only explain selfish beliefs, beliefs which are in accordance with our own motivations. But moral beliefs are often not at all selfish. In contrast, "suffering is bad" could just be part of what "bad" means. No motivated reasoning required. It would be a "foundational belief" in the same sense "Bachelors are unmarried" could be called "foundational".

Replies from: david-johnston, david-johnston
↑ comment by David Johnston (david-johnston) · 2024-10-22T01:46:49.476Z · LW(p) · GW(p)

For what it's worth, one idea I had as a result of our discussion was this:

  • We form lots of beliefs as a result of motivated reasoning
  • These beliefs are amenable to revision due to evidence, reason or (maybe) social pressure
  • Those beliefs that are largely resilient to these challenges are "moral foundations"

So philosophers like "pain is bad" as a moral foundation because we want to believe it + it is hard to challenge with evidence or reason. Laypeople probably have lots of foundational moral beliefs that don't stand up as well to evidence or reason, but (perhaps) are equally attributable to motivated reasoning.

Social pressure is a bit iffy to include because I think lots of people relate to beliefs that they adopted because of social pressure as moral foundations, and believing something because you're under pressure to do so is an instance of motivated reasoning.

I don't think this is a response to your objections, but I'm leaving it here in case it interests you.

↑ comment by David Johnston (david-johnston) · 2024-10-22T01:09:42.086Z · LW(p) · GW(p)

I can explain why I believe bachelors are unmarried: I learned that this is what the word bachelor means, I learned this because it is what bachelor means, and the fact that there's a word "bachelor" that means "unmarried man" is contingent on some unimportant accidents in the evolution of language. A) it is certainly not the result of an axiomatic game and B) if moral beliefs were also contingent on accidents in the evolution of language (I think most are not), that would have profound implications for metaethics.

Motivated belief can explain non-purely-selfish beliefs. I might believe pain is bad because I am motivated to believe it, but the belief still concerns other people. This is even more true when we go about constructing higher order beliefs and trying to enforce consistency among beliefs. Undesirable moral beliefs could be a mark against this theory, but you need more than not-purely-selfish moral beliefs.

I'm going to bow out at this point because I think we're getting stuck covering the same ground.

comment by Mike S (StuffnStuff) (mike-s-stuffnstuff) · 2024-10-22T01:17:58.581Z · LW(p) · GW(p)

A lot of it comes down to timescales and sequences of events, long term vs short term.

"I will incur a little suffering today, but my well-being will be much better tomorrow".

"We need to get rid of the <undesired national or social group> at cost of suffering, but our nation will have bright future as a result".

People can be tricked into doing unspeakable evil to themselves and others if they have incorrect predictions of the future well-being.