LessWrong 2.0 Reader

What's the desired outcome of this debate?  Are you looking for cruxes (axioms or modeling choices that lead to the disagreement, separate from resolvable empirical measurements that you don't disagree on)?  Are you hoping to update your own beliefs, or to convince your partner (or readers) to update theirs?

I do not necessarily endorse my comments in this piece.

That's likely to need some explanation about why it's valuable to put such comments on LessWrong.  It's fine to put non-endorsed views here, but they should be labeled as to why they're worth mentioning.  Putting misleading or known-suspect arguments-as-soldiers on LW, especially mixed in with things you DO support, is a mistake.

I also think the name is off, but for a different reason. When I hear "the west" with no other context, I assume it means this, which doesn't make sense here, because the UK and FHI are very solidly part of The West. (I have not heard the "Harvard of the west" phrase and I'm guessing it's pretty darn obscure, especially to the international audience of LW.)

I'm not sure. My second thoughts were eg, 'Interactions with the media often don't go the way people expected' and 'Sensationalizable research often gets spun into pre-existing narratives and can end up having net-negative consequences.' It's possible that my original suggestion makes sense, but my uncertainty is high enough that on reflection I'm not comfortable endorsing it, especially given my own lack of experience dealing with the media.

Suddenly more poignant: 

If in extreme situations the ethical ideas fall apart, it might make sense to add an extra rule to stay away from the extreme situations. Like maybe not forever, but to proceed sufficiently slowly so that we have time to reflect on how we feel about that.

Thanks for the post. I don’t know the answer to whether a self-consistent ethical framework can be constructed, but I’m working [EA · GW] on it [LW · GW] (without funding). My current best framework is a utilitarian one with incorporation of the effects of rights, self-esteem (personal responsibility) and conscience. It doesn’t “fix” the repugnant or very repugnant conclusions, but it says how you transition from one world to another could matter in terms of the conscience(s) of the person/people who bring it about.

It’s an interesting question as to what the implications are if it’s impossible to make a self-consistent ethical framework. If we can’t convey ethics to an AI in a self-consistent form, then we’ll likely rely in part on giving it lots of example situations (that not all humans/ethicists will agree on) to learn from and hope it’ll augment this with learning from human behavior, and then generalize well to outside all this not perfectly consistent training data. (Sounds a bit sketchy, doesn't it - at least for the first AGI's, but perhaps ASI [LW · GW]'s could fare better?) Generalize "well” could be taken to mean that an AI won’t do anything that most people would strongly disapprove of if they understood the true implications of the action.

[This paragraph I'm less sure of, so take it with a grain of salt:] An AI that was trying to act ethically and taking the approval of relatively wise humans as some kind of signal of this might try to hide/avoid ethical inconsistencies that humans would pick up on. It would probably develop a long list of situations where inconsistencies seemed to arise and of actions it thought it could "get away with" versus not. I'm not talking about deception with malice, just sneakiness to try to keep most humans more or less happy, which, I assume would be part of what its ethics system would deem as good/valuable. It seems to me that problems may come to the surface if/when an "ethical" AI is defending against bad AI, when it may no longer be able to hide inconsistencies in all the situations that could rapidly come up. 

If it is possible to construct a self-consistent ethical framework and we haven't done it in time or laid the groundwork for it to be done quickly by the first "transformative" AI's, then we'll have basically dug our own grave for the consequences we get, in my opinion. Work to try to come up with a self-consistent ethical framework seems to me to be a very under explored area for AI safety. 

To me as a programmer and not a mathematitian, the distinction doesn't make practical intuitive sense.

If we can create 3 functions f, g, h so that they "do the same thing" like f(a, b, c) == g(a)(b)(c) == average(h(a), h(b), h(c)), it seems to me that cross-entropy can "do the same thing" as some particular objective function that would explicitly mention multiple future tokens.

My intuition is that cross-entropy-powered "local accuracy" can approximate "global accuracy" well enough in practice that I should expect better global reasoning from larger model sizes, faster compute, algorithmic improvements, and better data.

Implications of this intuition might be:

• myopia is a quantity not a quality, a model can be incentivized to be more or less myopic, but I don't expect it will be proven possible to enforce it "in the limit"
• instruct training on longer conversations outght to produce "better" overall conversations if the model simulates that it's "in the middle" of a conversation and follow-up questions are better compared to giving a final answer "when close to the end of this kind of conversation"

What nuance should I consider to understand the distinction better?

I like the rest of the article, but...

Cold calls. It's ok if you have a terrible response rate.

It's ok for you, but you generate negative externality as a side effect (waste other people's time and attention).

I think that insisting on comparing unmeasurable and different things is an error.  If forced to do so, you can make up whatever numbers you like, and nobody can prove you wrong.  If you make up numbers that don't fully contradict common intuitions based on much-smaller-range and much-more-complicated choices, you can probably convince yourself of almost anything.

Note that on smaller, more complicated, specific decisions, there are many that seem to be inconsistent with this comparison: some people accept painful or risky surgery over chronic annoyances, some don't.  There are extremely common examples of failing to mitigate pretty serious harm for distant strangers, in favor of mild comfort for oneself and closer friends/family (as well as some examples of the reverse).  There are orders of magnitude in variance, enough to overwhelm whatever calculation you think is universal.

Do you think a logarithmic scale makes more sense than a linear scale?

Assuming that this article is a reaction to "Torture vs. Dust Specks [LW · GW]", the hypothetical number of people suffering from dust specks was specified as 3^^^3, which in practice is an unimaginably large number. Big numbers such as "the number of particles in the entire known universe" are not sufficient even to describe its number of digits. Therefore, using a logarithmic scale changes nothing.

Logarithmic scale with a hard cap is an inelegant solution, comparable to a linear scale with a hard cap.

What you probably want instead is some formula like in the theory of relativity, where the speed of a rocket approaches but never reaches a certain constant c. For example, you might claim that if a badness of any specific thing is X, then the badness of this thing happening even to a practically infinite number of people is still only approaching some finite value C*X. (Not sure if C is constant across different kinds of suffering.)

That seems like a nice justification for scope insensitivity [? · GW]. We are not insensitive, it's just that saving 2,000 birds or saving 200,000 birds really has approximately the same moral value!

The problem with this justification is what qualifies as the "same kind of suffering". Suppose that infinite people getting a dust speck in their eyes aggregates into 1000 units of badness. If instead, an infinite number people get a dust speck in their left eyes, and an infinite number of different people get a dust speck in their right eyes, does this aggregate into 1000 or 2000 units of badness, and why? What about dusk specks vs sand specks?

Or is this supposed to aggregate over different kinds of suffering? So even an almost infinite number of people, each one mildly discomforted in a unique way, are a less bad outcome than one person suffering horribly?

...shortly, it is not enough to say "in this specific scenario, I would define the proper way to calculate utility this way", you should provide a complete theory, and then see how well it works in other scenarios.

(Also, you need to consider practically infinitely small numbers of people -- that is, people suffering certain fate with a microscopically tiny probability.)