LessWrong 2.0 Reader

Actually I guess that's kind of trivial (the belief state geometry should be tensored together). Maybe a more interesting question is what happens if you use a Markov chain to transduce it.

Interestingly, could similar behavior be used to explain some cases of human schizophrenia when word salad is produced. 

What's the desired outcome of this debate?  Are you looking for cruxes (axioms or modeling choices that lead to the disagreement, separate from resolvable empirical measurements that you don't disagree on)?  Are you hoping to update your own beliefs, or to convince your partner (or readers) to update theirs?

I do not necessarily endorse my comments in this piece.

That's likely to need some explanation about why it's valuable to put such comments on LessWrong.  It's fine to put non-endorsed views here, but they should be labeled as to why they're worth mentioning.  Putting misleading or known-suspect arguments-as-soldiers on LW, especially mixed in with things you DO support, is a mistake.

I also think the name is off, but for a different reason. When I hear "the west" with no other context, I assume it means this, which doesn't make sense here, because the UK and FHI are very solidly part of The West. (I have not heard the "Harvard of the west" phrase and I'm guessing it's pretty darn obscure, especially to the international audience of LW.)

I'm not sure. My second thoughts were eg, 'Interactions with the media often don't go the way people expected' and 'Sensationalizable research often gets spun into pre-existing narratives and can end up having net-negative consequences.' It's possible that my original suggestion makes sense, but my uncertainty is high enough that on reflection I'm not comfortable endorsing it, especially given my own lack of experience dealing with the media.

Suddenly more poignant: 

If in extreme situations the ethical ideas fall apart, it might make sense to add an extra rule to stay away from the extreme situations. Like maybe not forever, but to proceed sufficiently slowly so that we have time to reflect on how we feel about that.

Thanks for the post. I don’t know the answer to whether a self-consistent ethical framework can be constructed, but I’m working [EA · GW] on it [LW · GW] (without funding). My current best framework is a utilitarian one with incorporation of the effects of rights, self-esteem (personal responsibility) and conscience. It doesn’t “fix” the repugnant or very repugnant conclusions, but it says how you transition from one world to another could matter in terms of the conscience(s) of the person/people who bring it about.

It’s an interesting question as to what the implications are if it’s impossible to make a self-consistent ethical framework. If we can’t convey ethics to an AI in a self-consistent form, then we’ll likely rely in part on giving it lots of example situations (that not all humans/ethicists will agree on) to learn from and hope it’ll augment this with learning from human behavior, and then generalize well to outside all this not perfectly consistent training data. (Sounds a bit sketchy, doesn't it - at least for the first AGI's, but perhaps ASI [LW · GW]'s could fare better?) Generalize "well” could be taken to mean that an AI won’t do anything that most people would strongly disapprove of if they understood the true implications of the action.

[This paragraph I'm less sure of, so take it with a grain of salt:] An AI that was trying to act ethically and taking the approval of relatively wise humans as some kind of signal of this might try to hide/avoid ethical inconsistencies that humans would pick up on. It would probably develop a long list of situations where inconsistencies seemed to arise and of actions it thought it could "get away with" versus not. I'm not talking about deception with malice, just sneakiness to try to keep most humans more or less happy, which, I assume would be part of what its ethics system would deem as good/valuable. It seems to me that problems may come to the surface if/when an "ethical" AI is defending against bad AI, when it may no longer be able to hide inconsistencies in all the situations that could rapidly come up. 

If it is possible to construct a self-consistent ethical framework and we haven't done it in time or laid the groundwork for it to be done quickly by the first "transformative" AI's, then we'll have basically dug our own grave for the consequences we get, in my opinion. Work to try to come up with a self-consistent ethical framework seems to me to be a very under explored area for AI safety. 

To me as a programmer and not a mathematitian, the distinction doesn't make practical intuitive sense.

If we can create 3 functions f, g, h so that they "do the same thing" like f(a, b, c) == g(a)(b)(c) == average(h(a), h(b), h(c)), it seems to me that cross-entropy can "do the same thing" as some particular objective function that would explicitly mention multiple future tokens.

My intuition is that cross-entropy-powered "local accuracy" can approximate "global accuracy" well enough in practice that I should expect better global reasoning from larger model sizes, faster compute, algorithmic improvements, and better data.

Implications of this intuition might be:

• myopia is a quantity not a quality, a model can be incentivized to be more or less myopic, but I don't expect it will be proven possible to enforce it "in the limit"
• instruct training on longer conversations outght to produce "better" overall conversations if the model simulates that it's "in the middle" of a conversation and follow-up questions are better compared to giving a final answer "when close to the end of this kind of conversation"

What nuance should I consider to understand the distinction better?

I like the rest of the article, but...

Cold calls. It's ok if you have a terrible response rate.

It's ok for you, but you generate negative externality as a side effect (waste other people's time and attention).