LessWrong 2.0 Reader

You mean, "ban superintelligence"? Because superintelligences are not human-like.

That's the problem with your proposal of "ethics module". Let's suppose that we have system of "ethics module" and "nanotech design module". Nanotech design module outputs 3D-model of supramolecular unholy abomination. What exactly should ethics module do to ensure that this abomination doesn't kill everyone? Tell nanotech module "pls don't kill people"? You are going to have hard time translating this into nanotech designer internal language. Make ethics module sufficiently smart to analyse behavior of complex molecular structures in wide range of environments? You have now all problems with alignment of superintelligences.

Second part being "there will be an AISC10"?

Very sure. 

As long as me and Remmelt are still alive and healthy a couple of months from now, then we're doing it.

Remmelt have organised 8 previous AISCs, I've been part of 3 of those. We know what we are doing. We know we can rely on each other, and we want to do this.

We just needed to make sure we have money to live and eat and such, before we could commit to running a next camp. But we have received the money now, so that's all good. Manifund have sent us the money, it's in our bank accounts. 

I'll bet anyone who like, that there will be an AISC10 at 1:10 odds in your favour. I'm much more confident that that, but if you give me worse odds, then I don't think can be bothered about it.

Rob Long works on these topics.

indeed. But still… if she wonders out loud “what day is it?” at the very moment she says that, it has an answer.

There is no "but". As long as the Beauty is unable to distinguish between Monday and Tuesday awakenings, as long as the decision process which leads her to say the phrase "what day is it" works the same way, from her perspective there is no one "very moment she says that". On Tails, there are two different moments when she says this, and the answer is different for them. So there is no answer for her

An experimenter who overhears her knows the answer. It seems to me that you “resolve” this tension is that the two of them are technically asking a different question, even though they are using the same words

Yes, you are correct. From the position of the experimenter, who knows which day it is, or who is hired to work only on one random day this is a coherent question with an actual answer. The words we use are the same but mathematical formalism is different. 

For an experimenter who knows that it's Monday the probability that today is Monday is simply:

P(Monday|Monday) = 1

For an experimenter who is hired to work only on one random day it is:

P(Monday|Monday xor Tuesday) = 1/2

But still… how surprised should she be if she were to learn that today is Monday? It seems that taking your stance to its conclusion, the answer would be “zero surprise: she knew for sure she would wake up on Monday so no need to be surprised it happened”

And even if she were to learn that the coin landed tails, so she knows that this is just one of a total of two awakenings, she should have zero surprise upon learning the day of the week, since she now knows both awakenings must happen.

Completely correct. Beauty knew that she would be awaken on Monday either way and so she is not surprised. This is a standard thing with non-mutually exclusive events. Consider this:

A coin is tossed and you are put to sleep. On Heads there will be a red ball in your room. On Tails there will be a red and a blue ball in your room. How surprised should you be to find a red ball in your room?

Which seems to violate conservation of expected evidence, except you already said that the there’s no coherent probabilities here for that particular question, so that’s fine too.

The appearance of violation of conservation of expected evidence comes from the belief that awakening on Monday and on Tuesday are mutually exclusive, while they are, in fact sequential.

This makes sense, but I’m not used to it. For instance, I’m used to these questions having the same answer:

1. P(today is Monday)?
2. P(today is Monday | the sleep lab gets hit by a tornado)

Yet here, the second question is fine (assuming tornadoes are rare enough that we can ignore the chance of two on consecutive days) while the first makes no sense because we can’t even define “today”

It makes sense but it’s very disorienting, like incompleteness theorem level of disorientation or even more

I completely understand. It is counterintuitive because evolution didn't prepare us to deal with situations where an experience is repeated the same while we receive memory loss. As I write in the post:

If I forget what is the current day of the week in my regular life, well, it's only natural to start from a 1/7 prior per day and work from there. I can do it because the causal process that leads to me forgetting such information can be roughly modeled as a low probability occurrence which can happen to me at any day. 

It wouldn't be the case, if I was guaranteed to also forget the current day of the week on the next 6 days as well, after I forgot it on the first one. This would be a different causal process, with different properties - causation between forgetting - and it has to be modeled differently. But we do not actually encounter such situations in everyday life, and so our intuition is caught completely flat footed by them.

The whole paradox arises from this issue with our intuition, and just like with incompleteness theorem  (thanks for the flattering comparison, btw), what we need to do now is to re-calibrate our intuitions, make it more accustomed to the truth, preserved by the math, instead of trying to fight it.

"here is how to make LLMs more capable but less humanlike, it will be adopted because it makes LLMs more capable". 

Thankfully, this is a class of problems that humanity has an experience dealing with. The solution boils down to regulating all the ways to make LLMs less human-like out of existence.

Good to see you! 

I'm noticing two things:

1. It's suspicious to me that values of humans-who-like-paperclips are inherently tied to acquiring an unlimited amount of resources (no matter in which way). Maybe I don't treat such values as 100% innocent, so I'm OK keeping them in check. Though we can come up with thought experiments where the urge to get more resources is justified by something. Like, maybe instead of producing paperclips those people want to calculate Busy Beaver numbers, so they want more and more computronium for that.
2. How consensual were the trades if their outcome is predictable and other groups of people don't agree with the outcome? Looks like coercion.

I feel like I am a victim of transparency illusion. First part of OP argument is "LLMs need data, data is limited and synthetic data is meh". Direct counterargument to this is "here is how to avoid drawbacks of sythetic data". Second part of OP argument is "LLMs are humanlike and will remain so", and direct counterargument is "here is how to make LLMs more capable but less humanlike, it will be adopted because it makes LLMs more capable". Walking around telling everyone ideas of how to make AI more capable and less alignable is pretty much ill-adviced.

Adversarial examples suggest to me that by default ML systems don't necessarily learn what we want them to learn:

1. They put too much emphasis on high frequency features, suggesting a different inductive bias from humans.
2. They don't handle contradictory evidence in a reasonable way, i.e., giving a confident answer when high frequency features (pixel-level details) and low frequency features (overall shape) point to different answers.

The evidence from adversarial training suggests to me that AT is merely patching symptoms (e.g., making the ML system de-emphasize certain specific features) and not fixing the underlying problem. At least this is my impression from watching this video on Adversarial Robustness, specifically the chapters on Adversarial Arms Race and Unforeseen Adversaries.

Aside from this, it's also unclear how to apply AT to your original motivation:

A function that tells your AI system whether an action looks good and is right virtually all of the time on natural inputs isn’t safe if you use it to drive an enormous search for unnatural (highly optimized) inputs on which it might behave very differently.

because in order to apply AT we need a model of what "attacks" the adversary is allowed to do (in this case the "attacker" is a superintelligence trying to optimize the universe, so we have to model it as being allowed to do anything?) and also ground-truth training labels.

For this purpose, I don't think we can use the standard AT practice of assuming that any data point within a certain distance of a human-labeled instance, according to some metric, has the same label as that instance. Suppose we instead let the training process query humans directly for training labels (i.e., how good some situation is) on arbitrary data points, well that's slow/costly if the process isn't very sample efficient (which modern ML isn't), and also scary if human implementations of human values may already have adversarial examples. (The "perceptual wormholes" work and other evidence suggest that humans also aren't 100% adversarially robust.)

My own thinking is that we probably need to go beyond adversarial training for this, along the lines of solving metaphilosophy [LW · GW] and then using that solution to find/fix existing adversarial examples and correctly generalize human values out of distribution.

With every technology there is a way to make it stop working. There are any number of ways to make a plane unable to fly. But the important thing is that we know a way to make a plane fly - therefore humans can fly via a plane.

Likewise, the point that LLM-based-architecture can in principle be safe still stands even if there is a way to make an unsafe LLM-based-architecture. 

And this is a huge point. Previously we were in a state where alignment wasn't even a tractable problem. Where capabilities progressed and alignment stayed in the dirt. Where AI system may understand human values but still not care about them and we didn't know what to do with it.

But now we can just

have an “ethics module” where the underlying LLM produces text which then feeds into other parts of the system to help guide behavior.

Which makes alignment tractable. Alignment can now be reduced to capability of the ethics module. We know that the system will care about our values as it understands them because we can explicitly code it to do this way via an if-else statement. This is an enormous improvement over the previous status quo.