LessWrong 2.0 Reader

I think Abram is saying the following:

• Currently, AIs are lacking capabilities that would meaningfully speed up AI Safety research.
• At some point, they are gonna get those capabilities.
• However, by default, they are gonna get those AI Safety-helpful capabilities roughly at the same time as other, dangerous capabilities (or at least, not meaningfully earlier).
  • In which case, we're not going to have much time to use the AI Safety-helpful capabilities to speed up AI Safety research sufficiently for us to be ready for those dangerous capabilities.
• Therefore, it makes sense to speed up the development of AIS-helpful capabilities now. Even if it means that the AIs will acquire dangerous capabilities sooner, it gives us more time to use AI Safety-helpful capabilities to prepare for dangerous capabilities.

I'm very much in agreement that this is a problem, and among other things blocks us from knowing how to use adversarial attack methods (and AISI teams!) from helping here. Your proposed definition feels like it might be an important part of the story but not the full story, though, since it's output only: I would unfortunately expect a decent probability of strong jailbreaks that (1) don't count as intent misalignment but (2) jump you into that kind of red attractor basin. Certainly ending up in that kind of basin could cause a catastrophe, and I would like to avoid it, but I think there is a meaningful notion of "the AI is unlikely to end up in that basin of its own accord, under nonadversarial distributions of inputs".

Have you seen good attempts at input-side definitions along those lines? Perhaps an ideal story here would be a combination of an input-side definition and the kind of output-side definition you're pointing at.

Also, the solution is obviously to friendship is optimal the system that humans and AI coordinate in. Create an opt-in secure system that allows more resources if you cooperate and you will be able to outperform those silly defectors.

When it comes to solutions I think that humans versus AI axis doesn't make sense for the systems that we're in, it is rather about desirable system properties such as participation, exploration and caring for the participants in the system. 

If we can foster a democratic, caring, open-ended decision making process where humans and AI can converge towards optimal solutions then I think our work is done. 

Human disempowerment is okay as long as it is replaced by a better and smarter system so whilst I think the solutions are pointing in the right direction, the main axis of validation should rather be around system properties and not power distribution. 

Good summary though, it is great that we finally have a great paper to point towards for these problems.

Well, the government could take all your property away at a whim; millions of people have experienced that. Plus there are things like eminent domain.

Also, the government can tax your property (and punish you if you don't pay e.g. by taking away that thing), which is kinda weird philosophically... on one hand, something already belongs to you, on the other hand, you need to keep paying to keep it so -- but this is how renting is supposed to work, not owning.

A more obvious example is the intellectual property. There are no Schelling points; twenty years of copyright versus hundred years, there is no number that seems obviously correct.

And if after Singularity, the superintelligent AIs decide that they won't respect human property claims (but may respect each other's property), then... that's just the new reality. Just like a human ignores how the ants in his garden have distributed the territory among themselves.

So the fact that you have a property at all is downstream from "there is a consensus among those with power that this definition of 'property' is generally a good thing to have".

There is the way these things are generally done in the Western civilization, but if you asked some people living in tribes, they probably would have wildly different intuitions about what a person can own.

I can only own something if I am strong enough to defend it, or if there is a consensus among the others that they should let me keep it.

I interpreted it as: not by "usual means", but rather something like suicide or murder.

Logic simply preserves truth. You can arrive to a valid conclusion that one should act altruistically if you start from some specific premises, and can't if you start from some other presimes.

What are the premises you start from?

First and foremost, I totally agree with your point on this sort of thing being instrumentally useful, I'm still having issues seeing how to apply it to my real life. Here are two questions that arise for me:

I'm curious about two aspects of deliberate practice that seem interconnected:

1. On OODA loops: I currently maintain yearly, quarterly, weekly, and daily review cycles where I plan and reflect on progress. However, I wonder if there are specific micro-skills you're pointing to beyond this - perhaps noticing subtle emotional tells when encountering uncomfortable topics, or developing finer-grained feedback mechanisms. How does this type of systematic review practice fit into your framework for deliberate practice? Are there particular refinements or additional elements you'd recommend? Is it noticing when I'm not doing OODA?
2. On unlearning: While your post focuses extensively on learning practices, I'm interested in your thoughts on "unlearning" - the process of identifying and releasing ineffective patterns or beliefs. In my experience with meditation, there seems to be a distinction between intellectual understanding and emotional understanding, where sometimes what holds us back isn't insufficient practice but rather old patterns that need to be examined and released. How do you see the relationship between building new skills and creating space for new patterns through deliberate unlearning? One of the sayings I've heard said is that "meditation is the process of taking intellectual understanding and turning it into emotional understanding" which I find quite interesting.

Yeah, I don’t think this phenomenon requires any deliberate strategic intent to deceive / collude. It’s just borne of having a subtle preference for how things should be said. As you say, humans probably also have these preferences

You could interpret that as "rationalists are moving to the left... expect more leftism in the future", or you could interpret that as "rationalists are choosing the parts that seem correct to them (and rejecting the parts that don't)... expect more cherry-picking from all directions in the future".