LessWrong 2.0 Reader

I agree with your view about organizational problems. Your discussion gave me an idea: Is it possible to shift employees dedicated to capability improvement to work on safety improvement? Set safety goals for these employees within the organization. This way, they will have a new direction and won't be idle, worried about being fired or resigning to go to other companies.

That seems to solve problem #4. Employees quitting becomes much less of an issue, since in any case they would only be able to share knowledge about safety (which is a good thing).

Do you think this plan will be able to solve problems #1, #2, #3 and #5? I think such discussions are very important, because many people (me included) worry much more about organizational side of alignment than about technical side.

I like the simplicity of your alignment plan. But I do wonder how it would be able to deal with 5 "big" organizational problems of alignment (most of my current p(doom) comes specifically from organizational problems):

1. Moving slowly and carefully is annoying. There's a constant tradeoff about getting more done, and elevated risk. Employees who don't believe in the risk will likely try to circumvent or goodhart the security procedures. Filtering for for employees willing to take the risk seriously (or training them to) is difficult. There's also the fact that many security procedures are just security theater. Engineers have sometimes been burned on overzealous testing practices. Figuring out a set of practices that are actually helpful, that your engineers and researchers have good reason to believe in, is a nontrivial task.

2. Noticing when it's time to pause is hard. The failure modes are subtle, and noticing things is just generally hard unless you're actively paying attention, even if you're informed about the risk. It's especially hard to notice things that are inconvenient and require you to abandon major plans.

3. Getting an org to pause indefinitely is hard. Projects have inertia. My experience as a manager, is having people sitting around waiting for direction from me makes it hard to think. Either you have to tell people "stop doing anything" which is awkwardly demotivating, or "Well, I dunno, you figure it out something to do?" (in which case maybe they'll be continuing to do capability-enhancing work without your supervision) or you have to actually give them something to do (which takes up cycles that you'd prefer to spend on thinking about the dangerous AI you're developing). Even if you have a plan for what your capabilities or product workers should do when you pause, if they don't know what that plans is, they might be worried about getting laid off. And then they may exert pressure that makes it feel harder to get ready to pause. (I've observed many management decisions where even though we knew what the right thing to do was, conversations felt awkward and tense and the manager-in-question developed an ugh field around it, and put it off)

4. People can just quit the company and work elsewhere if they don't agree with the decision to pause. If some of your employees are capabilities researchers who are pushing the cutting-edge forward, you need them actually bought into the scope of the problem to avoid this failure mode. Otherwise, even though "you" are going slowly/carefully, your employees will go off and do something reckless elsewhere.

5. This all comes after an initial problem, which is that your org has to end up doing this plan, instead of some other plan. And you have to do the whole plan, not cutting corners. If your org has AI capabilities/scaling teams and product teams that aren't bought into the vision of this plan, even if you successfully spin the "slow/careful AI plan" up within your org, the rest of your org might plow ahead.

does include the parenting example I gave

A normal and/or healthy parent-child relationship doesn't have the parent extracting as much value as possible from the child regardless of ethics or harm!

is strictly broader than my proposed definition.

Therefore not.

This is cool! I wonder if it can be fixed. I imagine it could be improved some amount by nudging the prefix distribution, but it doesn't seem like that will solve it properly. Curious if this is a large issue in real LMs. It's frustrating that there aren't ground-truth features we have access to in language models.

I think how large of a problem this is can probably be inferred from a description of the feature distribution. It'd be nice to have a better sense of what that distribution is (assuming the paradigm is correct enough).

Yes, I find terminal rationality irrational (I hope my thought experiment helps illustrate that).

I have another formal definition of "rational". I'll expand a little more.

Once, people had to make a very difficult decision. People had five alternatives and had to decide which was the best. Wise men from all over the world gathered and conferred.

The first to speak was a Christian. He pointed out that the first alternative was the best and should be chosen. He had no arguments, but simply stated that he believed so.

Then a Muslim spoke. He said that the second alternative was the best and should be chosen. He did not have any arguments either, but simply stated that he believed so.

People were not happy, it has not become clearer yet.

The humanist spoke. He said that the third alternative was the best and should be chosen. "It is the best because it will contribute the most to the well-being, progress and freedom of the people," he argued.

Then the existentialist spoke. He pointed out that there was no need to find a common solution, but that each individual could make his own choice of what he thought best. A Catholic can choose the first option, a Muslim the second, a humanist the third. Everyone must decide for himself what is best for him.

Then the nihilist spoke. He pointed out that although the alternatives are different, there is no way to evaluate which alternative is better. Therefore, it does not matter which one people choose. They are all equally good. Or equally bad. The nihilist suggested that people simply draw lots.

It still hasn't become clearer to the people, but patience was running out.

And then a simple man in the crowd spoke up:

• We still don't know which is the best alternative, right?
• Right, - murmured those around.
• But we may find out in the future, right?
• Right.
• Then the better alternative is the one that leaves the most freedom to change the decision in the future.
• Sounds reasonable, - murmured those around.

You may think - it breaks Hume's law. No it doesn't. Facts and values stay distinct. Hume's law does not state that values must have an author, this was a wrong interpretation by Nick Bostrom.

I'm not sure if I'm persuaded that we can say that "evolution has intentions" - isn't evolution just a convenient word to describe a pattern? Evolution isn't an entity, it isn't a system, it doesn't have an identity. It is a quality or pattern that we notice in certain systems and at this point we risk getting into some kind of Hegelian matroishka doll about weltgeist - something which I'm afraid to even bring up.

I also feel like I'm missing something that "genocide and factory farms are instrumentally useful" especially if you're anthropamorphising evolution.

That being said, if the use of the Pathetic Fallacy allows you to make your beliefs pay rent - then I retract the above!

At any rate, thank you for posting this - because I realized something!

I think what this has made me realize is that I DO NOT have a conscious totalizing worldview or philosophy. But obviously I must have a worldview, I assume everyone does, all humans beings have a myriad of beliefs that lies on a continuum between a eclectic and byzantine (and very inconsistent) hodge-podge and the kind of totalizing and self-consistent system you describe.

Now, when I say I don't have a "conscious totalizing worldview or philosophy" - I am saying I don't have the self-awareness to know how self-consistent my beliefs are, and where on the continuum mine lies and this is partly because I couldn't summarize it. As such I'm guessing I'm somewhere on the inconsistent, eclectic, hodge-podge side - but perhaps unconsciously my beliefs are actually really self-consistent and I'm more on the totalizing side, but I doubt it.

This realization is surprising to me, because one thing I am is an aesthete with a curatorial bent - or in laymans terms "I know what I like, and I know what I don't like and avoid it like the plague". An aesthete is someone who is especially sensitive to (or excessively concerned with) the beautiful, especially in art. And by curatorial, I mean, someone who wants those beautiful art things to be coordinated in a certain way, to exclude anything which isn't beautiful rah rah rah. In this regard I have a very bright but narrow spotlight of self-awareness. 

I don't mean to say that I am some kind of superior tastemaker or have a better sense of what is beautiful than others, but I do believe I am especially attuned to my own, idiosyncratic and totally subjective sense of beauty or what I enjoy experiencing.  In fact if you ask me "what do most people like?" I would throw up my hands. If pushed, I would mumble something about Taylor Swift, Jeff Koons, and Michael Kors - the kind of answer you give when you have no idea what you're talking about.

To put it another way, I couldn't possibly be an aesthetic elitist, because I don't even know what most people like, so I couldn't even have something to compare my own aesthetics against.

I don't know if that counts as a totalizing worldview, since it is only a partial worldview - it is a hyperacuity about art, fashion, music, narrative, the written word, performance etc. that I "like". Me, myself, and only I.


 

I agree with your view about organizational problems. Your discussion gave me an idea: Is it possible to shift employees dedicated to capability improvement to work on safety improvement? Set safety goals for these employees within the organization. This way, they will have a new direction and won't be idle, worried about being fired or resigning to go to other companies. Besides, it's necessary to make employees understand that improving safety is a highly meaningful job. This may not rely solely on the organization itself, but also require external pressure, such as from the government, peers, or the public. If the safety cannot be ensured, your product may face a lot of criticism and even be restricted from market access. And there will be some third-party organizations conducting safety evaluations of your product, so you need to do a solid job in safety rather than just going through the motions. 

Some people think the Bible or Quran was written by the infinitely intelligent creator of the universe. I think this is a failure of imagination.

What do I mean? 

Like really, you don't think such a being would write a *better* book? 

Imagine a book that you could read both forwards and backwards. As in, the letters in all the words just so happen to be arranged such that the book could be meaningfully read both ways with different messages. That alone would be insane. But then also the chapter titles formed an acrostic and the whole book rhymed. 

Oh and the book contains so much scientific and mathematical knowledge that it would make scientists and mathematicians irrelevant for hundreds of years. 

Oh and the book is so beautifully written that human beings 99% of the time cry and convert upon reading it. 

Imagine a book that not only gives fantastic advice on current issues, with all their nuances and sub-nuances, but gives detailed advice about situations that will not occur for thousands of years. 

Oh and it gives detailed advice about how to interpret it, so there are no feuds about the correct way to interpret it.

An infinitely intelligent God could definitely write such a book.

So why would he give us... the Bible or the Quran?

If your response to this is, "Oh, a limited mind could never comprehend what an infinite mind would create," then you are committed to believing that if, if the Bible/Quran consisted of hundreds of pages saying, "I, Lord Poopy Butt, commandeth you to rub poop on your poopy faces, throw poop at each other, roll around in poop, and make poop hats..." you couldn’t say, "I don’t think an infinite mind would create that..." Instead, you’d have to say, "A limited being like me could never know what an infinite mind might do."

For the bard in the party: 'Sharp' plans are those that are so unlikely to actually work, that if they do, there will be stories told about them for the ages. 'Blunt' plans are grinding in the mines. Nobody writes ballads about grinding in the mines. Obviously if you are an elvish freakin' bard, whose job is to improvise the next token, you want to follow sharp plans; or at least follow the tweets and warbles of the sharp plans' authors and mimic them like the liar-bird you are. But if you are a proper dwarf-lord who updates your weights with every blow of the mattock, you will take the blunt plan every time, boring away until it bores you into delving too deep.

(My guess is you meant to agree with that, but kind of the whole point of my comment was that the dimension that is more important than legibility and alignment with you is the buy-in your audience has for a given status system. Youtube is not very legible, and not that aligned, but for some audiences has very high buy-in.)