LessWrong 2.0 Reader

Eli wrote [LW(p) · GW(p)]:

There are at least types of people that the term "Zizian" might refer to:

1. Someone who has read Sinceriously.fyi and is generally sympathetic to Ziz's philosophy.
2. A member of a relatively tightly-coordinated anarchist conspiracy, that has (allegedly) planned and carried out a series of violent crimes.

Octavia is a Zizian in the first sense, but is not (to my knowledge) a Zizian in the second sense. In fact, she seems unaware or disbelieving that a network of Zizians of the second sense exists. She appears to think that there are only 'people who have benefited from reading Ziz's blog', and no coordinated criminal network to speak of.

I would be very surprised if there was no "inner Ziz crew", as inner circles around leaders / prominent figures in a community seem like a default thing that forms in movements/cultural groups.

But is it true that you don't think this inner circle is a coordinated group responsible for the murders?

My quick impression is that this is a brutal and highly significant limitation of this kind of research. It's just incredibly expensive for others to read and evaluate, so it's very common for it to get ignored.

I'd predict that if you improved the arguments by 50%, it would lead to little extra uptake.

I think this is wrong. The introduction of the GD paper takes no more than 10 minutes to read and no significant cognitive effort to grasp, really. I don't think there is more than 10% potential of making it any clearer or approachable.

She's not but to the extent that people put the AI labs in one bucket with LW/EA (TESCREAL or sth), the Annie Altman incident may cause us additional reputational damage.

If you want, it would help me learn to write better, for you to list off all the words (or sentences) that confused you.

I would love to render any assistance I can in that regard, but my fear is this is probably more of a me-problem than a general problem with your writing.

What I really need though is a all encompassing, rigid definition of a 'terminal goal' - what is and isn't a terminal goal. Because "it's a goal which is instrumental to no other goal" just makes it feel like the definition ends wherever you want it to. Because, consider a system which is capable of self-modification and changing it's own goals, now the difference between an instrumental goal and a terminal goal erodes. 

Never the less some of your formatting was confusing to me, for example a few replies back you wrote:

As for the case of idealized terminal-goal-pursuers, any two terminal goals can be combined into one, e.g. {paperclip-amount×2 + stamp-amount} or {if can create a black hole with p>20%, do so, else maximize stamps}, etc.

The bit " {paperclip-amount×2 + stamp-amount}" and " {if can create a black hole with p>20%, do so, else maximize stamps}" was and is very hard for me to understand. If it was presented in plain English, I'm confident I'd understand it. But using computer-code-esque variables, especially when they are not assigned values introduces a point of failure for my understanding. Because now I need to understand your formatting, and the pseudo-code correctly (and as not a coder, I struggle to read pseudo-code at the best of times) just to understand the allusion you're making.

Also the phrase "idealized terminal-goal-pursuers" underspecifies what you mean by 'idealized'? I can think of at least four possible senses you might be gesturing to:

A. a terminal-goal-pursuer who's terminal goals are "simple" enough to lend themselves as good candidates for a thought experiment - therefore ideal from the point of view of a teacher and a student.

B. ideal as in extremely instrumentally effective in accomplishing their goals,

C. ideal as in they encapsulate the perfect undiluted 'ideal' of a terminal goal (and therefore it is possible to have pseudo-terminial goals) - i.e. a 'platonic ideal/essence' as opposed to a platonic appearance, 

D. "idealized" as in that these are purely theoretical beings (at this point in time) - because while humans may have terminal goals, they are not particularly good or pure examples of terminal-goal-havers? The same for any extant system we may ascribe goals to?

E. "idealized" in a combination of A and B which is very specific to entities that have multiple terminal goals, which is unlikely, but for the sake of argument if they did have two or more terminal goals would display certain behaviors.

I'm not sure which you mean. But suspect it's none-of-the-above.

For the record, I know you absolutely don't mean "ideal" as in "moral ideal". Nor in an Aesthetic or Freudian sense, like when a teenager "idealizes" their favourite pop-star and raves on about how perfect they are in every way

But going back to my confusion over terminal goals, and what is or isn't:

For example: "I value paperclips. I also value stamps, but one stamp is only half as valuable as a paperclip to me" → "I have the single value of maximizing this function over the world: {paperclip-amount×2 + stamp-amount}". (It's fine to think of it in either way)

I'm not sure what this statement is saying, because that describes a possibly very human attribute - that we may have two terminal goals in that they are not subservient or means of pursuing anything else. Which is what I understand a 'terminal' goal to mean. The examples in the video describe very "single-minded" entities that have a single terminal goal that they seek to optimize, like a stamp collecting machine.

There's a few assumptions I'm making here: that a terminal goal is "fixed" or permanent. You see when I said sufficiently superintelligent entities would converge on certain values, I was assuming that they would have some kind of self-modification abilities. And therefore their terminal values would look a lot like common convergent instrumental values of other, similarly self-adapting/improving/modifying entities. 

However if this is not a terminal goal, then what is a terminal goal? And for a system that is capable of adapting and improving itself, what would be it's terminal goals?

Is terminal goal simply a term of convenience?

I wonder if this is 

1. funding - the company need money to perform research on safety alignment (X risks, and assuming they do want to to this), and to get there they need to publish models so that they can 1) make profits from them, 2) attract more funding. A quick look on the funding source shows Amazon, Google, some other ventures, and some other tech companies
2. empirical approach - they want to take empirical approach to AI safety and would need some limited capable models

But both of the points above are my own speculations

I think for this specific example the superior is wrong because realistically we can form an expectation of the distribution of those factors. Just because we don't know doesn't mean it's actually necessarily a gaussian - some factors, like the Coriolis force, are systematic. If the distribution was "a ring of 1 m around the aimed point" then you would know for sure you won't hit the terrorist that way, but have no clue whether you'll hit the kid.

Also, even if the distribution was gaussian, if it's broad enough the difference in probability between hitting the terrorist and hitting the kid may simply be too small to matter.

Exactly! I've also noticed there are so many ideas and theories out there, relative to the available resources to evaluate them and find the best to work on.

A lot of good ideas which I feel deserve a ton of further investigation, seem to be barely talked about after they're introduced. E.g. off the top of my head,

• Why Don't We Just... Shoggoth+Face+Paraphraser? [LW · GW]
• Applying superintelligence without collusion [LW · GW]
• Self-Other Overlap [LW · GW]
• Multi-objective homeostasis [LW · GW]
• MONA: Managed Myopia with Approval Feedback [LW · GW]

My opinion is that there isn't enough funds and manpower. I have an idea [LW · GW] on increasing that, which ironically also got ignored, yay!

Interesting, thanks!

I guess one could object that in you're even more clueless sniper example, applying the POI between Hit and Not Hit is just as arbitrary as applying it between, e.g., Hit, Hit on his right, and Hit on his left. This is what Greaves (2016) -- and maybe others? -- called the "problem of multiple partitions". In my original scenario, people might argue that there isn't such a problem and that there is only one sensible way to apply POI. So it'd be ok to apply it in my case and not in yours. 

I don't know what to make of this objection, though. I'm not sure it makes sense. It feels a bit arbitrary to say "we can apply POI but only when there is one way of applying it that clearly seems more sensible". Maybe this problem of multiple partitions is a reason to reject POI altogether (in situations of what Greaves call "complex cluelessness" at least, like in my sniper example).

No one with the money has offered to fund it yet. I'm not even sure they're aware this is happening.

Um, this seems bad. I feel like I should do something, but I don't personally have that kind of money to throw around. @habryka [LW · GW], is this the LTFF's job??

Can you elaborate further on how Gato is proof that just supplementing the training data is sufficient? I looked on youtube and can't find any videos of task switching.