LessWrong 2.0 Reader

"as important as ever": no, because our potential influence is lower, and the influence isn't on things shaped like our values, there has to be a translation, and the translation is different from the original.

CEV: while it addresses "extrapolation" it seems broadly based on assuming the extrapolation is ontologically easy, and "our CEV" is an unproblematic object we can talk about (even though we need logical uncertainty over it, and logical induction has additional free parameters in the limit). I'm really trying to respond to orthogonality not CEV though.

from a practical perspective: notice that I am not behaving like Eliezer Yudkowsky. I am not saying the Orthogonality Thesis is true and important to ASI, I am instead saying intelligence/values are Oblique and probably nearly Diagonal (though it's unclear what I mean by "nearly"). I am not saying a project of aligning superintelligence with human values is a priority. I am not taking research approaches that assume a Diagonal/Orthogonal factorization. I left MIRI because I didn't like their security policies, I thought discussion of abstract research ideas was more important. I am not calling for a global AI shutdown so this project (which is in my view confused) can be completed. I am actually against AI regulation on the margin (I don't have a full argument for this, it's a political matter at this point).

I think practicality looks more like having near-term preferences related to modest intelligence increases (as with current humans vs humans with neural nets; how do neural nets benefit or harm you, practically?), and not expecting your preferences to extend into the distant future with many ontology changes, so don't worry about grabbing hold of the whole future etc, think about how to reduce value drift while accepting intelligence increases on the margin. This is a bit like CEV except CEV is in a thought experiment instead of reality.

The "Models of ASI should start with realism" bit IS about practicalities, namely, I think focusing on first forecasting absent a strategy of what to do about the future is practical with respect to any possible influence on the far future; practically, I think your attempted jump to practicality (which might be related to philosophical pragmatism) is impractical in this context.

It occurs to me that maybe you mean something like "Our current (non-extrapolated) values are our real values, and maybe it's impossible to build or become a superintelligence that shares our real values so we'll have to choose between alignment and superintelligence." Is this close to your position?

Close. Alignment of already-existing human values with superintelligence is impossible (I think) because of the arguments given. That doesn't mean humans have no preferences indirectly relating to superintelligence (especially, we have preferences about modest intelligence increases, and there's some iterative process).

ChatGPT4 generates social psychology hypotheses that are rated as original as those proposed by human experts https://x.com/BogdanIonutCir2/status/1836720153444139154 

Noted, that does seem a lot more tractable than using natural latents to pin down details of CEV by itself

Yeah I had vaguely remembered this story but not the details.

X

extreme nit, you probably meant for this be lowercase. I love this series!

This is amazing to me, frankly. Thank you for your comment. I mean, good bless you, but I can’t view slave morality as I’ve described it as anything other than pathetic. I also value kindness, but, for instance, I admire people like Elon Musk infinitely more than the kind, soft person who’s life will ultimately be close to meaningless, and I’d still admire him if he was much crueller and sadistic than he actually is.

We are at the brink of changing the course of the universe forever, even thinking about the downtrodden too much feels outright immoral to me at this point.

As per our discussions on our other posts, I don't think we can say that value learning in itself solves the problem. The issue of whether the ASI's interpretation of its central goal or instructions changing is not automatically solved by adopting that approach. The value mutability problem you link to is a separate issue. I'm not addressing here whether human values might change, but whether an AGI's interpretations of its central goal/values might change.

My claim is that the natural latents the AI needs to share for this setup are not about the details of what a 'CEV' is. They are about what researchers mean when they talk about initializing, e.g., a physics simulation with the state of the Earth at a specific moment in time.

I am not asking about ‘true’ general intelligence? Or whatever that implies.

If your not sure, I am asking regarding the term commonly called ‘general intelligence’, or sometimes also known as ‘general mental ability factor’ or ‘g-factor’, in mainstream academic papers. Such as those found in pedagogy, memetics, genetics, etc…

See: https://scholar.google.com/scholar?hl=en&as_sdt=0%252C5&q=“general+intelligence”&btnG=

Where many many thousands of researchers over the last few decades are referring to this.

Here is a direct quote by a pretty well known expert among intelligence researchers, writing in 2004:

“ During the past few decades, the word intelligence has been attached to an increasing number of different forms of competence and accomplishment-emo-tional intelligence, football intelligence, and so on. Researchers in the field, however, have largely abandoned the term, together with their old debates over what sorts of abilities should and should not be classified as part of intelligence. Helped by the advent of new technologies for researching the brain, they have increasingly turned their attention to a century-old concept of a single overarching mental power. They call it simply g, which is short for the general mental ability factor. The g factor is a universal and reliably measured distinction among humans in their ability to learn, reason, and solve problems. It corresponds to what most people mean when they describe some individuals as smarter than others, and it's well measured by IQ (intelligence quotient) tests, which assess high-level mental skills such as the ability to draw inferences, see similarities and differences, and process complex information of virtually any kind. Understanding g's biological basis in the brain is the new frontier in intelligence research today. The g factor was discovered by the first mental testers, who found that people who scored well on one type of mental test tended to score well on all of them. Regardless of their contents (words, numbers, pictures, shapes), how they are administered (individually or in groups; orally, in writing, or pantomimed), or what they're intended to measure (vocabulary, mathematical reasoning, spatial ability), all mental tests measure mostly the same thing. This common factor, g, can be distilled from scores on any broad set of cognitive tests, and it takes the same form among individuals of every age, race, sex, and nation yet studied. In other words, the g factor exists independently of schooling, paper-and-pencil tests, and culture.”

I think my terminology isn't totally clear. By "goals" in that statement, I mean what we mean by "'values" in humans. The two are used in overlapping and mostly interchangable ways in my writing

1. Humans aren't sufficiently intelligent to be all that internally consistent
2. In many cases of humans changing goals, I'd say they're actually changing subgoals, while their central goal (be happy/satisfied/joyous) remains the same. This may be described as changing goals while keeping the same values.
3. Note 'in the short term' (I think you're quoting Bostrom? The context isn't quite clear). In the long term, with increasing intelligence and self-awareness, I'd expect some of people's goals to change as they become more self-aware and work toward more internal coherence (e.g., many people change their goal of eating delicious food when they realize it conflicts with their more important goal of being happy and living a a long life).

Yes, humans may change exactly that way. A friend said he'd gotten divorced after getting a CPAP to solve his sleep apnea: "When we got married, we were both sad and angry people. Now I'm not." But that's because we're pretty random and biology determined.