LessWrong 2.0 Reader

As per our discussions on our other posts, I don't think we can say that value learning in itself solves the problem. The issue of whether the ASI's interpretation of its central goal or instructions changing is not automatically solved by adopting that approach. The value mutability problem you link to is a separate issue. I'm not addressing here whether human values might change, but whether an AGI's interpretations of its central goal/values might change.

My claim is that the natural latents the AI needs to share for this setup are not about the details of what a 'CEV' is. They are about what researchers mean when they talk about initializing, e.g., a physics simulation with the state of the Earth at a specific moment in time.

I am not asking about ‘true’ general intelligence? Or whatever that implies.

If your not sure, I am asking regarding the term commonly called ‘general intelligence’, or sometimes also known as ‘general mental ability factor’ or ‘g-factor’, in mainstream academic papers. Such as those found in pedagogy, memetics, genetics, etc…

See: https://scholar.google.com/scholar?hl=en&as_sdt=0%252C5&q=“general+intelligence”&btnG=

Where many many thousands of researchers over the last few decades are referring to this.

Here is a direct quote by a pretty well known expert among intelligence researchers, writing in 2004:

“ During the past few decades, the word intelligence has been attached to an increasing number of different forms of competence and accomplishment-emo-tional intelligence, football intelligence, and so on. Researchers in the field, however, have largely abandoned the term, together with their old debates over what sorts of abilities should and should not be classified as part of intelligence. Helped by the advent of new technologies for researching the brain, they have increasingly turned their attention to a century-old concept of a single overarching mental power. They call it simply g, which is short for the general mental ability factor. The g factor is a universal and reliably measured distinction among humans in their ability to learn, reason, and solve problems. It corresponds to what most people mean when they describe some individuals as smarter than others, and it's well measured by IQ (intelligence quotient) tests, which assess high-level mental skills such as the ability to draw inferences, see similarities and differences, and process complex information of virtually any kind. Understanding g's biological basis in the brain is the new frontier in intelligence research today. The g factor was discovered by the first mental testers, who found that people who scored well on one type of mental test tended to score well on all of them. Regardless of their contents (words, numbers, pictures, shapes), how they are administered (individually or in groups; orally, in writing, or pantomimed), or what they're intended to measure (vocabulary, mathematical reasoning, spatial ability), all mental tests measure mostly the same thing. This common factor, g, can be distilled from scores on any broad set of cognitive tests, and it takes the same form among individuals of every age, race, sex, and nation yet studied. In other words, the g factor exists independently of schooling, paper-and-pencil tests, and culture.”

I think my terminology isn't totally clear. By "goals" in that statement, I mean what we mean by "'values" in humans. The two are used in overlapping and mostly interchangable ways in my writing

1. Humans aren't sufficiently intelligent to be all that internally consistent
2. In many cases of humans changing goals, I'd say they're actually changing subgoals, while their central goal (be happy/satisfied/joyous) remains the same. This may be described as changing goals while keeping the same values.
3. Note 'in the short term' (I think you're quoting Bostrom? The context isn't quite clear). In the long term, with increasing intelligence and self-awareness, I'd expect some of people's goals to change as they become more self-aware and work toward more internal coherence (e.g., many people change their goal of eating delicious food when they realize it conflicts with their more important goal of being happy and living a a long life).

Yes, humans may change exactly that way. A friend said he'd gotten divorced after getting a CPAP to solve his sleep apnea: "When we got married, we were both sad and angry people. Now I'm not." But that's because we're pretty random and biology determined.

As long as all mature superintelligences in our universe don't necessarily have (end up with) the same values, and only some such values can be identified with our values or what our values should be, AI alignment seems as important as ever. You mention "complications" from obliqueness, but haven't people like Eliezer recognized similar complications pretty early, with ideas such as CEV?

It seems to me that from a practical perspective, as far as what we should do, your view is much closer to Eliezer's view than to Land's view (which implies that alignment doesn't matter and we should just push to increase capabilities/intelligence). Do you agree/disagree with this?

It occurs to me that maybe you mean something like "Our current (non-extrapolated) values are our real values, and maybe it's impossible to build or become a superintelligence that shares our real values so we'll have to choose between alignment and superintelligence." Is this close to your position?

No one that I know based on first hand information, otherwise I probably would have included them as an example.

Most likely due the fact that people who succeeded at such have little reason to advertise it beyond a small circle.

So then what is the issue you want to discuss?

Neither of us can do more than offer our guesses and opinions on these two points.

That is, confounding could go both ways here; the effect could be greater than it appears, rather than less.

Absolutely, but if we assume the null hypothesis until proven otherwise, we will prefer to think of confounding as creating effect that is not there, rather than subduing an even stronger effect.

I'll reanalyse that way and post results, if I remember.

Yes, please do! I suspect (60 % confident maybe?) the effect will still be at least a standard error, but it would be nice to know.

I made a script run in the background on my PC, something lik

Ah, bummer! I also have this problem solved for computer time, and I was hoping you had done something for smartphone carriage.

(Note, by the way, that a uniformly random delay is not as surprising as an exponentially distributed delay. Probably does not matter for your usecase, and you might already know all of that...)

I've posted it there. Had to use a linkpost because I didn't have an existing account there and you can't crosspost without 100 karma (presumably to prevent spam) and you can't funge LW karma for EAF karma.

But there’s no denying that expanding the They franchise will necessarily increase ambiguity by slurring two well-worn axes of distinction (he/she & singular/plural). By no means would this be the end of the world, but it will require some compensating efforts in other areas to maintain clarity, perhaps by relying more on proper nouns and less on pronouns.

I believe the psychological perception of others by gender, and the 'defaultness' of the notion of gender in humans, cause(d) more bad than good (at least when discluding the evolutionary era). This motivated me to switch to using the non-gendering pronoun 'they' for almost^[1] everyone.

I haven't found my use of 'they' by default to require nontrivial compensation to maintain clarity. Any ambiguity introduced in a draft is removed by one of the simple checks I try to run across all of my writing for others: if referent of a word (namely 'that', 'this', 'it', or 'they') is unclear : replace with direct referent word or rephrase to remove unclarity.

Also, I think this helps match the reader's interpretation to my intended meaning. Among humans, a being's 'gender' has a lot of connotative meaning. I think not introducing those connotations [LW · GW] is instrumental to eliminating unintended ways my text could be interpreted, which in my experience is the real difficulty with writing.

1. ^{^}

   excepting beings who this would harm

   and excepting some contexts where I expect some readers might be confused by singular they