LessWrong 2.0 Reader

I really like the observation in your Further Thoughts point. I do think that is a problem people need to look at as I would guess many will view the government involvement from a acting in public interests view rather than acting in either self interest (as problematic as that migh be when the players keep changing) or from a special interest/public choice perspective.

Probably some great historical analysis already written about events in the past that might serve as indicators of the pros and cons here. Any historians in the group here?

Mathematics students are often annoyed that they have to worry about "bizarre or unnatural" counterexamples when proving things. For instance, differentiable functions without continuous derivative  are pretty weird. Particularly engineers tend to protest that these things will never occur in practice, because they don't show up physically. But these adversarial examples show up constantly in the practice of mathematics - when I am trying to prove (or calculate) something difficult, I will try to cram the situation into a shape that fits one of the theorems in my toolbox, and if those tools don't naturally apply I'll construct all kinds of bizarre situations along the way while changing perspective. In other words, bizarre adversarial examples are common in intermediate calculations - that's why you can't just safely forget about them when proving theorems. Your logic has to be totally sound as a matter of abstraction or interface design - otherwise someone will misuse it. 

Is there a term for a (hypothetical) phenomenon where AI systems might come to mirror the values and behavior of their creators, analogous to enculturation in humans? Claude suggests 'Ethos Imprinting' but I'm not sure if there's something standard out there.

Have you read Byrnes [? · GW] on how we should expect certain emotional reactions to seemingly prohibitively complex stimuli, to be basically hardcoded?

I think I have learned a certain amount of my eye-contact-aversion, but I also suspect it is hardcodedly unusually difficult for me. When I was younger, my parents and teachers constantly harangued me to "get better at eye contact", and I really tried [I hated their haranguing!] but the eye contact itself was just too emotionally painful.

When I first went on Ritalin in November of 2020, I immediately noticed that it became much more possible to voluntarily choose to sustain eye contact with most people for ~3-10-second periods; I was excited and thought maybe I could make the whole aversion go away. But that didn't work either. It stayed effortful, although easier.

I still don't make much eye contact, and at this point in my life nobody ever remarks on it, and it causes no problem at all. I regularly meet people who are worse at it than I am, and some of them seem to have less of a general social handicap than I do!

If I am reading this entry correctly, I may beg to differ with it. It would be a sign of higher intelligence if someone IS able to be a generalist and collate and interpolate data and meaning across a wide variety of topics, but who also is able to wave the white flag of humility and admit when they are in over their head. If by contrast they are unable to do so, and just tend to forge ahead on the SS Dunning-Kruger to the Land of the Grand Fallacies of Ignorance, then yes I think I am justified to look down on their overall intellectual abilities. That they may be competent in their one narrow area of expertise may simply indicate that they have successfully absorbed the surface tenets therein (as put forth by people wiser and more knowledgeable) but without necessarily pondering them more deeply. That by itself doesn't impress me (much).

[Newspapers are a somewhat inapt entity to focus such analysis on since we are by definition dealing with many individual minds and not just one, but even there implicit editorial and institutional biases may be at work. I'd rather focus on individuals tho.]

My fave example from my own sphere is the baseball analyst Bill James. I occasionally dropped into his own personal web site, and had always looked up to him as an impartial analyst who would bend over backward to be objective.

Imagine my total shock to see him casually dismiss some political positions of people he came across on another site as "social justice warriors", a right-wing dog whistle term of course, which indicated that he at least has a lot of sympathy with MAGA views.  I was thus instantly confused, since I'd figure a mind such as his would approach politics with the same exact objectivity and rigor that he did baseball, and thus grasping the subtleties of how out society and political systems operate that he would be the LAST pundit to use such loaded and biased language. I was wrong, obviously.

Note I had however by that point already noted some blind spots in his baseball analysis, and in fact had picked up on a subtle but telling little throwaway note in his The New Bill James Historical Baseball Abstract  [from 2001], about my favorite pitcher of all time in point of fact, Greg Maddux. He had earlier outlined situations where a player misses playing time but still deserves credit, such as wartime seasons. However I noted that he DIDN'T include time missed due to player strikes, something rather glaring by omission. 

Maddux's two best seasons of course were the strike-shortened years 1994 and 1995; give him credit for the missed starts there and his peak in those seasons becomes one of the best ever.

He dismissed those concerns with a casual throwaway line, calling it "crying over spilled milk." I was rather puzzled by such a lapse in his objectivity for years afterward, until I saw the SJW note, and realized that, being an extreme right-winger, he must have been HIGHLY biased against labor rights and thus the rights of workers to strike for better conditions, and allowed this bias to affect his baseball analysis. When I put 2 and 2 together I was yes rather appalled. His books lie in a box somewhere (after a couple of moves in the last 6 years) and I haven't felt much of an urge to consult them since then, since noting these blind spots made me wonder where else his objectivity has lapsed. [In any event his work, while certainly groundbreaking, has been subsequently surpassed by others in the field.]

IOW if someone has a lapse or blind spot outside their field of expertise, it likely will feed back into their core profession or area of putative expertise as well, and you will likely spot such lapses if you dig deep enough, and in any event I think I am more than justified to judge anything they say (in or out) at least a bit more harshly, because such a lapse isn't likely to just be a one-time thing but indicative of deeper deficiencies that will likely cut across a wide swath of knowledge in the case of the individual in question.

The first few that came to mind have, it turns out, already retired since I last talked to them.

The next few are basically all bloggers with a tighter focus that I learned about either here on LW or through recommendations that ultimately chain back to SSC/ASX. 

There are a lot of good sources of data in the world, and very few good sources of analysis, and those that exist have very little relationship to popularity or price or prestige.

Beyond that, it really is "buyer beware," and learning to know your own limited and improve your own speed at sorting the nonsense and spotting the bad assumptions and wrong inferences. That's why I'm not being more specific - without knowing your own habits of thought, it's hard to guess whose habitual mistakes and quirks will be transparent to you, and which will mislead you or just not paint the intended picture for you [? · GW].

Probably my advice is, if something you read seems worth understanding, try to spot and discount (what Zvi calls) the Obvious Nonsense. Back in 2008 my aunt sent me a NYT article that (seemingly) claimed having granite countertops was like smoking a pack of cigarettes a day. Obvious nonsense. It was basically pretending the cancer risk from cigarettes was radiological and not chemical, but unless you knew enough about biology and physics it was easy to miss. I was primed for that, because I was a physics undergrad and because I'd recently been reading about how coal plants release more radiation than nuclear plants even if you ignore all other chemical pollution. Eventually this kind of thinking became habitual for me, and it became easier to learn useful things from inadequate sources. 

It's also been educational to go through a few healthcare situations where you need to find doctors that are Actually Good and not just cargo-cult-style going through the motions. There's a vibe, a style, that just shines through regardless of specific topic, related to curiosity and excitement about something new and different.

(This is a repost of my comment [LW(p) · GW(p)] on John's "My AI Model Delta Compared To Yudkowsky [LW · GW]" post which I wrote a few months ago. I think points 2-6 (especially 5 and 6) describe important and neglected difficulties of AI alignment.)

My model (which is pretty similar to my model of Eliezer's model) does not match your model of Eliezer's model. Here's my model, and I'd guess that Eliezer's model mostly agrees with it:

1. Natural abstractions (very) likely exist in some sense. Concepts like "chair" and "temperature" and "carbon" and "covalent bond" all seem natural in some sense, and an AI might model them too (though perhaps at significantly superhuman levels of intelligence it rather uses different concepts/models). (Also it's not quite as clear whether such natural abstractions actually apply very well to giant transformers (though still probable in some sense IMO, but it's perhaps hard to identify them and to interpret what "concepts" actually are in AIs).)
2. Many things we value are not natural abstractions, but only natural relative to a human mind design. Emotions like "awe" or "laughter" are quite complex things evolved by evolution, and perhaps minds that have emotions at all are just a small space in minddesignspace. The AI doesn't have built-in machinery for modelling other humans the way humans model other humans. It might eventually form abstractions for the emotions, but probably not in a way it understands "how the emotion feels from the inside".
   1. There is lots of hidden complexity in what determines human values. Trying to point an AI to human values directly (in a similar way to how humans are pointed to their values) would be incredibly complex. Specifying a CEV process / modelling one or multiple humans and identifying in the model where the values are represented and pointing the AI to optimize those values is more tractable, but would still require a vastly greater mastering of understanding of minds to pull of, and we are not on a path to get there without human-augmentation.
3. When the AI is smarter than us it will have better models which we don't understand, and the concepts it uses will diverge from the concepts we use. As an analogy, consider 19th-century humans (or people who don't know much about medicine) being able to vaguely classify health symptoms into diseases, vs the AI having a gears-level model of the body and the immune system which explains the observed symptoms.
4. I think a large part of what Eliezer meant with Lethalities#33 is that the way thinking works deep in your mind looks very different from the English sentences which you can notice going through your mind and which are only shallow shadows of what actual thinking is going on in your mind; and for giant transformers the way the actual thinking looks there is likely even a lot less understandable from the way the actual thinking looks in humans.
5. Ontology idenfication (including utility rebinding) is not nearly all of the difficulty of the alignment problem (except possibly in so far as figuring out all the (almost-)ideal frames to model and construct AI cognition is a requisite to solving ontology identification). Other difficulties include:
   1. We won't get a retargetable general purpose search by default, but rather the AI is (by default) going to be a mess of lots of patched-together optimization patterns [LW · GW].
   2. There are lots of things that might cause goal drift [LW · GW]; misaligned mesa-optimizers which try to steer or get control of the AI; Goodhart; the AI might just not be smart enough initially and make mistakes which cause irrevocable value-drift; and in general it's hard to train the AI to become smarter / train better optimization algorithms, while keeping the goal constant.
   3. (Corrigibility.)
6. While it's nice that John is attacking ontology identification, he doesn't seem nearly as much on track to solve it in time as he seems to think. Specifying a goal in the AI's ontology requires finding the right frames for modelling how an AI imagines possible worldstates, which will likely look very different from how we initially naively think of it (e.g. the worldstates won't be modelled by english-language sentences or anything remotely as interpretable). The way we currently think of what "concepts" are might not naturally bind to anything in how the AI's reasoning looks like, and we first need to find the right way to model AI cognition and then try to interpret what the AI is imagining. Even if "concept" is a natural abstraction on AI cognition, and we'd be able to identify them (though it's not that easy to concretely imagine how that might look like for giant transformers), we'd still need to figure out how to combine concepts into worldstates so we can then specify a utility function over those.

"Cyborgism or AI-assisted research that gets up 5x speedups but applies differentially to technical alignment research"

How do you do you make meaningful progress and ensure it does not speed up capabilities?

It seems unlikely that a technique exists that is exclusively useful for alignment research and can't be tweaked to help OpenMind develop better optimization algorithms etc.

(This is an abridged version of my comment here [LW(p) · GW(p)], which I think belongs on my shortform. I removed some examples which were overly long. See the original comment for those.)

Here are some lessons I learned over the last months from doing alignment research on trying to find the right ontology for modelling (my) cognition:

• make examples: if you have an abstract goal or abstract hypothesis/belief/model/plan, clarify on an example what it predicts.
  • e.g. given thought "i might want to see why some thoughts are generated" -> what does that mean more concretely? -> more concrete subcases:
    • could mean noticing a common cognitive strategy [LW · GW]
    • could mean noticing some suggestive concept similarity
    • maybe other stuff like causal inference (-> notice i'm not that clear on what i mean by that -> clarify and try come up with example):
      • e.g. "i imagine hiking a longer path" -> "i imagine missing the call i have in the evening"
  • (yes it's often annoying and not easy, especially in the beginning)
  • (if you can't you're still confused.)
• generally be very concrete. also Taboo your words [? · GW] and Replace the Symbol with the Substance [LW · GW].
• I want to highlight the "what is my goal" part
  • also ask "why do i want to achieve the goal?"
    • (-> minimize goodhart)
  • clarify your goal as much as possible.
    • (again Taboo your words...)
    • clarify your goal on examples
      • when your goal is to understand something, how will you be able to apply the understanding on a particular example?
• try to extract the core subproblems/subgoals.
  • e.g. for corrigibility a core subproblem is the shutdown problem (where further more precise subproblems could be extracted.)
  • i guess make sure you think concretely and list subproblems and summarize the core ones and iterate. follow up on confusions where problems still seem sorta mixed up. let your mind find the natural clusters. (not sure if that will be sufficient for you.)
• tie yourself closely to observations.
• drop all assumptions. apply generalized hold off on proposing solutions.
  • in particular, try not to make implicit non-well-founded assumptions about how the ontology looks like, like asking questions like "how can i formalize concepts" or "what are thoughts". just see the observations as directly as possible and try to form a model of the underlying process that generates those.
• first form a model about concrete narrow cases and only later generalize
  • e.g. first study precisely what thoughtchains you had on particular combinatorics problems before hypothesizing what kind of general strategies your mind uses.
  • special case: (first) plan how to solve specific research subproblems rather than trying to come up with good general methodology for the kinds of problems you are attacking.
• don't overplan and rather try stuff and review how it's going and replan and iterate.
  • this is sorta an application of "get concrete" where you get concrete through actually trying the thing rather than imagining how it will look like if you attack it.
• often review how you made progress and see how to improve.
• (also generally lots of other lessons from the sequences (and HPMoR): notice confusion, noticing mysterious answers, know how an actual reduction looks like, and probably a whole bunch more)

Tbc those are sorta advanced techniques. Most alignment researchers are working on line of hopes that pretty obviously won't work while thinking it has a decent chance of working, and I wouldn't expect those techniques to be much use for them.
There is this quite foundational skill of "notice when you're not making progress / when your proposals aren't actually good" which is required for further improvement, and I do not know how to teach this. It's related to be very concrete and noticing mysterious answers or when you're too abstract or still confused. It might sorta be what Eliezer calls security mindset.

(Also other small caveat: I did not yet get very clear great results out of my research, but I do think I am making faster progress (and I'm setting myself a very high standard). I'd guess the lessons can probably be misunderstood and misapplied, but idk.)

Yeah, I'm not surprised that English-positive is a lot weaker than English (in terms of short statements), I'm just surprised that the weakness turns up so quickly with such natural-seeming sentences rather than pathological or complex statements. You see an example like "the sky is not green", and it's so easy to turn into "the sky is blue" or "grass is inedible" to "grass provides 0 calories", and you start to think maybe most or all normal natural statements have feasible rewrites - and then you hit "We didn't see John."