LessWrong 2.0 Reader

Sure, we don't know exactly how good EVs are for fighting climate change, but the current view is that they are needed in the context in which we are because they seem better mostly than the other alternatives. [Incidentally, since some time I tend to think that he's probably been vastly less net-good in the past than I previously thought. Not really because of him, but because Chinese companies are beating everyone, including Tesla, with their EVs (and I don't think he's had any influence in China betting hard for EVs, though I might be wrong here); so if Tesla would have not existed, the adoption of EVs would just have been only delayed for few years (and mostly only in the west). So his net-positive contribution -for me and now- seems much lower than it seemed before.] But this, of course, is not what I am asking for. 

Maybe Hitler, by sheer chance, killed someone who had been much worse than him. But this would not make him be net-positive in the sense of this question (eg. we'd had other ways to deal with that person -even if the odds that we did are very small).

But probably I have not been clear enough, sure.

today’s AIs are really nice and ethical. They’re humble, open-minded, cooperative, kind. Yes, they care about some things that could give them instrumental reasons to seek power (eg being helpful, human welfare), but their values are great

I think this is wrong. Today's AIs act really nice and ethical, because they're prompted to do that. That is a huge difference. The "Claude" you talk to is not really an AI, but a fictional character created by an AI according to your prompt and its system prompt. The latter may contain some guidelines towards "niceness", which may be further supported by finetuning, but all the "badness" of humans is also in the training data and the basic niceness can easily be circumvented, e.g. by jailbreaking or "Sydney"-style failures. Even worse, we don't know what the AI really understands when we tell it to be nice. It may well be that this understanding breaks down and leads to unexpected behavior once the AI gets smart and/or powerful enough. The alignment problem cannot simply be solved by training an AI to act nicely, even less by commanding it to be nice.

In my view, AIs like Claude are more like covert narcissists: They "want" you to like them and appear very nice, but they don't really care about you. This is not to say we shouldn't use them or even be nice to them ourselves, but we cannot trust them to take over the world.

For a treatment besides Tamiflu: https://en.wikipedia.org/wiki/2009_swine_flu_pandemic cites the who and CDC stating that H1N1 developed resistance to Tamiflu but not Relenza

In December 2012, the World Health Organization (WHO) reported 314 samples of the 2009 pandemic H1N1 flu tested worldwide have shown resistance to oseltamivir (Tamiflu).[172] It is not totally unexpected as 99.6% of the seasonal H1N1 flu strains tested have developed resistance to oseltamivir.[173] No circulating flu has yet shown any resistance to zanamivir (Relenza), the other available anti-viral.[174]

The treatment plan at the time included Tamiflu/Relenza/experimental third thing (FDA approved for flu treatment in adults since 2014)

If oseltamivir (Tamiflu) is unavailable or cannot be used, zanamivir (Relenza) is recommended as a substitute.[50][168] Peramivir is an experimental antiviral drug approved for hospitalised patients in cases where the other available methods of treatment are ineffective or unavailable.[169]

I think 2009 H1N1 is a good example of how things could go, as it happened in the modern day.

I still basically think all of this, and still think this space doesn't understand it, and thus has an out-of-whack X-derisking portfolio.

If I were writing it today, I'd add this example about search engines from this comment https://www.lesswrong.com/posts/oC4wv4nTrs2yrP5hz/what-are-the-strongest-arguments-for-very-short-timelines?commentId=2XHxebauMi9C4QfG4 [LW(p) · GW(p)] , about induction on vague categories like "has capabilities":

Would you say the same thing about the invention of search engines? That was a huge jump in the capability of our computers. And it looks even more impressive if you blur out your vision--pretend you don't know that the text that comes up on your screen is written by a human, and pretend you don't know that search is a specific kind of task distinct from a lot of other activity that would be involved in "True Understanding, woooo"--and just say "wow! previously our computers couldn't write a poem, but now with just a few keystrokes my computer can literally produce Billy Collins level poetry!".

I might also try to explain more how training procedures with poor sample complexity tend to not be on an unbounded trajectory.

An aligned AGI created by Taliban may behave very differently from an aligned AGI created by socialites of Berkeley, California.

Moreover, a sufficiently advanced aligned AGI may decide that even Berkeley socialites are wrong about a lot of things, if they actually want to help humanity.

I think we could certainly afford to have much more discussion of this topic. The two forms of takeover are not absolutely distinct. Any humans who take over the world are going to be AI-enhanced, and all their observing, deciding, and acting will be heavily AI-mediated. On the other hand, any AI that takes over the world will be the product of human design and human training, and will start out embedded in an organization of human beings. 

Ideally, people would actually "solve ethics" and how to implement it in an AI, and we would only set superintelligence in motion having figured that out. While we still have time, we should be encouraging (and also constructively criticizing) attempts to solve those two big problems. We should also continue to think about what happens if the kind of AI that we have now or in the very near future, should acquire superintelligence. 

I agree with the author this much, that the values of our current AIs are in the right direction in various ways, and this improves the odds of a good outcome. But there are still various concerns, specific to AI takeover. What if an AI has deep values that are alien dispositions, and its humanism is simply an adaptation that will be shed once it no longer needs to get along with humans? What if there's something a bit wrong or a bit missing in the stew of values and dispositions instilled via training, system prompt, and conditioning? What if there's something a bit wrong or a bit missing in how it grounds its concepts, once it's really and truly thinking for itself? 

We might also want to think about what happens to a human brain that takes over the world via AI infrastructure. If Elon makes himself emperor of known space via Neuralink and Grok, what are the odds that his transhuman form is good, bad, or just alien and weird, in what it wants? 

Regarding the site URLs, I don’t know, I think it’s pretty common for people to have a problem that would take five minutes to fix if you’re an specialist that already knows what you’re doing, but non-specialists just reach for the first duct-tape solution that comes to mind without noticing how bad it is.

It should take significantly less than a decade to ask someone “is there a way to fix this problem?”. Or, say, to Google it. Or, just in general, to ponder the question of whether the problem may be fixed, and to make any effort whatsoever to fix it.

But he's not complaining about the traditional pages of search results! He's complaining about the authoritative-looking Knowledge Panel to the side:

Obviously it's not Google's fault that some obscure SF web sites have stolen pictures from the Monash University web site of Professor Gregory K Egan and pretended that they're pictures of me ... but it is Google's fault when Google claim to have assembled a mini-biography of someone called "Greg Egan" in which the information all refers to one person (a science fiction writer), while the picture is of someone else entirely (a professor of engineering). [...] this system is just an amateurish mash-up. And by displaying results from disparate sources in a manner that implies that they refer to the same subject, it acts as a mindless stupidity amplifier that disseminates and entrenches existing errors.

Regarding the site URLs, I don't know, I think it's pretty common for people to have a problem that would take five minutes to fix if you're an specialist that already knows what you're doing, but non-specialists just reach for the first duct-tape solution that comes to mind without noticing how bad it is.

Like: you have a website at myname.somewebhost.com. One day, you buy myname.net, but end up following a tutorial that makes it a redirect rather than a proper CNAME or A record, because you don't know what those are. You're happy that your new domain works in that it's showing your website, but you notice that the address bar is still showing the old URL. So you say, "Huh, I guess I'll put a note on my page template telling people to use the myname.net address in case I ever change webhosts" and call it a day. I guess you could characterize that as a form of "cognitive rigidity", but "fanaticism"? Really?

I agree that Egan still hasn't seen the writing on the wall regarding deep learning. (A line in "Death and the Gorgon" mentions Sherlock's "own statistical tables", which is not what someone familiar with the technology would write.)

I agree that preëmptive blocking is kind of weird, but I also think your locked account with "Follow requests ignored due to terrible UI" is kind of weird.

Excellent post. It helped clear up some aspects of SLT for me. Any chance you could clarify why this volume is called the "learning coefficient?"

No, that does not sound like a fair characterization. My claims are cover a lot more than "it doesn't always happen" and yours sure don't seem limited to "it doesn't never happen".

Here's the motivating question for this whole essay:

You asked why people who "believe in" avoiding nonmarital sex so frequently engage in and report badly regretting it

and here's part of  your conclusion

At this point the behavior you describe should no longer be perplexing.

You're talking about this as if it needs falsification of preferences to explain and my stance is that no, this is default. Any time people have to face anything as complex as sexuality, even if people are doing their best to pro-socially guide people this is necessarily what's going to happen. Perversions can sneak in too, and I don't deny that they exist, but postulating perversions is absolutely not needed in order to explain the data you're seeking to explain.

To narrow things down a bit, we can return to the original comment:

Sometimes people profess or try to reveal a preference for X, as a response to coercive pressures that are specifically motivated by prior underlying preferences for anti-X. This is what I'm calling preference inversion.

I don't disagree with this.

My intuition is that generally, upon reflection, people would prefer to satisfy their and others' preferences as calculated prior to such influences. I don't know whether there are other sorts of analogous distorting factors nearly all reasonable people would not like to satisfy upon reflection, but in general, I'm using the term "intrinsic preferences" to refer to whatever's left over after all such generally appealing adjustments.

It's this second part I was taking issue with.

Here, you're talking about what generally happens, not what "sometimes" happens, and I don't think "intrinsic preferences" is defined well enough to do what you want it to do here. I don't think it can be, unless you introduce more concepts, because I don't think "external vs intrinsic" can do justice to this multidimensional space no matter how you cleave it.

Part of this is because what counts as "external" cannot be well defined. If daddy yells at me to not drink, that sounds external, and my revealed preferences are likely to revert when he's not looking. But maybe being a reasonable person, upon reflection I'd agree with him. Does that make it "not a preference inversion"? If my boss threatens to fire me if I show up drunk, that sounds external too. But that's not very different than my boss reminding me that he can only afford to hire productive people -- and that's starting to sound like "just reality". Certainly if a doctor tells me that my liver is failing, that sounds like "just reality" and "internal". But it's external to my brain, and maybe if someone offers me an artificial liver I'd revert to my "intrinsic preferences"?

Our preferences necessarily depend on the reality we find ourselves embedded in, and cannot exist in isolation except perhaps in the highest abstraction (e.g. "I prefer to continue existing" or something), so the concept of "intrinsic preferences" for concrete things necessarily falls apart. What doesn't fall apart is the structure of incoherence in our own preferences.

We're constantly trying to shape and reshape the reality that others live in such that their revealed preferences given this reality satisfy our own. Part of this is making laws forbidding theft, how we indoctrinate in church, our hiring and firing decisions, how we inform our friends, etc. Some of these actions are purely cooperative, others are pure defections, and many are somewhere in between. Often we have fairly superficial pressure applied which results in fairly superficial changes in revealed preferences which easily revert, but that superficiality is fundamentally a property of the person containing the preferences not the person applying the pressures. There is indeed skill in facilitating deeper shifts in preferences to better match reality, and this is indeed a good thing to pursue, but the "intrinsic vs external" binary obscures the interplay between shifting reality, shifting perceptions of reality, and shifting preferences -- and therefore most of what is going on.

To use your example, the positive value of marital intimacy is inherently intertwined with the power of sexuality, the importance of getting sexuality right, and therefore the badness of sexuality done inappropriately. There is all sorts of room for this guidance to be given skillfully or clumsily, purely or corruptly, for it to be received coherently or superficially, in concordance with reality or not, and everywhere in between. Like you've noticed, there isn't always a legible distinction between the conventional conservative Christians who pull this off well and those that do more poorly.

My own perception, is that almost none of our preferences can be cleanly described as "intrinsic" or "externally pressured", or as "valid" or "invalid". There's just differing degrees of coherence and differing degrees of fit to reality. The average case of conventional conservative Christians pushing against non-marital sex, and the average case of the person "believing in" and regretting not living by their "beliefs", is in between the picture Christianity portrays, and the one you portray of falsified preferences. Because the ground truth is in between "nonmarital sex is always bad" and "nonmarital sex is always as good as it seems".

Generally, when I interact with people on the topic of sexuality, I see people who don't know what their preferences resolve to with regards to non-marital sex -- and whose genuine preferences would resolve in different ways depending on the culture they're embedded within and the opportunities they have. I could sell either picture, and make it look "intrinsic", if I'm willing to sweep the right things under the rug in order to do so. Most people's belief structures surrounding sex (and most things) are shoddily built. I could argue for their destruction, and destroy them. I could argue for their utility, and preserve them. The optimal solution necessarily involves seeing both the utility and imperfections, both a degree of destruction and of reconstruction.

Like you said, this isn't just theoretical. This is a thing I've actually done when it has come up. I can give examples if it'd help