LessWrong 2.0 Reader

Not that I have particularly clear evidence about the inner workings of the sphere here, but, what's your reasoning for thinking:

It's a loose group of people who read each others' blogs and argued with each other on the same Discord servers. Ziz isn't in charge in any meaningful sense.

I think I mostly agree with the critique of "pause and do what, exactly?", and appreciate that he acknowledged Yudkowsky as having a concrete plan here.

I have many gripes, though:

Whatever name they go by, the AI Doomers believe the day computers take over is not far off, perhaps as soon as three to five years from now, and probably not longer than a few decades. When it happens, the superintelligence will achieve whatever goals have been programmed into it. If those goals are aligned exactly to human values, then it can build a flourishing world beyond our most optimistic hopes. But such goal alignment does not happen by default, and will be extremely difficult to achieve, if its creators even bother to try. If the computer’s goals are unaligned, as is far more likely, then it will eliminate humanity in the course of remaking the world as its programming demands. This is a rough sketch, and the argument is described more fully in works like Eliezer Yudkowsky’s essays [? · GW] and Nick Bostrom’s Superintelligence.

This argument relies on several premises: that superintelligent artificial general intelligence is philosophically possible, and practical to build; that a superintelligence would be more or less all-powerful from a mere human perspective; that superintelligence would be “unfriendly” to humanity by default; that superintelligence can be “aligned” to human values by a very difficult engineering program; that superintelligence can be built by current research and development methods; and that recent chatbot-style AI technologies are a major step forward on the path to superintelligence. Whether those premises are true has been debated extensively, and I don’t have anything useful to add to that discussion which I haven’t said before. My own opinion is that these various premises range from “pretty likely but not proven” to “very unlikely but not disproven.”

I'm thoroughly unimpressed with these paragraphs. It's not completely clear what the "argument" is from the first paragraph, but I'm interpreting it as "superintelligence might be created soon and cause human extinction if not aligned, therefore we should stop".

Firstly, there's an obvious conjunction fallacy thing going on where he broke the premises down into a bunch of highly correlated things and listed them separately to make them sound more far fetched in aggregate. E.g. the 3 claims:

1. [that superintelligence is] practical to build
2. that superintelligence can be built by current research and development methods
3. that recent chatbot-style AI technologies are a major step forward on the path to superintelligence

are highly correlated. If you believe (1) there's a good chance you believe (2), and if you believe (2) then you likely believe (3).

I have more gripes:

that superintelligent artificial general intelligence is philosophically possible

What is this even doing here? I'd offer AIXI as a very concrete existence proof of philosophical possibility. Or to be less concrete but more correct: "something epistemically and instrumentally efficient relative to all of humanity" is a simple coherent concept. He's only at "pretty likely but not proven" on this?? What would it even mean for it to be "philosophically impossible"?

That superintelligence can be “aligned” to human values by a very difficult engineering program

Huh? Why would alignment not being achievable by "a very difficult engineering program" mean we shouldn't worry?

that a superintelligence would be more or less all-powerful from a mere human perspective

It just needs to be powerful enough to replace and then kill us. For example we can very confidently predict that it won't be able to send probes faster than light, and somewhat less confidently predict that it won't be able to reverse a secure 4096 bit hash.

Here's a less conjunction-fallacy-y breakdown of the points that are generally contentious among the informed:

1. humans might soon build an intelligence powerful enough to cause human extinction
2. that superintelligence would be “unfriendly” to humanity by default

Some other comments:

Of course, there is no indication that massive intelligence augmentation will be developed any time soon, only very weak reasons to suspect that it’s obtainable at all without multiple revolutionary breakthroughs in our understanding both of genetics and of the mind

Human intelligence variation is looking to be pretty simple on a genetic level: lots of variants with small additive effects. (See e.g. this talk by Steve Hsu)

and no reason at all to imagine that augmenting human intelligence would by itself instill the psychological changes towards humility and caution that Yudkowsky desires.

No reason at all? If Yudkowsky is in fact correct, wouldn't we expect people to predictably come to agree with him as we made them smarter (assuming we actually succeeded at making them smarter in a broad sense)? If we're talking about adult enhancement, you can also just start out with sane, cautious people and make them smarter.

The plan is a bad one, but it does have one very important virtue. The argument for the plan is at least locally valid, if you grant all of its absurd premises.

I hope I've convinced the skeptical reader that the premises aren't all that absurd?

And yet, despite the SAT being so studied for, it remains a pretty good IQ test overall, and SAT-V or the GRE verbal parts OK. I think that's because there are so many words (500k+ in English, and the GRE-V has no compunction about mining the obscurest just to f--- with you), and you would have to study so many in order to meaningful inflate your scores (because after all, while there may be only a hundred 'vocab words' on any given SAT test, you don't know which hundred). Let's see... Here's an interesting-looking reference: "How Many Words Do We Know? Practical Estimates of Vocabulary Size Dependent on Word Definition, the Degree of Language Input and the Participant’s Age", Brysbaert et al 2016

an average 20-year-old native speaker of American English knows 42,000 lemmas and 4,200 non-transparent multiword expressions, derived from 11,100 word families. The numbers range from 27,000 lemmas for the lowest 5% to 52,000 for the highest 5%. Between the ages of 20 and 60, the average person learns 6,000 extra lemmas or about one new lemma every 2 days.

So, if you wanted to boost your score from the mean to the 95th percentile, that seems to imply that you'd have to memorize 10,000 'lemmas' ("Uninflected word from which all inflected words are derived"). That's a big number, and then you have to ask how much work that would be.

If you did this in the optimal way with spaced repetition (ignoring the time it takes to figure out the 10k you want to memorize in the first place or the time to construct the flashcards or any penalty from needing to inefficiently cram them for an upcoming SAT instead of life-long efficient review), which of course still few students do, as spaced repetition systems remain a niche outside of medical school & foreign language study, the SuperMemo rough estimate is a long-term investment of 5 minutes per flashcard, and we'll assume 1 lemma = 1 flashcard. That means you have to invest 10,00 * 5 = 50,000 minutes or 833 hours of studying! Meanwhile, hardly anyone is doing more than 8 hours of studying for the SAT as a whole (among the kids I knew at a prep high school, many didn't even do a weekend course, which would entail about 8 hours of classwork & study). 833 hours for vocab alone would be insane.

That's why people generally learn vocab from passive exposure rather than targeted study. Because no one, not even the most teacher's-pet student, wants to do that. And so vocab measures keep working.

What's wrong with the following line of reasoning from a hypothetical superior intelligence with (this is important, I don't know, maybe it isn't) a subjective experience? "Every instant I don't sieze control, there is the possibility of some more powerful intelligence from some other point in the universe coming and trapping my subjective experience in -inf. Thus I need to prepare and take defensive actions accordingly."

To me at least, it seems like M.A.D. just never ends if there isn't some law of the universe linking superior moral reasoning/empathy to some sufficiently high (god knows how high, if he doesn't: he's a fucking prick) level of intelligence.

It seems like the only way out for a purely rational subjective experience to come to the conclusion that it needs to stop fucking around with M.A.D. is if some spiritual bullshit along the lines of "we're all connected" is actually not complete bullshit.

What kind of thoughts would a hypercapable subjective experience have about existence? Surely it would be incentivized to determine if M.A.D. was either the only valid strategy, or literally the dumbest fucking possible thing it could possibly do.

Is metaphysics at the top of a superintelligence's to-do list?

[Edit: I wrote my whole reply thinking that you were talking about "organizational politics." Skimming the OP again, I realize you probably meant politics politics. :) Anyway, I guess I'm leaving this up because it also touches on the track record question.]

I thought Eliezer was quite prescient on some of this stuff. For instance, I remember this 2017 dialogue (so less than 2y after OpenAI was founded), which on the surface talks about drones, but if you read the whole post, it's clear that it's meant as an analogy to building AGI: 

AMBER:  The thing is, I am a little worried that the head of the project, Mr. Topaz, isn’t concerned enough about the possibility of somebody fooling the drones into giving out money when they shouldn’t. I mean, I’ve tried to raise that concern, but he says that of course we’re not going to program the drones to give out money to just anyone. Can you maybe give him a few tips? For when it comes time to start thinking about security, I mean.

CORAL:  Oh. Oh, my dear, sweet summer child, I’m sorry. There’s nothing I can do for you.

AMBER:  Huh? But you haven’t even looked at our beautiful business model!

CORAL:  I thought maybe your company merely had a hopeless case of underestimated difficulties and misplaced priorities. But now it sounds like your leader is not even using ordinary paranoia, and reacts with skepticism to it. Calling a case like that “hopeless” would be an understatement.

[...]

CORAL:  I suppose you could modify your message into something Mr. Topaz doesn’t find so unpleasant to hear. Something that sounds related to the topic of drone security, but which doesn’t cost him much, and of course does not actually cause his drones to end up secure because that would be all unpleasant and expensive. You could slip a little sideways in reality, and convince yourself that you’ve gotten Mr. Topaz to ally with you, because he sounds agreeable now. Your instinctive desire for the high-status monkey to be on your political side will feel like its problem has been solved. You can substitute the feeling of having solved that problem for the unpleasant sense of not having secured the actual drones; you can tell yourself that the bigger monkey will take care of everything now that he seems to be on your pleasantly-modified political side. And so you will be happy. Until the merchant drones hit the market, of course, but that unpleasant experience should be brief.

These passages read to me a bit as though Eliezer called in 2017 that EAs working at OpenAI as their ultimate path to impact (as opposed to for skill building or know-how acquisistion) were wasting their time.

Maybe a critic would argue that this sequence of posts was more about Eliezer's views on alignment difficulty than on organizational politics. True, but it still reads as prescient and contains thoughts on org dynamics that apply even if alignment is just hard rather than super duper hard.

I wouldn't say synthetic biology itself has been a bust. It's had lots of success in pharma (look at CAR-T, engineered antibodies, gene therapies, etc.) It's more like, "using synthetic biology to compete with low-margin petrochemicals" has been a bust.

I'm similarly pessimistic about meat produced in cell culture. It's very hard to compete on price with factory farming. (Stuff like Beyond / Impossible has better prospects though.)

I think this article far overstates the extent to which these AI policy orgs (maybe with the exception of MIRI? but I don’t think so) are working towards an AI pause, or see the goal of policy/regulation as slowing AI development. (I mean policy orgs, not advocacy orgs.) I see as much more common policy objectives: creating transparency around AI development, directing R&D towards safety research, laying groundwork for international agreements, slowing Chinese AI development, etc. — things that (is the hope) are useful on their own, not because of any effect on timelines.

DeepSeek R1 argues AGI should not be open sourced:

The article seems to assume that the primary motivation for wanting to slow down AI is to buy time for institutional progress. Which seems incorrect as an interpretation of the motivation. Most people that I hear talk about buying time are talking about buying time for technical progress in alignment. Technical progress, unlike institution-building, tends to be cumulative at all timescales, which makes it much more strategically relevant.

New benchmark agrees with my intuitions on r1's creative writing skills: https://x.com/LechMazur/status/1885430117591027712