LessWrong 2.0 Reader

This letter was an important milestone in the evolution of MIRI's strategy over 2020-2024. As of October 2023 [LW · GW] Yudkowsky is MIRI's chair and "the de facto reality (is) that his views get a large weight in MIRI strategic direction".

MIRI used to favor technical alignment over policy work. In April 2021, in comments to Death with Dignity [LW · GW] Yudkowsky argued [LW(p) · GW(p)] that:

How about if you solve a ban on gain-of-function research first, and then move on to much harder problems like AGI? A victory on this relatively easy case would result in a lot of valuable gained experience, or, alternatively, allow foolish optimists to have their dangerous optimism broken over shorter time horizons.

People were not completely swayed by this advice, and the Best of LessWrong 2022 included What an Actually Pessimistic Containment Strategy Looks Like [LW · GW] in April 2022 and Let's Think About Slowing Down AI [LW · GW] in December 2022.

In this Time letter from March 2023 we see Yudkowsky doing AI policy work. In January 2024 the new MIRI CEO announced in the MIRI 2024 Mission and Strategy Update [LW · GW] that AI policy work provides "a glimmer of hope". In December 2024 we get Communications in Hard Mode - My New Job at MIRI [LW · GW].

A question for all of us: "How could I have thought that faster? [LW · GW]"

Could you give some examples of ontology pyramid schemes? 

Also, can ontologies that are actually true, be the subject of pyramid schemes, as you define them?

Did you collect the data for their actual median timelines, or just its position relative to 2030? If you collected higher-resolution data, are you able to share it somewhere?

I don't habitually use the concept so I don't have an opinion on how to use the term.

(In general, any human who might be worth enslaving is also a person whom it would be improper to enslave.)

...I don’t see what that has to do with LLMs, though.

This claim by you about the conditions under which slavery is profitable seems wildly optimistic, and not at all realistic, but also a very normal sort of intellectual move.

If a person is a depraved monster (as many humans actually are) then there are lots of ways to make money from a child slave.

I looked up a list of countries where child labor occurs. Pakistan jumped out as “not Africa or Burma” and when I look it up in more detail, I see that Pakistan’s brick industry, rug industry, and coal industry all make use of both “child labor” and “forced labor”. Maybe not every child in those industries is a slave, and not every slave in those industries is a child, but there’s probably some overlap.

It seems like you have quite substantially misunderstood my quoted claim. I think this is probably a case of simple “read too quickly” on your part, and if you reread what I wrote there, you’ll readily see the mistake you made. But, just in case, I will explain again; I hope that you will not take offense, if this is an unnecessary amount of clarification.

The children who are working in coal mines, brick factories, etc., are (according to the report you linked) 10 years old and older. This is as I would expect, and it exactly matches what I said: any human who might be worth enslaving (i.e., a human old enough to be capable of any kind of remotely useful work, which—it would seem—begins at or around 10 years of age) is also a person whom it would be improper to enslave (i.e., a human old enough to have developed sapience, which certainly takes place long before 10 years of age). In other words, “old enough to be worth enslaving” happens no earlier (and realistically, years later) than “old enough such that it would be wrong to enslave them [because they are already sapient]”.

(It remains unclear to me what this has to do with LLMs.)

Since “we” (you know, the good humans in a good society with good institutions) can’t even clean up child slavery in Pakistan, maybe it isn’t surprising that “we” also can’t clean up AI slavery in Silicon Valley, either.

Maybe so, but it would also not be surprising that we “can’t” clean up “AI slavery” in Silicon Valley even setting aside the “child slavery in Pakistan” issue, for the simple reason that most people do not believe that there is any such thing as “AI slavery in Silicon Valley” that needs to be “cleaned up”.

In asking the questions I was trying to figure out if you meant “obviously AI aren’t moral patients because they aren’t sapient” or “obviously the great mass of normal humans would kill other humans for sport if such practices were normalized on TV for a few years since so few of them have a conscience” or something in between.

Like the generalized badness of all humans could be obvious-to-you (and hence why so many of them would be in favor of genocide, slavery, war, etc and you are NOT surprised) or it might be obvious-to-you that they are right about whatever it is that they’re thinking when they don’t object to things that are probably evil, and lots of stuff in between.

None of the above.

You are treating it as obvious that there are AIs being “enslaved” (which, naturally, is bad, ought to be stopped, etc.). Most people would disagree with you. Most people, if asked whether something should be done about the enslaved AIs, will respond with some version of “don’t be silly, AIs aren’t people, they can’t be ‘enslaved’”. This fact fully suffices to explain why they do not see it as imperative to do anything about this problem—they simply do not see any problem. This is not because they are unaware of the problem, nor is it because they are callous. It is because they do not agree with your assessment of the facts.

That is what is obvious to me.

(I once again emphasize that my opinions about whether AIs are people, whether AIs are sapient, whether AIs are being enslaved, whether enslaving AIs is wrong, etc., have nothing whatever to do with the point I am making.)

I've been thinking about this a lot lately. It seems to link to many things. And might be a bit too much for just a comment. But here are some key concepts from mostly psych that I think link to why sleeping on a problem makes it easier.

• Hebb's Law
• Learning is assumed to take place over a 24hr span
• Chunking
• The Multi-component Model of Working Memory
• Mice developing 'Maze Neurons' when learning a maze
• People who are woken mid-sleep and self report dreaming about a problem they've tried to solve, do better the next day than people who are woken and don't report dreaming about the problem

If I boil it down, I have two hypotheses that could both be true.

1. When you dream about a problem you're brain is formulating ideas that can help you solve it. All you have to do the next day is try again and those ideas will become available to you as if you had just 'had an idea'
2. Sleeping on a problem breaks it up into more manageable chunks that you can better manipulate in working memory the next time you try to solve it. 

There are other things that happen during sleep that will just make every problem easier to solve the next day. For example: 

• Cleaning up chemical 'garbage' that collects in your brain during the day.
• Forgetting things that the brain doesn't think you have a use for
• Resetting/reducing your emotions. (If you're stressed about a new problem, you'll find it easier to solve it when you're less stressed.)

Do you think that cyber professionals would take multiple hours to do the tasks with 20-40 min first-solve times? I'm intuitively skeptical.

Yes, that would be my guess, medium confidence.

One component of my skepticism is that someone told me that the participants in these competitions are less capable than actual cyber professionals, because the actual professionals have better things to do than enter competitions. I have no idea how big that selection effect is, but it at least provides some countervailing force against the selection effect you're describing.

I'm skeptical of your skepticism. Not knowing basically anything about the CTF scene but using the competitive programming scene as an example, I think the median competitor is much more capable than the median software engineering professional, not less. People like competing at things they're good at.

Do you think that cyber professionals would take multiple hours to do the tasks with 20-40 min first-solve times? I'm intuitively skeptical.

One (edit: minor) component of my skepticism is that someone told me that the participants in these competitions are less capable than actual cyber professionals, because the actual professionals have better things to do than enter competitions. I have no idea how big that selection effect is, but it at least provides some countervailing force against the selection effect you're describing.

I believe Cybench first solve times are based on the fastest top professional teams, rather than typical individual CTF competitors or cyber employees, for which the time to complete would probably be much higher (especially for the latter).

Report on an experiment in playing builder-breaker games [AF · GW] with language models to brainstorm and critique research ideas 

---

Today I had the thought: "What lessons does human upbringing have for AI alignment?"

Human upbringing is one of the best alignment systems that currently exist. Using this system we can take a bunch of separate entities (children) and ensure that, when they enter the world, they can become productive members of society. They respect ethical, societal, and legal norms and coexist peacefully. So what are we 'doing right' and how does this apply to AI alignment? 

---

To help brainstorm some ideas here I first got Claude to write a short story about an AI raised as a human child. (If you want to, you can read it first). 

There are a lot of problems with this story of course, such as overly anthropomorphising the AI in question. And overall the story just seems fairly naive / not really grounded in what we know about current frontier models.  

Some themes emerged which seemed interesting and plausible to me were that: 

1. Children have trusted mentors who help them contextualize their experiences through the lens of human values. They are helped to understand that sometimes bad things happen, but also to understand the reasons these things happen. I.e. experience ('data') is couched in a value-oriented framework.
2. This guidance can look like socratic questioning, i.e. being guided to think through the implications of your ideas / statements. 

I think these are worth pondering in the context of AI alignment. 

One critical flaw in the story is that it seems to assume that AIs will be altruistic. Without this central assumption the story kind of falls apart. That might suggest that "how to make AIs be altruistic towards humans" is an important question worth thinking about. 

---

Meta-report. What did I learn from doing this exercise? The object level findings and ideas are pretty speculative and it's not clear how good they are. But I think AI is an underrated source of creativity / inspiration for novel ideas. (There's obviously a danger of being misled, which is why it's important to remain skeptical, but this is already true in real life anyway.) 

It's worth noting that this ended up being kind of like a builder-breaker game [AF · GW], but with AI taking the role of the builder. I think stuff like this is worth doing more of, if only to get better at identifying sketchy parts of otherwise plausible arguments. See also; the importance of developing good taste [LW(p) · GW(p)]