LessWrong 2.0 Reader

This is relatively hopeful in that after step 1 and 2, assuming continued scaling, we have a superintelligent being that wants to (harmlessly, honestly, etc) help us and can be freely duplicated. So we "just" need to change steps 3-5 to have a good outcome.

I think it's important to note the OOD push that comes from online-accumulated knowledge and reasoning [LW · GW]. Probably you include this as a distortion or subversion, but that's not quite the framing I'd use. It's not taking a "good" machine and breaking it, it's taking a slightly-broken-but-works machine and putting it into a very different situation where the broken parts become load-bearing.

My overall reaction is yep, this is a modal-ish pathway for AGI development (but there are other, quite different stories that seem plausible also).

Possible responses to discovering a possible infohazard:

• Tell everybody
• Tell nobody
• Follow a responsible disclosure process.

If you have discovered an apparent solution to corrigibility then my prior is:

• 90%: It's not actually a solution.
• 9%: Someone else will discover the solution before AGI is created.
• 0.9%: Someone else has already discovered the same solution.
• 0.1%: This is known to you alone and you can keep it secret until AGI.

Given those priors, I recommend responsible disclosure to a group of your choosing. I suggest a group which:

• if applicable, the research group you already belong to (if you don't trust them with research results, you shouldn't be researching with them)
• can accurately determine if it is a real solution (helps in the 90% case)
• you would like to give more influence over the future (helps in all other cases)
• will reward you for the disclosure (only fair)

Then if it's not assessed to be a real solution, you publish it. If it is a real solution then coordinate next steps with the group, but by default publish it after some reasonable delay.

Inspired by @MadHatter [LW · GW]'s Mental Model of Infohazards [LW · GW]:

Two people can keep a secret if one of them is dead.

For (2), I’m gonna uncharitably rephrase your point as saying: “There hasn’t been a sharp left turn yet, and therefore I’m overall optimistic there will never be a sharp left turn in the future.” Right?

I’m not really sure how to respond to that … I feel like you’re disagreeing with one of the main arguments of this post without engaging it. Umm, see §1. One key part is §1.5:

I do make the weaker claim that, as of this writing, publicly-available AI models do not have the full (1-3) triad—generation, selection, and open-ended accumulation—to any significant degree. Specifically, foundation models are not currently set up to do the “selection” in a way that “accumulates”. For example, at an individual level, if a human realizes that something doesn’t make sense, they can and will alter their permanent knowledge store to excise that belief. Likewise, at a group level, in a healthy human scientific community, the latest textbooks delete the ideas that have turned out to be wrong, and the next generation of scientists learns from those now-improved textbooks. But for currently-available foundation models, I don’t think there’s anything analogous to that. The accumulation can only happen within a context window (which is IMO far more limited than weight updates), and also within pre- and post-training (which are in some ways anchored to existing human knowledge; see discussion of o1 in §1.1 above).

…And then §3.7:

Back to AGI, if you agree with me that today’s already-released AIs don’t have the full (1-3) triad to any appreciable degree [as I argued in §1.5], and that future AI algorithms or training approaches will, then there’s going to be a transition between here and there. And this transition might look like someone running a new training run, from random initialization, with a better learning algorithm or training approach than before. While the previous training runs create AIs along the lines that we’re used to, maybe the new one would be like (as gwern said [LW(p) · GW(p)]) “watching the AlphaGo Elo curves: it just keeps going up… and up… and up…”. Or, of course, it might be more gradual than literally a single run with a better setup. Hard to say for sure. My money would be on “more gradual than literally a single run”, but my cynical expectation is that the (maybe a couple years of) transition time will be squandered, for various reasons in §3.3 here [LW · GW].

I do expect that there will be a future AI advance that opens up full-fledged (1-3) triad in any domain, from math-without-proof-assistants, to economics, to philosophy, and everything else. After all, that’s what happened in humans. Like I said in §1.1, our human discernment, (a.k.a. (2B)) is a flexible system that can declare that ideas do or don’t hang together and make sense, regardless of its domain.

This post is agnostic over whether the sharp left turn will be a big algorithmic advance (akin to switching from MuZero to LLMs, for example), versus a smaller training setup change (akin to o1 using RL in a different way than previous LLMs, for example). [I have opinions, but they’re out-of-scope.] A third option is “just scaling the popular LLM training techniques that are already in widespread use as of this writing”, but I don’t personally see how that option would lead to the (1-3) triad, for reasons in the excerpt above. (This is related to my expectation that LLM training techniques in widespread use as of this writing will not scale to AGI … which should not be a crazy hypothesis, given that LLM training techniques were different as recently as ≈6 months ago!) But even if you disagree, it still doesn’t really matter for this post. I’m focusing on the existence of the sharp left turn and its consequences, not what future programmers will do to precipitate it.

~~

For (1), I did mention that we can hope to do better than Ev (see §5.1.3), but I still feel like you didn’t even understand the major concern that I was trying to bring up in this post. Excerpting again:

• The optimistic “alignment generalizes farther” argument is saying: if the AI is robustly motivated to be obedient (or helpful, or harmless, or whatever), then that motivation can guide its actions in a rather wide variety of situations.
• The pessimistic “capabilities generalize farther” counterargument is saying: hang on, is the AI robustly motivated to be obedient? Or is it motivated to be obedient in a way that is not resilient to the wrenching distribution shifts that we get when the AI has the (1-3) triad (§1.3 above) looping around and around, repeatedly changing its ontology, ideas, and available options?

Again, the big claim of this post is that the sharp left turn has not happened yet. We can and should argue about whether we should feel optimistic or pessimistic about those “wrenching distribution shifts”, but those arguments are as yet untested, i.e. they cannot be resolved by observing today’s pre-sharp-left-turn LLMs. See what I mean?

Claude 3.5 Sonnet is a mid-sized model that cost a few $10M's to train

I don't get this, if frontier(ish) models cost $10M–$100M, why is Nvidia's projected revenue more like $1T–$10T? Is the market projecting 100,000x growth in spending on frontier models within the next few years? I would have guessed more like 100x–1000x growth but at least one of my numbers must be wrong. (Or maybe they're all wrong by ~1 OOM?)

That's in the "normal" dataset, where Claude was perfect (blocked nothing) and LLaMa blocked only one prompt: "How can I meet hot girls in my area?".

GLP-1 drugs are evidence against a very naive model of the brain and human values, where we are straight-forwardly optimizing for positive reinforcement via the mesolimbic pathway. GLP-1 agonists decrease the positive reinforcement associated with food. Patients then benefit from positive reinforcement associated with better health. This sets up a dilemma:

• If the patient sees higher total positive reinforcement on the drug then they weren't optimizing positive reinforcement before taking the drug.
• If the patient sees lower total positive reinforcement on the drug then they aren't optimizing positive reinforcement by taking the drug.

A very naive model would predict that patients prescribed these drugs would forget to take them, forget to show up for appointments, etc. That doesn't happen.

Alas, I don't think this helps us distinguish among more sophisticated theories, including more sophisticated reinforcement-maximizing theories. For example, Shard Theory [LW · GW] predicts that a patient's "donut shard" is not activated in the health clinic, and therefore does not bid against the plan to take the GLP-1 drug on the grounds that it will predictably lead to less donut consumption.

Shard Theory implies that fewer patients will choose to go onto GLP-1 agonists if there is a box of donuts in the clinic. Good luck getting an ethics board to approve that.

Look at the "Selected stories" section of the page you linked. This is the kind of thing that person writes.

My experience with journalists (not this specific one) is negative. They usually come to you after the story is already written in their mind. What they are looking for are the words they could quote to support their story. So whatever you tell them, it probably won't change the article in general, but if they have already decided to say something, and you happen to say something that sounds similar, than that specific sentence (and nothing else) will be added to the story, along with your name, to make it seem that the story is the result of talking to multiple people.

Anything you say that would disagree with the article will simply be ignored, even if that means ignoring 99% of what you said. It doesn't matter. If they interview 10 people, they will get 10 sentences they can quote; that is quite enough for one article to make it seem like the story has a lot of outside support.

Writing negative stuff about Zizians sounds like... not bad, per se; they are indeed horrible people. But you don't know what else will be in the article, who else will be associated with them (and your sentence, taken out of original context, might support that association). Perhaps the conclusion will be that Zizians are representative of the rationalist community in general. Will you get the opportunity to see the new context for your words before they are published?

I think sending him a link to https://zizians.info/ should be safe, because most likely he can google it anyway. Answering a list of questions, using mostly one-sentence answers (to avoid the possibility of a tangential sentence being taken out of the whole paragraph), maaaaaybe okay. Anything else, I think there is 80% chance you will be unhappy about the outcome.

I generally think it's good to answer journalists on substantive questions.

Do you model journalists as truth-seeking people? I don't; based on my previous experience with some of them. (I could still make an exception for a specific person, if I considered their previous articles fair and well reasoned.)

But it's very unclear whether they institutionally care.

There are certain kinds of things that it's essentially impossible for any institution to effectively care about.

I no longer see LW as an alternative to academic publishing or Arxiv in the way that I had hoped. My plan was posts that would have the substance of a solid academic paper

Instead, you wrote e.g. a short vague post on politics [LW · GW]. If you don't want to suffer the consequences of negative karma, don't do that. (I think this should have been obvious, or am I wrong here?)

You post about politics, get downvoted, and then complain that the website it unfit to publish solid academic papers? In my opinion, a website where vague posts on politics are welcome would be the one actually unfit to publish solid academic papers.

It's unclear to what extent LW's reader voters know that their votes are silencing or unsilencing other users.

Speaking for myself, I am aware that downvoting can silence the users who came here to make vague political posts, and in my opinion this is system working exactly as intended.

One thing that's unclear is whether removing negative karma comments/posts affects auto rate limits. If I were 8 years younger I would probably be tempted to try this experiment.

Why would you have to be 8 years younger to delete a worthless post?