LessWrong 2.0 Reader

I think the concern is less "am I making intellectual progress on some project" and more "is the project real / valuable" 

For (2), I’m gonna uncharitably rephrase your point as saying: “There hasn’t been a sharp left turn yet, and therefore I’m overall optimistic there will never be a sharp left turn in the future.” Right?

Hm, I wouldn't have phrased it that way. Point (2) says nothing about the probability of there being a "left turn", just the speed at which it would happen. When I hear "sharp left turn", I picture something getting out of control overnight, so it's useful to contextualize how much compute you have to put in to get performance out, since this suggests that (inasmuch as it's driven by compute) capabilities ought to grow gradually.

I feel like you’re disagreeing with one of the main arguments of this post without engaging it.

I didn't mean to disagree with anything in your post, just to add a couple points which I didn't think were addressed.

You're right that point (2) wasn't engaging with the (1-3) triad, because it wasn't mean to. It's only about the rate of growth of capabilities (which is important because if each subsequent model is only 10% more capable than the one which came before then there's good reason to think that alignment techniques which work well on current models will also work on subsequent models).

Again, the big claim of this post is that the sharp left turn has not happened yet. We can and should argue about whether we should feel optimistic or pessimistic about those “wrenching distribution shifts”, but those arguments are as yet untested, i.e. they cannot be resolved by observing today’s pre-sharp-left-turn LLMs. See what I mean?

I do see, and I think this gets at the difference in our (world) models. In a world where there's a real discontinuity, you're right, you can't say much about a post-sharp-turn LLM. In a world where there's continuous progress, like I mentioned above, I'd be surprised if a "left turn" suddenly appeared without any warning.

I learned how to sing and make music because I decided to sign up for Men's Choir in HS, even after (mistakenly) believing that I was tone-deaf. It was one of the best decisions I've made.

I confess that the ending is lost on me.

Batteries also help the efficiency of hybrid peaker plants by reducing idling and smoothing out ramp-up and ramp-down logistics.

Thanks. It still seems to me like the problem recurs. The application of Occam's razor to questions like "will the Sun rise tomorrow?" seems more solid than e.g. random intuitions I have about how to weigh up various considerations. But the latter do still seem like a very weak version of the former. (E.g. both do rely on my intuitions; and in both cases, the domain have something in common with cases where my intuitions have worked well before, and something not-in-common.) And so it's unclear to me what non-arbitrary standards I can use to decide whether I should let both, neither, or just the latter be "outweighed by a principle of suspending judgment".

Such an entity as described could absolutely be an existential threat to humanity

I agree. I think [LW · GW] you don't even need most of the stuff on the "superhuman" list, the equivalent of a competent IQ-130 human upload probably does it, as long as it has the speed + self-copying advantages.

Depends on your assumptions. If you assume that a pretty-well-intent-aligned pretty-well-value-aligned AI (e.g. Claude) scales to a sufficiently powerful tool to gain sufficient leverage on the near-term future to allow you to pause/slow global progress towards ASI (which would kill us all)...

We can drop the assumption that ASI inevitably kills us all/we should pause and still have the above argument work, or as I like to say it, practical AI alignment/safety is very much helped by computer security, especially against state adversaries.

I think Zach-Stein Perlman is overstating the case, but here it is:

https://www.lesswrong.com/posts/eq2aJt8ZqMaGhBu3r/zach-stein-perlman-s-shortform#ckNQKZf8RxeuZRrGH [LW(p) · GW(p)]

Well... One problem here is that a model could be superhuman at:

• thinking speed
• math
• programming
• flight simulators
• self-replication
• cyberattacks
• strategy games
• acquiring and regurgitating relevant information from science articles

And be merely high-human-level at:

• persuasion
• deception
• real world strategic planning
• manipulating robotic actuators
• developing weapons (e.g. bioweapons)
• wetlab work
• research
• acquiring resources
• avoiding government detection of its illicit activities

Such an entity as described could absolutely be an existential threat to humanity. It doesn't need to be superhuman at literally everything to be superhuman enough that we don't stand a chance if it decides to kill us.

So I feel like "RL may not work for everything, and will almost certainly work substantially better for easy to verify subjects" is... not so reassuring.

I once had a conversation with some math grad students. I mentioned there was a fluid dynamic problem that I was wondering about. I noticed that when a tap is pour out a thin stream of water and I obstruct it near the top, it makes a stable wavy pattern above where it is blocked. They weren't familiar with this phenomenon, so I said I'll show them. I went to the kitchen and turned on the tap with my finger on it. Someone said, (paraphrased,) "Oh, I thought you meant show a video."

Picture taken today

I still don't know the answer, I'd be delighted if anyone reading this linked to an explanation.