LessWrong 2.0 Reader

Because, based on the behavior of people here whose intelligence and ideas I have come to respect, this is an important topic.

Clearly I completely lack the background to understand the full theoretical argument. I also lack the background to understand the full theoretical argument behind general relatively and quantum uncertainty. Yet there are many real-world practical examples that I do understand and can work backwards from to get a roughly correct intuition about these ideas.

Every example I have seen for CDT falling short has been a hypothetical scenario that almost certainly never happened.

But if the only scenarios where CDT is a dominated strategy are hypothetical ones, I wouldn't expect smart people on LW to spend so much time and energy on them.

Entering, but not entered. The machines do not yet understand the prompts I write them. (Seriously, it's total garbage still, even with lots of high quality background material in the context.)

Thank you for responding to my post despite its negative rating.

Can you, as a human, give any practical real-world examples that do not rely on non-existent tech where anything outperforms non-naive CDT?

By non-naive I mean CDT that isn't myopically just trying to maximize the immediate payoff but rather trying to maximize the long term value to the player into account future interactions, reputation, uncertainty about causal relationships, etc.

Currently, during research programs such as MATS, many impactful AI Safety projects are being worked on

I think you could get to the problem faster than this. As I understand it, you're trying to motivate the shared repo by thinking about all of the duplicated work happening across the community & how valuable it would be for people trying to learn this style of research for the first time to work from a shared foundation. 

I think this is a pretty complex problem and needs to be called out more explicitly. Something like:

[For many early-career researchers, there's an unnecessarily steep learning curve for even figuring out what good norms for their research code should look like in the first place. We're all for people learning and trying things for themselves, but they should do that on top of a solid, trusted, well documented foundation. That's why things like e.g. the ARENA curriculum are so valuable.

But there aren't standardised templates/repos for doing most of the work in empirical alignment research, and we think this probably slows down new researchers a lot, and requires them to unnecessarily duplicate work and make mistakes that they might not notice are slowing them down. ML research in general involves so much tinkering and figuring things out, that building from a strong template can be a meaningful speedup and provide a helpful initial learning experience. 

For the MATS 7 scholars mentored by Ethan, Jan, Fabien, Mrinank, and others from the Anthropic Alignment Science team, we have created....

Mmm. You are entering the Cyborg Era. The only ideas you may take to the next epoch are those that can be uploaded to the machine intelligence. 

Yeah, I’m curious.

OK...

Some of this kind of puts words in your mouth by extrapolating from similar discussions with others. I apologize in advance for anything I've gotten wrong.

What's so great about failure?

This one is probably the simplest from my viewpoint, and I bet it's the one that's you'll "get" the least. Because it's basically my not "getting" your view at a very basic level.

Why would you ever even want to be able to fail big, in a way that would follow you around? What actual value do you get out of it? Failure in itself is valuable to you?

Wut?

It feels to me like a weird need to make your whole life into some kind of game to be "won" or "lost", or some kind of gambling addiction or something.

And I do have to wonder if there may not be a full appreciation for what crushing failure really is.

Failure is always an option

If you're in the "UBI paradise", it's not like you can't still succeed or fail. Put 100 years into a project. You're gonna feel the failure if it fails, and feel the success if it succeeds.

That's artificial? Weak sauce? Those aren't real real stakes? You have to be an effete pampered hothouse flower to care about that kind of made-up stuff?

Well, the big stakes are already gone. If you're on Less Wrong, you probably don't have much real chance of failing so hard that you die, without intentionally trying. Would your medieval farmer even recognize that your present stakes are significant?

... and if you care, your social prestige, among whoever you care about, can always be on the table, which is already most of what you're risking most of the time.

Basically, it seems like you're treating a not-particularly-qualitative change as bigger than it is, and privileging the status quo.

What agency?

Agency is another status quo issue.

Everybody's agency is already limited, severely and arbitrarily, but it doesn't seem to bother them.

Forces mostly unknown and completely beyond your control have made a universe in which you can exist, and fitted you for it. You depend on the fine structure constant. You have no choice about whether it changes. You need not and cannot act to maintain the present value. I doubt that makes you feel your agency is meaningless.

You could be killed by a giant meteor tomorrow, with no chance of acting to change that. More likely, other humans could kill you, still in a way you couldn't influence, for reasons you couldn't change and might never learn. You will someday die of some probably unchosen cause. But I bet none of this worries you on the average day. If it does, people will worry about you.

The Grand Sweep of History is being set by chaotically interacting causes, both natural and human. You don't know what most of them are. If you're one of a special few, you may be positioned to Change History by yourself... but you don't know if you are, what to do, or what the results would actually be. Yet you don't go around feeling like a leaf in the wind.

The "high impact" things that you do control are pretty randomly selected. You can get into Real Trouble or gain Real Advantages, but how is contingent, set by local, ephemeral circumstances. You can get away with things that would have killed a caveman, and you can screw yourself in ways you couldn't easily even explain to a caveman.

Yet, even after swallowing all the existing arbitrariness, new arbitrariness seems not-OK. Imagine a "UBI paradise", except each person gets a bunch of random, arbitrary, weird Responsibilities, none of them with much effect on anything or anybody else. Each Responsibility is literally a bad joke. But the stakes are real: you're Shot at Dawn if you don't Meet Your Responsibilities. I doubt you'd feel the Meaning very strongly.

... even though some of the human-imposed stuff we have already can seem too close to a bad joke.

The upshot is that it seems the "important" control people say they need is almost exactly the control they're used to having (just as the failures they need to worry about are suspiciously close to failures they presently have to worry about). Like today's scope of action is somehow automatically optimal by natural law.

That feels like a lack of imagination or flexibility.

And I definitely don't feel that way. There are things I'd prefer to keep control over, but they're not exactly the things I control today, and don't fall neatly into (any of) the categories people call "meaningful". I'd probably make some real changes in my scope of control if I could.

What about everybody else?

It's all very nice to talk about being able to fail, but you don't fail in a vaccuum. You affect others. Your "agentic failure" can be other people's "mishap they don't control". It's almost impossible to totally avoid that. Even if you want that, why do you think you should get it?

The Universe doesn't owe you a value system

This is a bit nebulous, and not dead on the topic of "stakes", and maybe even a bit insulting... but I also think it's related in an important way, and I don't know a better way to say it clearly.

I always feel a sense that what people who talk about "meaning" really want is value realism. You didn't say this, but this is what I feel like I see underneath practically everybody's talk about meaning:

Gosh darn it, there should be some external, objective, sharable way to assign Real Value to things. Only things that Real Value are "meaningful.

And if there is no such thing, it's important not to accept it, not really, not on a gut level...

... because I need it, dammit!

Say that or not, believe it or not, feel it or not, your needs, real or imagined, don't mean anything to the Laws that Govern All. They don't care to define Real Value, and they don't.

You get to decide what matters to you, and that means you have to decide what matters to you. Of course what you pick is ultimately caused by things you don't control, because you are caused by things you don't control. That doesn't make it any less yours. And it won't exactly match anybody else.

... and choosing to need the chance to fail, because it superficially looks like an externally imposed part of the Natural Order(TM), seems unfortunate. I mean, if you can avoid it.

"But don't you see, Sparklebear? The value was inside of YOU all the time!"

A practical exercise which is both fun and helps me think better? Sign me up.

I definitely enjoyed doing thinking physics exercises in my free time, they feel similar to chess in the way that they're a fun activity to do in my free time while also making me feel like I'm spending my time doing something really useful, which is really great to feel.

They also provide a tangible way of seeing your "prediction ability" for your own thinking and planning improve, which is helpful in staying motivated in regard to self-improvement exercises.

I can recommend to anyone on the fence about this to try their hands at a few thinking physics exercises!

In Holden Karnofsky's "AI Could Defeat All Of Us Combined" a plausible existential risk threat model is described, in which a swarm of human-level AIs outmanoeuvre humans due to AI's faster cognitive speeds and improved coordination, rather than qualitative superintelligence capabilities. This scenario is predicated on the belief that "once the first human-level AI system is created, whoever created it could use the same computing power it took to create it in order to run several hundred million copies for about a year each." If the first AGIs are as expensive to run as o3-high (costing ~$3k/task), this threat model seems much less plausible.

I wonder how different the reasoning paradigm is, actually, from the picture presented here. After all, running a huge number of AI copies in parallel is... scaling up test-time compute. 

The overhang argument is a rough analogy anyway. I think you are invoking the intuition of replacing the AI equivalent of a very large group of typical humans with the AI equivalent of a small number of ponderous geniuses, but those analogies are going to be highly imperfect in practice.

especially if it controls your social media feed

but... it already does :(

I mean, on facebook and xitter and reddit; I am still free to control my browsing of substack

and yes, applying the same level of control to my real life sounds like a bad idea

Thanks! Yeah, I think I would have said something pretty similar to that.

Actually, I might have gone a bit further and said:

Maybe, people have the experience

• (A) “First, I reprocessed the childhood scare experience. Second, I found some that my adult anxiety was generally relieved to some extent.”

 …and they naturally conclude

• (B) “…Therefore, the childhood scare experience must have been (partly) causing the adult anxiety all along.”

…but I wonder if we could also entertain an alternate theory:

• (B’) “…Gee, I guess this reprocessing must have been a kind of ‘training / practice / exercise’ during which I could forge new better subconscious habits and associations related to ‘the feeling of anxiety’ in general. And these new subconscious habits and associations are now serving me well in a wide variety of adult contexts.”

After all, you can’t form new subconscious habits and associations related to “the feeling of anxiety” except by invoking “the feeling of anxiety” somehow in the process. It seems plausible to me that childhood memories would be very effective way to do that. After all, (1) I think emotions are generally very strong in childhood and teenage years, and (2) maybe there’s some sense in which long-ago memories are objectively “safer” since the situation is long over, and thus it’s easier to entertain the idea that the feeling is not serving any real purpose.

Also, AFAICT, people achieve great therapeutic success by methods that involve bringing up childhood memories, but other people [LW · GW] also achieve great therapeutic success by methods that don’t.  :)

I’m not an expert like you are—indeed I have no personal experience whatsoever—so you can tell me if that doesn’t ring true.  :)