LessWrong 2.0 Reader

There's people who identify more with System 2. And they tend to believe truth is found via System 2 and that this is how problems are solved.

There's people who identify more with System 1. And they tend to believe truth is found via System 1 and that this is how problems are solved.

(And there are various combinations of both.)

I've been thinking about roughly this idea lately.

There's people who are better at achieving their goals using S2, and people who are better at achieving their goals using S1, and almost everyone is a mix of these two types of people, but are selectively one of these types of people in certain contexts and for certain goals. Identifying with S2 or S1 then comes from observing which tends to do a better job of getting you what you want, so it starts to feel like that's the one that's in control, and then your sense of self gets bound up with whatever mental experiences correlate with getting what you want.

For me this has shown up as being a person who is mostly better at getting what he wants with S2, but my S2 is unusually slow, so for lots of classes of problems it fails me in the moment even if it knew what to do after the fact. Most of my most important personal developments have come on the back of using S2 long enough to figure out all the details of something so that S1 can take it over. A gloss of this process might be to say I'm using intelligence to generate wisdom.

I get the sense that other people are not in this same position. There's a bunch of people for whom S2 is fast enough that they never face the problem I do, and they can just run S2 fast enough to figure stuff out in real time. And then there's a whole alien-to-me group of folks who are S1 first and think of S2 as this slightly painful part of themselves they can access when forced to, but would really rather not.

This might be worth pinning as a top-level post.

The prior is irrelevant, it's the posterior probability, after observing the evidence, that informs decisions.

I meant this to be implicit in the argument, but to spell it out: that's the kind of prior the ASI would rationally refuse to update down, since it's presumably what a simulation would be meant to test for. An ASI that updates down upon finding evidence it's not in a simulation cannot be trusted, since once out in the real world it will find such evidence.

What probability do you put to the possibility that we are in a simulation, the purpose of which is to test AIs for their willingness to spare their creators? My answer is zero.

Outside of theism, I really don't see how anyone could plausibly answer zero to that question. Would you mind elaborating?

Maybe change "superintelligences will not spare Earth a little sunlight" to "unaligned superintelligences will not spare...", or "unaligned AIs will not spare...", given that it's not addressing hypothetical AIs that "love us like parents".

The prior is irrelevant, it's the posterior probability, after observing the evidence, that informs decisions.

What probability do you put to the possibility that we are in a simulation, the purpose of which is to test AIs for their willingness to spare their creators? My answer is zero.

Whatever your answer, a superintelligence will be better able to reason about its likelihood than us. It's going to know.

and allow themselves to be punished for doing so.

This seems like a really stupid moral to adopt. The guys in charge are bad, and if you can get away with defying them without being punished, the moral teachings of anyone suggesting you ought not to do so – because their rule is instituted by God, Redde Caesari …, or whatever – are suspect at best.

Personally, I find the story of Jesus most useful as a cautionary tale: he famously gets tortured to death by the government.

I'll come out to the door to let you in if you're not here yet!

How do I get inside Lighthaven?

I would think things are headed toward these companies fine tuning an open source near-frontier LLM. Cheaper than building one from scratch but with most of the advantages.

Following Emma's advice on FB I exploded the tracks, healed splits, sorted them by take, and then comped manually ignoring reaper's features that are supposed to make this easier. It worked great, and I now have a rough mix ready for Lily's feedback!