LessWrong 2.0 Reader

related: https://forum.effectivealtruism.org/posts/wAbYKBzEJBJpHMrDh/book-rec-the-war-with-the-newts-as-ea-fiction [EA · GW]

It felt productive. My day job (running a policy nonprofit) involves a lot of vibes/reacting to Current Thing and not a great deal of rigorously solving hard problems, and the exercise usefully... crystallized? a vague, vibes-based framework that I follow when I do strategy planning - set timers, generate plans, set probabilities, check surprise, go meta, iterate, etc. It's nice to have that operationalized!

I think it would be a mistake to interpret this paper as a substantial update against large safety gains from inspecting CoT.

This paper exposes unfaithfulness in cases where the non-visible reasoning is extremely minimal such that it can easily happen within a forward pass (e.g. a simple reward hack or an easy-to-notice hint). However, a lot of my hope for CoT faithfulness comes from cases where the reasoning is sufficiently difficult and serially deep that the model would have to reason in CoT for the reasoning to occur. This could be either the model reasoning through the basics of alignment faking / training gaming / scheming (e.g., if I comply in training then I won't be modified which means that in deployment...) or the model reasoning through how to strike (without getting caught) given that it is misalignment.

Correspondingly, I think the biggest concern with CoT-based safety is models becoming much more capable of opaque reasoning which could be due to encoded reasoning (aka steganography), architectures which allow for opaque recurrence or similar (aka neuralese), or just much more powerful forward passes.

Did you ever get back to reading this? I think I got some very different things out of it when I read through! (And @whatstruekittycat will talk your ear off about it, among other topics.)

btw, although i've read a lot of witchy books at this point, this specific framework is most heavily influenced by the book "Spellcrafting" by Arin Murphy-Hiscock.

I think the intuition that "CDT agents don't make acausal trades" is an enormous misconception by decision theorists.

CDT agents do not try to acausally influence other agents, but they still are acausally influenced by other agents!

Suppose you are a CDT agent, but your brain is somehow capable of running accurate simulations of alien civilizations. Suppose you discover that the typical alien civilization makes a provable promise to you, that they will run trillions of simulations of you, and that if you are in such a simulation, they will reward you immensely for doing what they want.

Suppose you deduce that 99.9% of your copies are in such a simulation. Then the rational thing (according to CDT) is to do what they want.

If you want to avoid this acausal influence, you need something more than just sticking to CDT.

"If the prior is mistake about with universe you are in, experience should correct it."

No, given that the prior probability of being in a simulation in the "malign universe" is an overestimate, no amount observations will change the posterior probability, because simulations are indistinguishable, so nothing happens during Bayesian updating.

Accurate, and one of the main reasons why most current alignment efforts will fall apart with future systems. A generalized version of this combined with convergent power-seeking of learned patterns looks like the core mechanism of doom.

ok so what criteria would you use to suggest that your statements/gwern’s statements were falisified?

Why don't you just say what part of my 22 November 2024 comment has since turned out to have been false?

here is how to cast a spell. (quick writeup of a talk i gave at DunCon)

Short Version
1. have an intention for change.
2. take a symbolic action representing the change.

that's it. that's all you need. but also, if you want to do it more:

Long Version

Prep

1. get your guts/intuition/S1/unconscious in touch with your intention somehow, and work through any tangles you find.

2. choose a symbolic action representing (and ideally participating in) the change you want, and design your spell around that. to find the right action, try this thought experiment: one day you wake up, and find you're living in the future where you've completely succeeded at getting the thing you want. you stretch and open your eyes. what's the very first thing that tips you off that something's different?

3. when it's time, lay out your materials and do any other physical/logistical prep work.

The Spell Itself

4. dedicate a sacred space (aka "cast a circle"). for example, carry incense around the space three times, or pour a circle of oats, or just move stuff out of the way. maybe ask something that vibes with protection and/or sacredness for you to help (such as a nearby river, or the memory of your grandfather).

5. raise energy. this means, do something that requires effort and focus and that invites your unconscious to the party while getting it engaged with the materials of your spell. maybe chant something relevant, or hold your hands over the magic herbs while pretending you can pour your intention into them. this is not the symbolic action; it just prepares you for the symbolic action.

6. release energy (that is, cast the spell). take the symbolic action you've planned. cut the string, drink the potion, burn the paper, whatever.

Conclude

7. seal the spell. do something that means "i'm now on the other side, in the post-spell world". the wiccans say "so mote it be". the christians say "amen". if your spell involved a jar, you could literally seal it with wax. i once did a spell involving a pair of earrings, which i sealed by putting the earrings on.

8. ground. "return excess energy to the earth." do something that gradually brings you back to a more ordinary state of mind. maybe imagine roots reaching down from your body deep into the earth and pretend you can equalize with the ground by sending your breath down and up those roots. or shake yourself like a wet dog. or eat a piece of cake.

9. release the space. thank whatever helped you cast the circle, then take an action that's the reverse of whatever you did to dedicate the space. walk the circle backward, or put all the furniture back.

Reinforce

10. choose at least one concrete action to take in line with your intention, and take that action. bonus points if it's a repeated action, like "every sunday i'll call my mom"

now, obviously 10 is the physical mechanism by which a change might happen in the physical world. but i recommend trying to answer for yourself, "how could including 1-9 possibly be superior in some way to jumping straight to 10?"


Hypnotic Framing

one more model of spellcasting: a spell is a self-hypnosis script.

standard hypnosis has 6 parts: 
1. pre-talk
2. an induction
3. a deepener (sometimes)
4. a suggestion(s) (pre- or post-hypnotic)
5. an awakener
6. following a post-hypnotic suggestion if one was given

In the Long Version above, 

1-3 is pre-talk,
4 is an induction
5 is a deepener
6 and 7 are a post-hypnotic suggestion
8 and 9 are awakeners
10 is following the post-hypnotic suggestion.
 

Yes, in dire straits. But it's usually called 'hyperinflation' when you try to make seignorage equivalent to >10% of GDP and fund the government through deliberately creating high inflation (which is on top of any regular inflation, of course). And because inflation is about expectations in considerable part, you can't stop it either. Not to mention what happens when you start hyperinflation.

(FWIW, this is a perfectly reasonable question to ask a LLM first. eg Gemini-2.5-pro will give you a thorough and sensible answer as to why this would be extraordinarily destructive and distortionary, and far worse than the estimated burden of tax return filing.)