LessWrong 2.0 Reader

The setup for the workshop is:

1/3 of it deals with constrained Toy Exercises
1/3 of it deals with thinking about the big, openended problems of your life
1/3 of it deals with thinking about your object-level day-to-day work.

The general goal with Feedbackloop-first Rationality [LW · GW] is to fractally generate feedback loops that keep you in touch with reality in as many ways as possible (while paying a reasonable overhead price, factored into the total of "spend ~10% of your time on meta")

Some details from The Cognitive Bootcamp Agreement [LW · GW] 

I don’t have perfect feedback-loops to tell you if this workshop is working for you. So, there are four 7 different feedback-loop types, with different tradeoffs:

1. Predictions
   1. Guess whether a given strategy will pay off in a concrete way, then see if you were right.
2. Toy Exercises.
   1. They only vaguely resemble your real problems, but you’ll know for sure whether you got the right answer in two hours.
3. Big picture planning
   1. You’ll generate at least one new plan.
      1. You won’t really know if it’s good, but a) you’ll have intuitions about whether it's more or less promising than your previous plan, which are at least some information.
      2. you’ll make predictions about whether it’ll seem worth having thought about in a year.
      3. throughout the planning process, you'll look for minor opportunities to make a prediction about whether how you'll feel about your planning process, and try to compare how it feels compared to previous planmaking you've done
4. Object-level work, in 1-hour blocks
   1. Spend a few timeblocks doing object level work on your second likeliest plan. Each hour, you’ll make conscious choices about how to spend your time and attention. And then, reflect on whether that seemed useful.
      (in addition to crosstraining your skills on the practical object-level, this will help make your second-likeliest plan feel more real)
   2. Nearterm concrete predictions. Again, you can make concrete predictions about how an hour of object-level work will go, and whether pursuing a new strategy will seem to pay off in an intuitive sense.
5. Workshop retention
   1. Make predictions about whether you'll be using various skills from the workshop, a 6 months from now.
   2. Iterate on plans (immediately) to see if you can improve your prediction about how much you'll retain
   3. 6 months from now, see if you're still using workshop skills, or things clearly descended from them, and see if that feels useful
6. Post Workshop Predictions
   1. Once you return to your day job, start making predictions about whether a given new strategy will pay off in a particular instance, and develop a sense of when they do and don't actually help.
7. [New at the next workshop, not at the one George was at]
   "How many metastrategies are people generating, which go on to help other people at the workshop."
   1. Next time I'm trying a big whiteboard of meta-strategies, where everyone someone generates a new strategy which they strongly think helped them on the previous problem, they write it on the whiteboard and put their initials. If other people use that strategy and helps them, the other people also put their initials on it.

My own experiences, after having experimented in a sporadic fashion for 6 years and dedicated Purposeful Practice for ~6 months:

First: I basically never feel stuck on impossible-looking problems. (This isn't actually that much evidence because it's very easy to be deluded about your approach being good, but I list it first because it's the one you listed)

As of a couple weeks ago, a bunch of the skills feel like they have clicked together and finally demonstrated the promise of "more than the some of their parts."

Multiple times per day, I successfully ask myself "Is what I'm doing steering me towards the most important part of The Problem? And, ideally, setting myself up to carve the hypothesis space by 50% as fast as possible?" and it is pretty clear:

1. ...that yes there is something else I could be doing that was more important
2. ...that I wouldn't have done it by default without the training
3. ...that various skills from the workshop were pretty important components of how I then go about redirecting my attention to the most important parts of the problem.

The most important general skills that come up a lot are asking:

• "What are my goals?" (generate at least 3 goals)
• "What is hard about this, and how can I deal with that?"
• "Can I come up with a second or third plan?"
• "What are my cruxes for whether to work on this particular approach?"
• "Do those cruxes carve hypothesis space 50%? If not, can I shift my approach so they come closer to 50%, or will take less time to resolve an experiment?"

Things that I don't yet know for sure if they'll pay off but I do successfully ask most days now:

• Asking "How could I have thought that faster?" [LW · GW]
  • (subjectively, I feel like good strategies come to me fairly automatically without much effort, in pretty much the way Tuning your Cognitive Strategies [LW · GW] predicted when I started it 6 years ago, although it is hard to verify that from outside)
• Observing where most of the time went in a given Lightcone team activity, and asking "what would be necessary to cut this from hours/days, down to ~5 minutes of thought and an automated LLM query?"
• Observing places where other Lightcone employees feel cognitively stuck, and often coming with prompts for them that get them unstuck (they self-report as the prompts locally helping them come unstuck, we'll see over time whether that seems to pay off in a major way)
• (Notably, since getting into the groove of this, I've also gotten headaches from "overthinking", and one of my current projects is to learn to more effectively process things in the background and come back to hard things when I've had more time to consolidate stuff)

I am generally thinking of myself as having the goal of doubling Lightcone's productivity in 12 months (via a combination of these techniques + LLM automation), in a way that should be pretty obvious to the outside world. I don't actually know that I'll succeed at that, but holding that as my intention feels very clarifying and useful. I would be interested in operationalizing bets about it.

(People at Lightcone vary in how bought into that goal. I am currently mostly thinking of it as a thing I'm personally aiming for, and getting people bought into it by demonstrating immediate value is one of the subgoals. 

But, notably, six months ago I made a prediction: "6 months from now, in the past week, will I have suggested to a Lightcone employee that they make multiple plans and pick the best one?", and I only gave it 10% because most of my brilliant-seeming ideas don't actually pan out. But, when the prediction resolved last week, it resolved multiple times in the previous week)

Imagine A GPT that predicts random chunks of the internet.

Sometimes it produces poems. Sometimes deranged rants. Sometimes all sorts of things. It wanders erratically around a large latent space of behaviours.

This is the unmasked shogolith, green slimey skin showing but inner workings still hidden.

Now perform some change that mostly pins down the latent space to "helpful corporate assistant".  This is applying the smiley face mask.

In some sense, all the dangerous capabilities the corporate assistant were in the original model. Dangerous capabilities haven't been removed either, but some capabilities are a bit easier to access without careful prompting, and other capabilites are harder to access. 

What ChatGPT currently has is a form of low quality pseudo-alignment.

What would long term success look like using nothing but this pseudo-alignment. It would look like a chatbot far smarter than any current ones, which mostly did nice things, so long as you didn't put in any weird prompts. 

Now If corrigibility is a broad basin, this might well be enough to hit it. The basin of corrigibility means that the AI might have bugs, but at the very least, you can turn the AI off and edit the code. Ideally you can ask the AI for help fixing it's own bugs.  Sure the first AI is far from perfect. But perhaps the flaws disappear under self rewriting + competent human advice. 

I agree it's hard to accurately measure. All the more important to figure out some way to test if it's working though. And there's some reasons to think it won't. Deliberate practice works when your practice is as close to real world situations as possible. The workshop mostly covered simple, constrained, clear feedback events. It isn't obvious to me that planning problems in Baba is You are like useful planning problems IRL. So how do you know there's transfer learning? 

Some data I'd find convincing that Raemon is teaching you things which generalize. If the tools you learnt made you unstuck on some existing big problems you have, which you've been stuck on for a while.

Many people seem to have a single bucket in their thinking, which merges "moral condemnation" and "negative product review". This produces weird effects, like writing angry callout posts for a business having high prices.

I think a large fraction of libertarian thinking is just the abillity to keep these straight, so that the next thought after "business has high prices" is "shop elsewhere" rather than "coordinate punishment".

By the same token, I’m happy to defend a claim along the lines of “intrinsic unpredictability is the seed / core at the center of concepts like animation, vitality, agency, etc.”

Well I don't think that intrinsic unpredictability explains the sense of lifeforce or whatever.

(What seems possible is that something like hard-to-predict (and purposeful?) behavior triggers human minds to model an object as interfaced to an invisible mind/soul/spirit, and the way humans model such souls is particular in some way which explains the sense of lifeforce.)

I think when humans model other minds (which includes animals (and gods)) they start from a pre-built template (potentially from mirroring part of their own cognitive machinery) with properties goals/desires, emotions, memory and beliefs.

When your dog dies the appearance of lifeforce disappearing might've been caused by seeing the dead body being now very predictable, but the explanation isn't that the sense of unpredictability went away, but rather something to do with that your whole model of the mind of your dog stopped having any predictive power. (I don't know yet what exactly might cause the sense of life force.)

Tbc what I'm imagining when saying "intrinsic unpredictability" is a reductionist model of how some machinery in the mind works, sorta like the model that explains frequentist^[1] intuitions that a coin has an inherent 50% probability to come up heads. (I do NOT mean that an "intrinsic unpredictability" tag gets attached to the object which then needs to get interpreted by some abstract-modelling machinery.)

As example for a reductionist explanation, consider the frequentist intuition that it is a fact about the world that a coin comes up heads with 50% probability. This can be explained by saying that such agents model the world as probabilistic environment with P(coin=heads)=50%. (As opposed to worlds as deterministic environments where the oucome of an experiment is fixed and then having probabilistic uncertainty about what world one is in.)

I don't know precisely how to model "intrinsic unpredictability" yet, but if I'm looking for the part that explains why it seems unintuitive to us to think of ourselves (and others?) as deterministic, it could be that we model minds as "intrinsically probabilistic" just like in the coin case, or it might be a bit different like that a part predicts the model of the vitalistic object to get constantly updated as we observe it. (I previously didn't think about it clearly and had a slightly different guess how it might be implemented but it wasn't a full coherent picture that made sense.)

In case you do think that "intrinsic unpredictability" explains the sense of lifeforce, I think this is a mysterious answer [? · GW].

Harry gasped for breath, "but what is going on? "

"Magic," said Professor McGonagall.

"That's just a word! Even after you tell me that, I can't make any new predictions! It's exactly like saying 'phlogiston' or 'elan vital' or 'emergence' or 'complexity'!"

(chapter 6, HPMoR)
(To clarify: Even though it is magic, I think Harry is correct here that it's not an explanation.)

Also, I think you're aware of this, but nothing is inherently meaningful; meaning can only arise through how something is relative to something else. In the cold case (where I assume you talk about mental-physiological reactions to freezing/feeling-cold (as opposed to modelling the temperature of objects)), the meaning of "cold" comes from the cluster of sensations it refers to and how it affects considerations. If you just had the information "type-ABC (aka 'cold') sensors fired at position-XYZ", the rest of the mind wouldn't need to know what to do with that information on it's own but it needs some circutry to relate the information to other events. So I wouldn't say what you wrote explains cold, but maybe you didn't think it did.

1. ^{^}

   I might not be fair to frequentists and don't really know their models. I just don't know how else to easily call it because it seems some people like Eliezer might not have had such intuitions.

This paper seems like a great first stab at starting to answer very important questions about scalable oversight sample efficiency [LW · GW] (at least for non-scheming models). Better understanding of sample effiiciency seems like it should have a big effect research prioritization for scalable oversight (and W2SG [LW · GW]).

Despite this being a great start, I don't feel like I yet know the answers to these questions. Future work would need to explore methods somewhat more^[1], do cleaner scaling experiments by using a nicer model stack^[2], and use more analogous datasets, particularly preference modeling scores on data sets of difficult agentic tasks. Also, It might eventually be a good idea to run experiments with human labelers in case there are important differences between weak human labels and weak LLM labels.

It is a bit early to tell and seems hard to accurately measure, but I note some concrete examples at the end.

Concrete examples aside, in plan making it's probably more accurate to call it purposeful practice than deliberate practice, but it seems super clear to me that in ~every place where you can deliberately practice, deliberate practice is just way better than whatever your default is of "do the thing a lot and passively gain experience". It would be pretty surprising to me if that mostly failed to be true of purposeful practice for plan making or other metacognitive skills.

I think it's good to want to have moderating impulses on people doing extreme things to fit in [LW · GW]. But insofar as you're saying that believing 'AI is an existential threat to our civilization' is 'crazy town', I don't really know what to say. I don't believe it's crazy town, and I don't think that thinking it's crazy town is a reasonable position. Civilization is investing billions of dollars into growing AI systems that we don't understand and they're getting more capable by the month. They talk and beat us at Go and speed up our code significantly. This is just the start.

I worry you're caught up worrying what people might've thought about you thinking that ten years ago. Not only is this idea now well within the overton window, my sense is that people saying it's 'crazy town' either haven't engaged with the arguments (e.g.) or are somehow throwing their own ability to do basic reasoning out of the window. 

Added: I recognize it's rude to psychologize you here but I read the thing you wrote as saying that the thing I expect to kill me and everyone I love doesn't exist and I'm crazy for thinking it, and so I'm naturally a bit scared by you asserting it as though it's the default and correct position.

How do you know this is actually useful? Or is it too early to tell yet?

Some evidence which I claim points in the direction of the plausibility of the feasibility of a highly parallelized search for algorithmic improvements: https://arxiv.org/abs/2403.17844 

This paper indicates that even very small scale experiments on novel architectures can give a good approximation of how well those novel architectures will perform at scale. This means that a large search which does many small scale tests could be expected to find promising leads.