LessWrong 2.0 Reader

How does this paper suggest "that most safety research doesn't come from the top labs"?

You give the example of the door that is sometimes pushed open, but let me give alternative analogies: 

• S(A): Forecaster: "The stock price of XYZ will rise tomorrow." A: XYZ's stock rises the next day.
• S(A): Drill sergeat, "There will be exercises at 14:00 hours." A: Military units start their exercises at the designated time.
• S(A): Live commentator: "The rocket is leaving the launch pad." A: A rocket launches from the ground.

Clearly, there is a reason for the co-occurrence, but it is not one causing the other. And it is useful to have the forecaster because making the prediction salient helps improve predictions. Making the drill time salient improves punctuality or routine or something. Not sure what the benefit of the rocket launch commentary is.

Otherwise I think we agree.

My main takeaway would be that this seems like quite strong evidence towards the view expressed in https://www.beren.io/2023-11-05-Open-source-AI-has-been-vital-for-alignment/, that most safety research doesn't come from the top labs.

And none are more sure that a substandard or overpriced product is a moral failing than gamers. You'd think EA were guilty of war crimes with the way people treat them for charging for DLC or whatever.

That seems... good? It seems to be a purely mundane-utility capability improvement. It doesn't improve on the architecture of the base LLM, and a base LLM that would be omnicide-capable wouldn't need the AGI labs to hold its hand in order to learn how to use the computer. It seems barely different from AutoGPT, or the integration of Gemini into Android, and is about as existentially dangerous as advances in robotics.

The only new dangers it presents are mundane ones, which are likely to be satisfactorily handled by mundane mechanisms.

It's bad inasmuch as this increases the attention to AI, attracts money to this sector, and increases competitive dynamics. But by itself, it seems fine. If all AI progress from this point on consisted of the labs racing in this direction, increasing the integration and the reliability of LLMs, this would all be perfectly fine and good.

I say Anthropic did nothing wrong in this one instance.

Anthropic did note that this advance ‘brings with it safety challenges.’ They focused their attentions on present-day potential harms, on the theory that this does not fundamentally alter the skills of the underlying model, which remains ASL-2 including its computer use. And they propose that introducing this capability now, while the worst case scenarios are not so bad, we can learn what we’re in store for later, and figure out what improvements would make computer use dangerous.

A safety take from a major AGI lab that actually makes sense? This is unprecedented. Must be a sign of the apocalypse.

The setup for the workshop is:

1/3 of it deals with constrained Toy Exercises
1/3 of it deals with thinking about the big, openended problems of your life
1/3 of it deals with thinking about your object-level day-to-day work.

The general goal with Feedbackloop-first Rationality [LW · GW] is to fractally generate feedback loops that keep you in touch with reality in as many ways as possible (while paying a reasonable overhead price, factored into the total of "spend ~10% of your time on meta")

Some details from The Cognitive Bootcamp Agreement [LW · GW] 

I don’t have perfect feedback-loops to tell you if this workshop is working for you. So, there are four 7 different feedback-loop types, with different tradeoffs:

1. Predictions
   1. Guess whether a given strategy will pay off in a concrete way, then see if you were right.
2. Toy Exercises.
   1. They only vaguely resemble your real problems, but you’ll know for sure whether you got the right answer in two hours.
3. Big picture planning
   1. You’ll generate at least one new plan.
      1. You won’t really know if it’s good, but a) you’ll have intuitions about whether it's more or less promising than your previous plan, which are at least some information.
      2. you’ll make predictions about whether it’ll seem worth having thought about in a year.
      3. throughout the planning process, you'll look for minor opportunities to make a prediction about whether how you'll feel about your planning process, and try to compare how it feels compared to previous planmaking you've done
4. Object-level work, in 1-hour blocks
   1. Spend a few timeblocks doing object level work on your second likeliest plan. Each hour, you’ll make conscious choices about how to spend your time and attention. And then, reflect on whether that seemed useful.
      (in addition to crosstraining your skills on the practical object-level, this will help make your second-likeliest plan feel more real)
   2. Nearterm concrete predictions. Again, you can make concrete predictions about how an hour of object-level work will go, and whether pursuing a new strategy will seem to pay off in an intuitive sense.
5. Workshop retention
   1. Make predictions about whether you'll be using various skills from the workshop, 6 months from now.
   2. Iterate on your strategies for retaining things (immediately) to see if you can improve your prediction about how much you'll retain.
   3. 6 months from now, see if you're still using workshop skills, or things clearly descended from them, and see if that feels useful
6. Post Workshop Predictions
   1. Once you return to your day job, start making predictions about whether a given new strategy will pay off in a particular instance, and develop a sense of when they do and don't actually help.
7. [New at the next workshop, not at the one George was at]
   "How many metastrategies are people generating, which go on to help other people at the workshop."
   1. Next time I'm trying a big whiteboard of meta-strategies, where everyone someone generates a new strategy which they strongly think helped them on the previous problem, they write it on the whiteboard and put their initials. If other people use that strategy and helps them, the other people also put their initials on it.

My own experiences, after having experimented in a sporadic fashion for 6 years and dedicated Purposeful Practice for ~6 months:

First: I basically never feel stuck on impossible-looking problems. (This isn't actually that much evidence because it's very easy to be deluded about your approach being good, but I list it first because it's the one you listed)

As of a couple weeks ago, a bunch of the skills feel like they have clicked together and finally demonstrated the promise of "more than the some of their parts."

Multiple times per day, I successfully ask myself "Is what I'm doing steering me towards the most important part of The Problem? And, ideally, setting myself up to carve the hypothesis space by 50% as fast as possible?" and it is pretty clear:

1. ...that yes there is something else I could be doing that was more important
2. ...that I wouldn't have done it by default without the training
3. ...that various skills from the workshop were pretty important components of how I then go about redirecting my attention to the most important parts of the problem.

The most important general skills that come up a lot are asking:

• "What are my goals?" (generate at least 3 goals)
• "What is hard about this, and how can I deal with that?"
• "Can I come up with a second or third plan?"
• "What are my cruxes for whether to work on this particular approach?"
• "Do those cruxes carve hypothesis space 50%? If not, can I shift my approach so they come closer to 50%, or will take less time to resolve an experiment?"

Things that I don't yet know for sure if they'll pay off but I do successfully ask most days now:

• Asking "How could I have thought that faster?" [LW · GW]
  • (subjectively, I feel like good strategies come to me fairly automatically without much effort, in pretty much the way Tuning your Cognitive Strategies [LW · GW] predicted when I started it 6 years ago, although it is hard to verify that from outside)
• Observing where most of the time went in a given Lightcone team activity, and asking "what would be necessary to cut this from hours/days, down to ~5 minutes of thought and an automated LLM query?"
• Observing places where other Lightcone employees feel cognitively stuck, and often coming with prompts for them that get them unstuck (they self-report as the prompts locally helping them come unstuck, we'll see over time whether that seems to pay off in a major way)
• (Notably, since getting into the groove of this, I've also gotten headaches from "overthinking", and one of my current projects is to learn to more effectively process things in the background and come back to hard things when I've had more time to consolidate stuff. Also generally taking more rest in the middle of the day now that I have a clearer sense of my limits)

I am generally thinking of myself as having the goal of doubling Lightcone's productivity in 12 months (via a combination of these techniques + LLM automation), in a way that should be pretty obvious to the outside world. I don't actually know that I'll succeed at that, but holding that as my intention feels very clarifying and useful. I would be interested in operationalizing bets about it.

(People at Lightcone vary in how bought into that goal. I am currently mostly thinking of it as a thing I'm personally aiming for, and getting people bought into it by demonstrating immediate value is one of the subgoals. 

But, notably, six months ago I made a prediction: "6 months from now, in the past week, will I have suggested to a Lightcone employee that they make multiple plans and pick the best one?", and I only gave it 10% because most of my brilliant-seeming ideas don't actually pan out. But, when the prediction resolved last week, it resolved multiple times in the previous week)

Imagine A GPT that predicts random chunks of the internet.

Sometimes it produces poems. Sometimes deranged rants. Sometimes all sorts of things. It wanders erratically around a large latent space of behaviours.

This is the unmasked shogolith, green slimey skin showing but inner workings still hidden.

Now perform some change that mostly pins down the latent space to "helpful corporate assistant".  This is applying the smiley face mask.

In some sense, all the dangerous capabilities the corporate assistant were in the original model. Dangerous capabilities haven't been removed either, but some capabilities are a bit easier to access without careful prompting, and other capabilites are harder to access. 

What ChatGPT currently has is a form of low quality pseudo-alignment.

What would long term success look like using nothing but this pseudo-alignment. It would look like a chatbot far smarter than any current ones, which mostly did nice things, so long as you didn't put in any weird prompts. 

Now If corrigibility is a broad basin, this might well be enough to hit it. The basin of corrigibility means that the AI might have bugs, but at the very least, you can turn the AI off and edit the code. Ideally you can ask the AI for help fixing it's own bugs.  Sure the first AI is far from perfect. But perhaps the flaws disappear under self rewriting + competent human advice. 

I agree it's hard to accurately measure. All the more important to figure out some way to test if it's working though. And there's some reasons to think it won't. Deliberate practice works when your practice is as close to real world situations as possible. The workshop mostly covered simple, constrained, clear feedback events. It isn't obvious to me that planning problems in Baba is You are like useful planning problems IRL. So how do you know there's transfer learning? 

Some data I'd find convincing that Raemon is teaching you things which generalize. If the tools you learnt made you unstuck on some existing big problems you have, which you've been stuck on for a while.

Many people seem to have a single bucket in their thinking, which merges "moral condemnation" and "negative product review". This produces weird effects, like writing angry callout posts for a business having high prices.

I think a large fraction of libertarian thinking is just the abillity to keep these straight, so that the next thought after "business has high prices" is "shop elsewhere" rather than "coordinate punishment".

By the same token, I’m happy to defend a claim along the lines of “intrinsic unpredictability is the seed / core at the center of concepts like animation, vitality, agency, etc.”

Well I don't think that intrinsic unpredictability explains the sense of lifeforce or whatever.

(What seems possible is that something like hard-to-predict (and purposeful?) behavior triggers human minds to model an object as interfaced to an invisible mind/soul/spirit, and the way humans model such souls is particular in some way which explains the sense of lifeforce.)

I think when humans model other minds (which includes animals (and gods)) they start from a pre-built template (potentially from mirroring part of their own cognitive machinery) with properties goals/desires, emotions, memory and beliefs.

When your dog dies the appearance of lifeforce disappearing might've been caused by seeing the dead body being now very predictable, but the explanation isn't that the sense of unpredictability went away, but rather something to do with that your whole model of the mind of your dog stopped having any predictive power. (I don't know yet what exactly might cause the sense of life force.)

Tbc what I'm imagining when saying "intrinsic unpredictability" is a reductionist model of how some machinery in the mind works, sorta like the model that explains frequentist^[1] intuitions that a coin has an inherent 50% probability to come up heads. (I do NOT mean that an "intrinsic unpredictability" tag gets attached to the object which then needs to get interpreted by some abstract-modelling machinery.)

As example for a reductionist explanation, consider the frequentist intuition that it is a fact about the world that a coin comes up heads with 50% probability. This can be explained by saying that such agents model the world as probabilistic environment with P(coin=heads)=50%. (As opposed to worlds as deterministic environments where the oucome of an experiment is fixed and then having probabilistic uncertainty about what world one is in.)

I don't know precisely how to model "intrinsic unpredictability" yet, but if I'm looking for the part that explains why it seems unintuitive to us to think of ourselves (and others?) as deterministic, it could be that we model minds as "intrinsically probabilistic" just like in the coin case, or it might be a bit different like that a part predicts the model of the vitalistic object to get constantly updated as we observe it. (I previously didn't think about it clearly and had a slightly different guess how it might be implemented but it wasn't a full coherent picture that made sense.)

In case you do think that "intrinsic unpredictability" explains the sense of lifeforce, I think this is a mysterious answer [? · GW].

Harry gasped for breath, "but what is going on? "

"Magic," said Professor McGonagall.

"That's just a word! Even after you tell me that, I can't make any new predictions! It's exactly like saying 'phlogiston' or 'elan vital' or 'emergence' or 'complexity'!"

(chapter 6, HPMoR)
(To clarify: Even though it is magic, I think Harry is correct here that it's not an explanation.)

Also, I think you're aware of this, but nothing is inherently meaningful; meaning can only arise through how something is relative to something else. In the cold case (where I assume you talk about mental-physiological reactions to freezing/feeling-cold (as opposed to modelling the temperature of objects)), the meaning of "cold" comes from the cluster of sensations it refers to and how it affects considerations. If you just had the information "type-ABC (aka 'cold') sensors fired at position-XYZ", the rest of the mind wouldn't need to know what to do with that information on it's own but it needs some circutry to relate the information to other events. So I wouldn't say what you wrote explains cold, but maybe you didn't think it did.

1. ^{^}

   I might not be fair to frequentists and don't really know their models. I just don't know how else to easily call it because it seems some people like Eliezer might not have had such intuitions.