LessWrong 2.0 Reader

Mod note: I normally leave this sort of post on personal blog because they are pretty niche, but, I frontpaged this one because it was a) short, and b) the principle seemed like it might generalize to other marketing situations.

Hmm, I wasn’t thinking about that because that sentence was nominally in someone else’s voice. But you’re right. I reworded, thanks.

Tesla fans will often claim that Tesla could easily do this

Tesla fan here. 

Yes, Tesla can easily do the situation you've described (stop and go traffic on a highway in good weather with no construction). With higher reliability than human beings.

I suspect the reason Tesla is not pursuing this particular certification is because given the current rate of progress it would be out of date by the time it was authorized.  There have been several significant leaps in capabilities in the last 2 years (11->12, 12->12.6, and I've been told 12->13).  Most likely Elon (who has undeniably been over optimistic) is waiting to get FSD certified until it is at least level 4.

It's worth noting that Tesla has significantly relaxed the requirements for FSD (from "hands on wheel" to "eyes on road") and has done so for all circumstances, not just optimal ones.

thank you :) 

Some have delusional optimism about [...]

I'm usually not a fan of tone-policing [LW · GW], but in this case, I feel motivated to argue that this is more effective if you drop the word "delusional." The rhetorical function of saying "this demo is targeted at them, not you" is to reassure the optimist that pessimists are committed to honestly making their case point by point, rather than relying on social proof and intimidation tactics to push a predetermined [LW · GW] "AI == doom" conclusion. That's less credible if you imply that you have warrant to dismiss all claims of the form "Humans and institutions will make reasonable decisions about how to handle AI development and deployment because X" as delusional regardless of the specific X.

("well approximates AIXI" seems unlikely to be an accurate description to me. Maybe "closer to what AIXI is doing than to base GPT-4 without a sampling loop" or "dominates human competence" are more accurate.)

Zvi has an expansion on the vibes-based vs gears-based thinking model that I have found useful for thinking about politics: his take on Simulacra levels. [LW · GW]

Nod. So, first of all: I don't know. My own guesses would depend on a lot of details of the individual person (even those in a similar situation to you).

(This feels somewhat outside the scope of the main thrust of this post, but, definitely related to my broader agenda of 'figure out a training paradigm conveying the skills and tools necessary to solve very difficult, confusing problems')

But, riffing so far. And first, summarizing what seemed like the key points:

• You want to predict optimization daemons as they arise in a system, and want a good mathematical basis for that, and don't feel satisfied with the existing tools.
• You're currently exploring this on-the-side while working on some more tractable problems.
• You've identified two broad-strategies, which are:
  • somehow "deliberate practice" this,
  • somehow explore and follow your curiosity intermitently

Three things I'd note:

• "Deliberate practice" is very openended. You can deliberate practice noticing and cultivating veins of curiosity, for example.
• You can strategize about how to pursue curiosity, or explore (without routing through the practice angle)
• There might be action-spaces other than "deliberate practice" or "explore/curiosity" that will turn out to be useful.

My current angle for deliberate practice is to find problem sets that feel somehow-analogous to the one you're trying to tackle, but simpler/shorter. They should be difficult enough that they feel sort of impossible while you're working on them, but, also actually solvable. They should be varied enough that you aren't overfitting to one particular sort of puzzle.

After the exercise, apply the Think It Faster [LW · GW] meta-exercise to it.

Part of the point here is notice strategies like "apply explicit systematic thinking" and strategies like "take a break, come back to it when you feel more inspired", and start to develop your own sense of which strategies work best for you.

Thanks for the feedback and references!

On catastrophic forgetting: our appendix includes a "control" version of ERA that doesn't use gradient routing but is otherwise the same (appendix C, figure 12). This shows that the effect of retain-set fine-tuning is negligible in the absence of gradient routing.

On gradient ascent or similar methods: there are many unlearning methods that don't target or achieve the kind of robust localization and removal that we care about, as mentioned in our discussion of related works, and, e.g., in this post [LW · GW]. We included RMU as a stand-in for this class, and I personally don't see much value in doing more extensive comparisons there.

On Corrective Unlearning: we weren't aware of other unlearning approaches that consider imperfect labeling, so this is a very helpful reference-- thanks! It would be interesting to compare ERA-type methods to these. My concern with fine-tuning methods is that they might not be suitable for robustly removing broader capabilities (like, "virology") as opposed to correcting for small perturbations to the training data.

In a perfect world, everyone would be concerned about the risks for which there are good reasons to be concerned, and everyone would be unconcerned about the risks for which there are good reasons to be unconcerned, because everyone would be doing object-level checks of everyone else’s object-level claims and arguments, and coming to the correct conclusion about whether those claims and arguments are valid.

And those valid claims and arguments might involve demonstrations and empirical evidence, but also might be more indirect.

See also: It is conceivable for something to be an x-risk without there being any nice clean quantitative empirically-validated mathematical model proving that it is [LW · GW].

I do think Turing and von Neumann reached correct object-level conclusions via sound reasoning, but obviously I’m stating that belief without justifying it.