LessWrong 2.0 Reader

I don't know if this is just hindsight, but tracr has in no way turned out to be safety relevant. Was it considered to be so at the time of commenting?

I sent the following offer (lightly edited) to discuss metaphysics to plex. I extend the same to anyone else.^[1]

(Note: I don't do psychedelics or participate in the mentioned communities (except arguably the qualia research institute discord server), and I'm deeply suspicious of 'intuitions'. I notice unquestioned intuitions across views on this, and the primary thing I'd like to do in discussing is try to get the other to see theirs.)

Hi, I saw [this shortform] by you

Wondering if you'd want to discuss metaphysics itself / let me socratic-question^[2] you. I lean qualia-monist but I'd like to mostly see if I can socratic-question you into noticing incoherence. I'm too uncertain about the true metaphysics to put forth a totalizing theory.

It might help if you've read this post [LW · GW] first.

My starting questions:

1. Do you believe phenomenal qualia exist? If you do, are they mathematical objects? If you don't, can you watch this short video on a thought experiment?

   (Commentary: someone in the comments of the linked post [LW · GW] told me that video was what changed their position from 'camp 1' (~illusionist, e.g. "qualia is an intuition from evolution") to 'camp 2' ("qualia is metaphysically fundamental"))

2. You mentioned 'matter' and 'math' as 'fundamental to reality'. My question depends on which is your view.

   1. If you believe matter is fundamental, what do you mean by matter; how does it differ from math?

      (Commentary: "matter" seems like an intuitive concept; one has an intuition that reality is "made of something", and calls that "thing" (undefined) "matter", and refers to "matter" without knowing "what it is" (also undefined); there being a word for this intuition prevents them from noticing their lack of knowledge)

   2. If you believe math is fundamental, what distinguishes this particular mathematical universe from other ones; what qualifies this world as "real", if anything; what 'breathes fire into the equations and creates a world for them to describe'?

      (Commentary: one self-consistent position answers "nothing" - that this world is just one of the infinitely many possible mathematical functions / programs. That 'real' secretly means 'the program(s?) we are part of'. Though I observe this position to be rare; most have a strong intuition that there is something which "makes reality real".)

1. ^{^}

   Though I'll likely lose interest if it seems like we're talking past each other / won't resolve any cruxy disagreements.

2. ^{^}

   (Questioning with the goal of causing the questioned one to notice specific assumptions or intuitions to their beliefs, as a result of trying to generate a coherent answer)

One upshot of this is that you can follow an explicitly non-(precise-)Bayesian decision procedure and still avoid dominated strategies. For example, you might explicitly specify beliefs using imprecise probabilities and make decisions using the “Dynamic Strong Maximality” rule, and still be immune to sure losses [LW · GW]. Basically, Dynamic Strong Maximality tells you which plans are permissible given your imprecise credences, and you just pick one. And you could do this “picking” using additional substantive principles. Maybe you want to use another rule for decision-making with imprecise credences (e.g., maximin expected utility or minimax regret). Or maybe you want to account for your moral uncertainty (e.g., picking the plan that respects more deontological constraints).

Let's say Alice have imprecise credences. Let's say Alice follows the algorithm: "At each time-step t, I will use 'Dynamic Strong Maximality' to find all plans that aren't dominated. I will pick between them using [some criteria]. Then I will take the action that plan recommends." (And then at the next timestep t+1, you re-do everything I just said in the quotes.)

If Alice does this, does she ended up being dynamically inconsistent? (Vulnerable to dutch-books etc.)

(Maybe it varies depending on the criteria. I'm interested if you have a hunch for what the answer will be for the sort of criteria you listed: maximin expected utility, minimax regret, picking the plan that respects more deontological constraints.)

I.e., I'm interested in: If you want to use dynamic strong maximality to avoid dominated strategies, does that require you to either have the ability to commit to a plan or the inclination to consistently pick your plan from some prior epistemic perspective. (Like an "updateless" agent might.) Or do you automatically avoid dominated strategies even if you're constantly recomputing your plan?

My sapience spell is to pay attention to the sensations of breathing at my abdomen whenever I notice my hair is in my face.

What's the case for it being a swiss cheese approach? That doesn't match how I think of it. 

I gave a lightning talk with my particular characterization, and included "swiss cheese" i.e. that gsai sources some layers of swiss cheese without trying to be one magic bullet. But if people agree with this, then really guaranteed-safe ai is a misnomer, cuz guarantee doesn't evoke swiss cheese at all

I think if you're describing planecrash as "the single work of fiction for which I most want to avoid spoilers", you probably just shouldn't read any reviews of it or anything about it until after you've read it.

If you do read this review beforehand, you should avoid the paragraph that begins with "By far the best …" (The paragraph right before the heading called "The competence".) That mentions something that I definitely would have considered a spoiler if I'd read it before I read planecrash.

Aside from that, it's hard to answer without knowing what kinds of things you consider spoilers and what you already know about planecrash.

I've updated over time to thinking that trusting AI systems and mostly handing things off to AI systems is an important dynamic to be tracking. As in, the notion of human researcher obsolescence discussed here [LW · GW]. In particular, it seems important to understand what is required for this and when we should make this transition.

I previously thought that this handoff should be mostly beyond our planning horizon (for more prosaic researchers) because by the time of this handoff we will have already spent many years being radically (e.g., >10x) accelerated by AI systems, but I've updated against this due to updating toward shorter timelines, faster takeoff, and lower effort on safety (indicating less pausing). I also am just more into end-to-end planning with more of the later details figured out.

As I previously discussed here [LW(p) · GW(p)], I still think that you can do this with Paul-corrigible AGIs, including AGIs that you wouldn't be happy trusting with building a utopia (at least building a utopia without letting you reflect and then consulting you about this). However, these AIs will be nearly totally uncontrolled in that you'll be deferring to them nearly entirely. But, the aim is that they'll ultimately give you the power they acquire back etc. I also think pretty straightforward training strategies could very plausibly work (at least if you develop countermeasures to relatively specific threat models), like studying and depending on generalization.

(I've also updated some toward various of the arguments in this post about vulnerability and first mover advantage, but I still disagree with the bottom line overall..)

If pretraining from human preferences works, why hasn’t there been follow up work?

Also, why can’t this be combined with the deepseek paradigm?

https://arxiv.org/abs/2302.08582

I remember right when the negative results started hitting. I could feel the cope rising. I recognized the pattern, the straining against truth. I queried myself for what I found most painful - it was actually just losing a bet. I forced the words out of my mouth: "I guess I was wrong to be excited about this particular research direction. And Ryan was more right than I was about this matter."  

After that, it was all easier. What was there to be afraid of? I'd already admitted it!