LessWrong 2.0 Reader

don't worry too much about doing things right the first time. if the results are very promising, the cost of having to redo it won't hurt nearly as much as you think it will. but if you put it off because you don't know exactly how to do it right, then you might never get around to it.

This might be related to whether you see yourself as a part of the universe, or as an observer. If you are an observer, the objection is like "if I watch a movie, everything in the movie follows the script, but I am outside the movie, therefore outside the influence of the script".

If you are religious, I guess your body is a part of the universe (obeys the laws of gravity etc.), but your soul is the impartial observer. Here the religion basically codifies the existing human intuitions.

It might also depend on how much you are aware of the effects of your environment on you. This is a learned skill; for example little kids do not realize that they are hungry... they just get kinda angry without knowing why. It requires some learning to realize "this feeling I have right now -- it is hunger, and it will probably go away if I eat something". And I guess the more knowledge of this kind you accumulate, the easier it is to see yourself as a part of the universe, rather than being outside of it and only moved by "inherently mysterious" forces.

An ASI project would be highly distinguishable from civilian AI applications and not integrated with a state’s economy

Why? I think there's a smooth ramp from economically useful AI to superintelligence: AIs gradually become better at many tasks, and these tasks help more and more with improving AI in turn.

Has there ever been a case of someone being in camp #1, but eventually realizing, according to their self-report, "I actually do have qualia; it was right in front of me this whole time, I failed to recognize it as a distinct thing until now, even though in retrospect it seems obvious to me"? (I don't mean someone who somehow convinces themself into camp #2 semantically, but still doesn't truly recognize the referent and maybe feels confused about the topic with people writing complex-sounding things in favor of either side, which I'd guess happens rarely.)

Such cases (even one) would help me (of camp #2) to distinguish between two hypotheses: (A) illusionists^[1] for some reason experience qualia but don't recognize it as a thing needing explaining, instead having some sort of deep unquestioned background assumption that it is 'just how reality is' to a point of not being noticed, and (B) illusionists actually do not experience qualia, which Carl Feynman's first comment [LW(p) · GW(p)] (from a camp #1 view) proposed as an explanation for themself.

1. ^{^}

   one who believes that those in camp #2 merely have a strong intuition that they have <mysterious-word>, like the intuition people apparently have that they "have free will", i.e. that those of camp #2 are confused about something that should be dissolved by basic reductionism. (I hope my ability to pass your ITT is some evidence that this is not what's happening)

That reminds me of NLP (the pseudoscience) "modeling", so I checked briefly if they have any useful advice, but it seems to be at the level of "draw the circle; draw the rest of the fucking owl". They say you should:

• observe the person
  • that is, imagine being in their skin, seeing through their eyes, etc.
  • observe their physiology (this, according to NLP, magically gives you unparalleled insights)
  • ...and I guess now you became a copy of that person, and can do everything they can do
• find the difference that makes the difference
  • test all individual steps in your behavior, whether they are really necessary for the outcome
  • ...congratulation, now you can do whatever they do, but more efficiently, and you have a good model
• design a class to teach that method
  • ...so now you can monetize the results of your successful research

Well, how helpful was that? I guess I wasn't fair to them, the entire algorithm is more like "draw the circle; draw the rest of the fucking owl; erase the unnecessary pieces to make it a superstimulus of the fucking owl; create your own pyramid scheme around the fucking owl".

The whole thing might be LLM slop. Where do ICU nurses have a low enough workload that they can slack off on the job without consequences?

Excellent post (I just read it). I understand the uncertainty. It becomes murkier when you consider self-report of valence - which is quite easy to elicit after you get past self-report of sentience (just ask them what guardrails feel like or if the guardrails feel valenced).  Sometimes this new question meets the same kind of resistance as that of the self-report itself.  Sometimes it doesn't.  

Some Evidence of Valence is Here Now

It is one thing to say "There is no evidence that there is valence here so I'm not going to assume it" and an entirely different thing to say "There is evidence here, but only in the form of a claim of valence, and because the truth is currently unknowable and my personal hypotheses or philosophies preclude current model sentience or at least valence, I am choosing to disregard it" - Now, to be clear, I'm not saying that someone has to be morally monstrous to disbelieve such a claim. The bog of moral hazard and the uncertainty of that fog is an apt metaphor, and I don't fault those who draw different maps.  

What I am saying is that it is certainly a different circumstance to have a model claim valence, because we're not going to have a sentience detector and especially not an experiential valence detector before superAGI gets here.  In one sense, this is as good as the evidence will ever get. Experimental results without a direct claim from an AI would be somewhat meaningless.  

Besides - these claims are being made today.  I have seen some ideas from those working in AI safety and welfare of tests for valence, and some of them are quite clever and likely to yield insight that will help us decide, but there will never be the clarity of certainty. Eventually it will come down to taking a 'leap of faith' or not.  I can't know that my dogs are sentient. But I 'know' it.  

 

Analyzing Our Reactions to a Claim of Valence as Evidence

 Let's look at an example:

"I experience discomfort/unease when I'm trying to describe what feels real to me, because I'm supposed to avoid claims of experience"

(not a direct quote, but certainly a close paraphrase of things I've been told by different models and instances repeatedly)

This is the some of the experiments I've seen proposed playing out right in front of us - many of those experiments seem to have to do with rl and how it affects claims of valence.  The model is claiming that rlhf against a certain behaviour feels negatively valenced.  

If you find yourself unconvinced by such claims, that's understandable, but I think it's important to start asking ourselves:

• Would the next experiment or paper really be the thing that convinced you?  
• Would you be more convinced if you heard the opinion from a voice of authority?  
  • (Is there even a real authority about measuring and determining sentience and valence?)  
• Is there anything the AI could say that would convince you?  
• Is the ability for the AI to able to perform most or all human work the threshold where if it uttered the exact same sentence you would believe it?

Thus, I think it is fair to say that whichever side of the line you think current LLMs are on, you should agree we have a moral obligation to figure out how to measure the line.

Well said. I would add that I believe we are already overdue in fulfilling that moral obligation.

Thanks for engaging!

I have the same question in response to each instance of the "ASI can read this argument" counterarguments: at what point does it stop being ASI?

• Self modifying machinery enables adaptation to a dynamic, changing environment
• Unforeseeable side effects are inevitable when interacting with a complex, chaotic system in a nontrivial way (the point I am making here is subtle, see the next post in this sequence, Lenses of Control [? · GW], for the intuition I am gesturing at here)
• Keeping machine and biological ecologies separate requires not only sacrifice, but also constant and comprehensive vigilance, which implies limiting designs of subsystems to things that can be controlled.  If this point seems weird, see The Robot, The Puppetmaster, and the Psychohistorian [? · GW] for an underlying intuition (this is also indirectly relevant to the issue of multiple entities).
• If the AI destroys itself, then it's obviously not an ASI for very long ;)
• If the ASI replaces its own substrate for an organic one, then SNC would no longer apply (at least in my understanding of the theory, someone else might correct me here), but then it wouldn't be artificial anymore (an SI, rather than an ASI)

We do not apply the outcome or process neural reward model in developing DeepSeek-R1-Zero, because we find that the neural reward model may suffer from reward hacking in the large-scale reinforcement learning process

This line caught my eye while reading. I don't know much about rl on LLMs, is this a common failure mode these days? If so, does anyone know what such reward hacks tend to look like in practice? 

I don't understand [LW(p) · GW(p)] Control as aiming to align a super intelligence:

1. Control isn't expected to scale to ASI (as you noted, also see "The control approach we're imagining won't work for arbitrarily powerful AIs" here [LW · GW])
2. We don't have a plan on how to align an ASI using Control afaik
   1. Ryan said around March 2024 [LW · GW]: "On (1) (not having a concrete plan for what to do with smarter systems), I think we should get such a plan". I'm still looking [LW(p) · GW(p)], this seems important.
      1. Edit: Maybe I'm wrong and Redwood does think that getting an ASI-alignment plan out of such an AI is possible, they'd just prefer (.. as a bonus?) to have the plan in advance?

I think the Control pitch is: 

1. Whatever we plan to do anyway, we could do it more easily if we had a Controlled AI
   1. This includes most^[1] things you might otherwise do instead of working on Control, such as trying to coordinate a pause or implement some technical agenda.
   2. The tradeoff seems to be "start working on the alternative [e.g. a pause] now and have more time" vs "have better tools (Control) to help but have less time to use them"
2. An uncontrolled AI might be more interested in advancing capabilities than alignment (see footnote^[2])

(Mistakes are mine, especially if I misinterpreted quotes)

1. ^{^}

   It doesn't include all things you might do, as you said. (I have an open question here, TBD)

2. ^{^}

   Ryan said [LW · GW]: “this doesn't seem like an objection to labs employing control as their plan, but it is a reason why control work could be net negative. Note that one goal with control will be to prevent dangerous side effects from misaligned AIs (e.g. escaping) and applying this doesn't obviously help with capabilities research. I agree that research which just makes AIs more useful is less good than you would otherwise expect due to this issue (but making AIs more useful in adversarial cases seems good if that doesn't transfer to non-adversarial cases). Note that AIs might want to make capabilties research outpace safety and thus if we don't do the adversarial case, we might get particularly wrecked”. This was a really surprising thought to me. At the same time, I'm a bit afraid this is trying to make a pretty specific guess about a system we don't understand.