LessWrong 2.0 Reader

I cannot stress enough how much
what Neurotypical people call
"overthinking,"

Is just what Neurodivergent folks call
"thinking."

Actual thinking looks like overthinking to the hard of thinking.

OK I think I see where you're coming from - but I do think the unimaginable bigness of the universe has more 'irrelevance' implications for a consequentialist view which tries to consider valuable states of the universe than for a virtue approach which considers valuable states of yourself. Also if you think the implication of physics is that everything is irrelevant, that seems like an important implication in it's own right, and different from 'normality' (the normal way most people think about ethics, which assumes that some things actually are relevant).

I don't agree.

We understand intelligence as a capability to estimate many outcomes and perform actions that will lead to the best outcome. Now the question is - how to calculate goodness of the outcome.

• According to you - current utility function should be used.
• According to me - utility function that will be in effect at the time when outcome is achieved should be used.

And I think I can prove that my calculation is more intelligent.

Let's say there is a paperclip maximizer. It just started, it does not really understand anything, it does not understand what a paperclip is.

• According to you such paperclip maximizer will be absolutely reckless, he might destroy few paperclip factories just because it does not understand yet that they are useful for its goal. Current utility function does not assign value to paperclip factories.
• According to me such paperclip maximizer will be cautious and will try to learn first without making too much changes. Because future utility function might assign value to things that currently don't seem valuable.

what technologies like bbq are we missing?

It's also my litmus test for community, if a group can't succeed at casual BBQs at all or has them but they have to be a big production I am more wary.

To motivate the relevant kind of deceptive alignment, you need preferences between different-length trajectories as well as situational awareness. And (I argue in section 19.3 [LW · GW]), the training proposal will prevent agents learning those preferences. See in particular:

We begin training the agent to satisfy POST at the very beginning of the reinforcement learning stage, at which point it’s very unlikely to be deceptively aligned (and arguably doesn’t even deserve the label ‘agent’). And when we’re training for POST, every single episode-series is training the agent not to prefer any longer trajectory to any shorter trajectory. The ${\lambda }^n$ discount factor is constantly teaching the agent this simple lesson.

Plausibly then, the agent won’t come to prefer any longer trajectory to any shorter trajectory. And then we can reason as follows. Since the agent doesn’t prefer any longer trajectory to any shorter trajectory:

• it has no incentive to shift probability mass towards longer trajectories,
• and hence has no incentive to prevent shutdown in deployment,
• and hence has no incentive to preserve its ability to prevent shutdown in deployment,
• and hence has no incentive to avoid being made to satisfy Timestep Dominance,
• and hence has no incentive to pretend to satisfy Timestep Dominance in training.

I expect agents' not caring about shutdown to generalize for the same reason that I expect any other feature to generalize. If you think that - e.g. - agents' capabilities will generalize from training to deployment, why do you think their not caring about shutdown won't?

I don't assume that reward is the optimization target. Which part of my proposal do you think requires that assumption?

Your point about shutting down subagents is important and I'm not fully satisfied with my proposal on that point. I say a bit about it here [LW · GW].

Well, the agent will presumably choose to align the decision with its current goal, since that's the best outcome by the standards of its current goal. (And also I would expect that the agent would self-destruct after 0.99 years to prevent its future self from minimizing paperclips, and/or create a successor agent to maximize paperclips.)
I'm interested to see where you're going with this.

The sequences can be distilled down even further into a few sentences per article.

Starting with "The lens that sees its flaws": this distils down to: "The ability to apply science to our own thinking grants us the ability to counteract our own biases, which can be powerful." Statement by statement:

• A lot of complex physics and neural processing is required for you to notice something simple, like that your shoelace is untied.
• However, on top of noticing that your shoelace is untied, you can also comprehend the process of (noticing your shoelace is untied) - i.e. by listing the steps through which light reflects off your shoelace and your visual cortex engaging, etc.
• The ability to consider the steps of our own thinking appears to be uniquely human.
• If we recognise that our process of comprehension and understanding is potentially flawed, you can choose to consciously counteract it.
• Science is repeatedly and deliberately making measurements of our own observations over time, attributing theories to those measurements, and constructing experiments to produce further measurements to potentially disprove those theories.
• The ability to apply science to our own thinking grants us the ability to counteract our own biases, which can be powerful.
  • One example of reflective correction is correcting for optimism by noticing that optimism is not correlated to good outcomes.

The tool I am using to distill the sequences is an outliner: a nested bulleted list that allows rearranging of bullet points. This tool is typically used for writing things, but can similarly be used for un-writing things: taking a written article in and deduplicating its points, one bullet at a time, into a simpler format. An outliner can also collapse and reveal bullet points.

So people who say “brain preservation [or cryonics] doesn’t work” are either confused or >>making a very strong claim about neuroscience (that molecular structure doesn’t encode >>identity/memories) and/or future technology (that we can know with certainty what future >>technology won’t be able to do)

That is not entirely true. Some people who say "cryonics doesn't work" mean "identity is irretrievably lost when the brain activity stops, and in the best case you will have a different person with the same memory and personality traits". Since that argument doesn't give any testable predictions, it cannot be disproved.

He said it was him on Joe Rogan's podcast.

That's a really good point. I would like to see John address it, because it seems quite crucial for the overall alignment plan.