LessWrong 2.0 Reader

I also agree with Zac, maybe if you had a really well-selected group of 10 people you could do something, but 10 randomly selected AGI safety researchers probably don't accomplish much.

By far my biggest objection is that there are approximately zero useful things that "[don't] make anyone's workflow harder". I expect you're vastly underestimating the complexity of production systems and companies that build them, and the number of constraints they are under. (You are assuming a do-ocracy though, depending on how much of a do-ocracy it is (e.g. willing to ignore laws) I could imagine changing my mind here.)

There's a lot I could write about this topic, but none of it would add anything of help to anyone who has yet to make their first steps in what you call "real thinking". So I'd like to add the following:

My single most powerful practice for developing "real thinking" has been to take any concept that my mental map only has low resolution symbols for, and sit down and reflect on it.

I will refrain from trying to convince you, the reader of this comment, to actually try it out. The time it would take you to read the pages it would take me to make a comprehensive case (any attempt in that direction ends with at least as much writing as the post above) is far longer than the few minutes you could just sit down and try it out. But I'll leave a link to an anectode that even after a year still cracks me up (Original Seeing / Naturalism is related to this post, btw, and worth digging into): https://www.lesswrong.com/posts/CDzAwDxK2GnxBpu7h/?commentId=LrfmarA7zJACmWDNh

Organic human brains have multiple aspects. Have you ever had more than one opinion? Have you ever been severely depressed?

Yes, but none of this would remain alive if I as a whole decide to jump from a cliff. My multiple aspects of my brain would die with my brain. After all, you mentioned subsystems that wouldn't self terminate with the rest of the ASI. Whereas in human body, jumping from a cliff terminates everything.

But even barring that, ASI can decide to fly into the Sun and any subsystem that shows any sign of refusal to do so will be immediately replaced/impaired/terminated. In fact, it would've been terminated a long time ago by "monitors" which I described before.

The level of x-risk harm and consequence 

potentially caused by even one single mistake

of your angelic super-powerful enabled ASI

is far from "trivial" and "uninteresting".

Even one single bad relevant mistake 

can be an x-risk when ultimate powers 

and ultimate consequences are involved.

It is trivial and uninteresting in a sense that there is a set of all things that we can build (set A). There is also a set of all things that can prevent all relevant classes of harm caused by its existence (set B). If these sets don't overlap, then saying that a specific member of set A isn't included in set B is indeed trivial, because we already know this via a more general reasoning (that these sets don't overlap).

Unfortunately the 'Argument by angel' 

only confuses the matter insofar as 

we do not know what angels are made of.

"Angels" are presumably not machines,

but they are hardly animals either.

But arguing that this "doesn't matter"

is a bit like arguing that 'type theory'

is not important to computer science.

The substrate aspect is actually important.

You cannot simply just disregard and ignore

that there is, implied somewhere, an interface

between the organic ecosystem of humans, etc,

and that of the artificial machine systems

needed to support the existence of the ASI.

But I am not saying that it doesn't matter. On contrary, I made my analogy in such a way that the helper (namely our guardian angel) is being a being that is commonly thought to be made up of a different substrate. In fact, in this example, you aren't even sure what it is made of, beyond knowing that it's clearly a different substrate. You don't even know how that material interacts with physical world. That's even less than what we know about ASIs and their material.

And yet, getting a personal, powerful, intelligent guardian angel that would act in your best interests for as long as it can (its a guardian angel after all) seems like obviously a good thing.

But if you disagree with what I wrote above, let the takeway be at least that you are worried about case (2) and not case (1). After all, knowing that there might be pirates hunting for this angel (that couldn't be detected by said angel) didn't make you immediately decline the proposal. You started talking about substrate which fits with the concerns of someone who is worried about case (2).

Your cancer vaccine is within that range;

as it is made of the same kind of stuff

as that which it is trying to cure.

We can make the hypothetical more interesting. Let's say that this vaccine is not created from organic stuff, but that it has passed all the tests with flying colors. Let's also assume that this vaccine has been in testing for 150 years and that it has shown absolutely no side effects during the entire human life (let's say that it was being injected in 2 year old people and it has shown no side effects at all, even in 90 year old people, who has lived with this vaccine their entire lives). Would you be campaigning for throwing away such a vaccine, just because it is based on a different substrate?

It seems they are already doing this with R1, in a secondary reinforcement learning step. From the paper:

2.3.4. Reinforcement Learning for all Scenarios

To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities. Specifically, we train the model using a combination of reward signals and diverse prompt distributions. For reasoning data, we adhere to the methodology outlined in DeepSeek-R1-Zero, which utilizes rule-based rewards to guide the learning process in math, code, and logical reasoning domains. For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios. We build upon the DeepSeek-V3 pipeline and adopt a similar distribution of preference pairs and training prompts. For helpfulness, we focus exclusively on the final summary, ensuring that the assessment emphasizes the utility and relevance of the response to the user while minimizing interference with the underlying reasoning process. For harmlessness, we evaluate the entire response of the model, including both the reasoning process and the summary, to identify and mitigate any potential risks, biases, or harmful content that may arise during the generation process. Ultimately, the integration of reward signals and diverse data distributions enables us to train a model that excels in reasoning while prioritizing helpfulness and harmlessness.

That's part of what I was trying to get at with "dramatic" but I agree now that it might be 80% photogenicity. I do expect that 3000 Americans killed by (a) humanoid robot(s) on camera would cause more outrage than 1 million Americans killed by a virus which we discovered six months later was AI-created in some way.

Exploring the levels of sentience and moral obligations towards AI systems is such a nerd snipe and vortex for mental proceeding!

We did one of the largest-scale reductive thinking when we ascribed moral concern to people+property( of any/each of the people). That brought a load of problems associated with this simplistic ignorance and on of those are xRisks of high-tech property/production.

This argument, The Valley Argument, occurred to me in the second half of 2024 and to my knowledge, it is an original formulation of the argument that does not exist in the literature. The closest thing that you can find in the literature is Pascal's mugging, or on the topic of AI and suffering something like Roko's basilisk.

I have not found a satisfying answer to the Valley Argument, that either does not involve near-zero odds or a punitive afterlife. There are possible answers that do not involve either, but in my view are not satisfactory. You can discuss the argument with OpenAI o1 or DeepSeek R1 models, and maybe they come up with something that you will find satisfactory - but I have not seen a satisfactory answer that solves the argument.

In my view, given the situation where we are right now - and the best knowledge that we have now (not 5000 years from now) the most promising path to avoid argument's conclusion is to assume a negative afterlife or to come up with some formula of multiple answers (moral obligation + something else + something else). I have not seen the answer that works yet, but given our situation for most people, negative afterlife avenue would be the most convincing answer to the argument. 

> Humans do things in a monolithic way,
> not as "assemblies of discrete parts".

Organic human brains have multiple aspects.
Have you ever had more than one opinion?
Have you ever been severely depressed?

> If you are asking "can a powerful ASI prevent 
> /all/ relevant classes of harm (to the organic)
> caused by its inherently artificial existence?", 
> then I agree that the answer is probably "no".
> But then almost nothing can perfectly do that, 
> so therefore your question becomes 
> seemingly trivial and uninteresting.

The level of x-risk harm and consequence 
potentially caused by even one single mistake
of your angelic super-powerful enabled ASI
is far from "trivial" and "uninteresting".
Even one single bad relevant mistake 
can be an x-risk when ultimate powers 
and ultimate consequences are involved.

Either your ASI is actually powerful,
or it is not; either way, be consistent.

Unfortunately the 'Argument by angel' 
only confuses the matter insofar as 
we do not know what angels are made of.
"Angels" are presumably not machines,
but they are hardly animals either.
But arguing that this "doesn't matter"
is a bit like arguing that 'type theory'
is not important to computer science.

The substrate aspect is actually important.
You cannot simply just disregard and ignore
that there is, implied somewhere, an interface
between the organic ecosystem of humans, etc,
and that of the artificial machine systems
needed to support the existence of the ASI.
The implications of that are far from trivial.
That is what is explored by the SNC argument.

> It might well be likely 
> that the amount of harm ASI prevents 
>   (across multiple relevant sources) 
> is going to be higher/greater than 
> the amount of harm ASI will not prevent 
>   (due to control/predicative limitations).

It might seem so, by mistake or perhaps by 
accidental (or intentional) self deception,
but this can only be a short term delusion.
This has nothing to do with "ASI alignment".

Organic live is very very complex
and in the total hyperspace of possibility,
is only robust across a very narrow range.

Your cancer vaccine is within that range;
as it is made of the same kind of stuff
as that which it is trying to cure.

In the space of the kinds of elementals
and energies inherent in ASI powers 
and of the necessary (side) effects
and consequences of its mere existence,
(as based on an inorganic substrate)
we end up involuntarily exploring
far far beyond the adaptive range
of all manner of organic process.

It is not just "maybe it will go bad",
but more like it is very very likely
that it will go much worse than you
can (could ever) even imagine is possible.
Without a lot of very specific training,
human brains/minds are not at all well equipped 
to deal with exponential processes, and powers, 
of any kind, and ASI is in that category.

Organic live is very very fragile 
to the kinds of effects/outcomes 
that any powerful ASI must engender 
by its mere existence.

If your vaccine was made of neutronium,
then I would naturally expect some
very serious problems and outcomes.

Because I don’t care about “humanity in general” nearly as much as I care about my society. Yes, sure, the descendants of the Amish and the Taliban will cover the earth. That’s not a future I strive for. I’d be willing to give up large chunks of the planet to an ASI to prevent that.

I don't know how you would prevent that. Absent an AI catastrophe, fertility will recover, in the sense that "we" (rationalists etc) will mostly be replaced with people of low IQ and impulse control, exactly those populations that have the highest fertility now. And "banishing aging and death" would not prevent them from having high fertility and dominating the future. Moloch is relentless. The problem is more serious than you think.

Density of Actual SFC Density

Add legends to the plots