I think that, perhaps, after reading this [LW(p) · GW(p)] clarifying comment from you, @quila [LW · GW] would change his perception of your position.

People do not want more than one child. Some don’t even want one. Two children is viewed as a stretch goal. 

The first Google result I get for "ideal family size in america" is a 2023 Gallup poll according to which 44% of Americans who had an opinion think two children is the idea family size, and 45% that three or more children is.

My current impression is that most people would prefer to have more children than they actually end up having, but there are various societal forces in play making this hard, so it'll take some time to develop the necessary social technologies to better align people's family sizes with their preferences.

Clearly ANNs are able to represent propositional content, but I haven't seen anything that makes me think that's the natural unit of analysis.

Well, we (humans) categorize our epistemic state largely in propositional terms, e.g. in beliefs and suppositions. We even routinely communicate by uttering "statements" -- which express propositions. So propositions are natural to us, which is why they are important for ANN interpretability.

Let me put my attitudes in practical terms: I don't kick dogs, but I have destroyed a wasp's nest in my garage, and I don't donate to animal charities. (I don't donate to many other charities either, but there have been a few.) Let those who think saving the shrimps is worthwhile do so, but I do not travel on that path

This is what I expected. Your take when put in these terms seems pretty moderate. Whereas, when I read your original comment, this take (which presumably stayed the same) seemed very extreme.

In other words, my personal beliefs haven't changed a single bit and yet my perception of your beliefs changed a lot. I can only imagine that your original comment has been so strongly disagree-voted because of the framing.

Non-temporal displacement model

Maybe rename this to "Simultaneous displacement model" or  "Displacement model under simultaneous appearance"

I also agree with Zac, maybe if you had a really well-selected group of 10 people you could do something, but 10 randomly selected AGI safety researchers probably don't accomplish much.

By far my biggest objection is that there are approximately zero useful things that "[don't] make anyone's workflow harder". I expect you're vastly underestimating the complexity of production systems and companies that build them, and the number of constraints they are under. (You are assuming a do-ocracy though, depending on how much of a do-ocracy it is (e.g. willing to ignore laws) I could imagine changing my mind here.)

Organic human brains have multiple aspects. Have you ever had more than one opinion? Have you ever been severely depressed?

Yes, but none of this would remain alive if I as a whole decide to jump from a cliff. My multiple aspects of my brain would die with my brain. After all, you mentioned subsystems that wouldn't self terminate with the rest of the ASI. Whereas in human body, jumping from a cliff terminates everything.

But even barring that, ASI can decide to fly into the Sun and any subsystem that shows any sign of refusal to do so will be immediately replaced/impaired/terminated. In fact, it would've been terminated a long time ago by "monitors" which I described before.

The level of x-risk harm and consequence 

potentially caused by even one single mistake

of your angelic super-powerful enabled ASI

is far from "trivial" and "uninteresting".

Even one single bad relevant mistake 

can be an x-risk when ultimate powers 

and ultimate consequences are involved.

It is trivial and uninteresting in a sense that there is a set of all things that we can build (set A). There is also a set of all things that can prevent all relevant classes of harm caused by its existence (set B). If these sets don't overlap, then saying that a specific member of set A isn't included in set B is indeed trivial, because we already know this via a more general reasoning (that these sets don't overlap).

Unfortunately the 'Argument by angel' 

only confuses the matter insofar as 

we do not know what angels are made of.

"Angels" are presumably not machines,

but they are hardly animals either.

But arguing that this "doesn't matter"

is a bit like arguing that 'type theory'

is not important to computer science.

The substrate aspect is actually important.

You cannot simply just disregard and ignore

that there is, implied somewhere, an interface

between the organic ecosystem of humans, etc,

and that of the artificial machine systems

needed to support the existence of the ASI.

But I am not saying that it doesn't matter. On contrary, I made my analogy in such a way that the helper (namely our guardian angel) is a being that is commonly thought to be made up of a different substrate. In fact, in this example, you aren't even sure what it is made of, beyond knowing that it's clearly a different substrate. You don't even know how that material interacts with physical world. That's even less than what we know about ASIs and their material.

And yet, getting a personal, powerful, intelligent guardian angel that would act in your best interests for as long as it can (its a guardian angel after all) seems like obviously a good thing.

But if you disagree with what I wrote above, let the takeway be at least that you are worried about case (2) and not case (1). After all, knowing that there might be pirates hunting for this angel (that couldn't be detected by said angel) didn't make you immediately decline the proposal. You started talking about substrate which fits with the concerns of someone who is worried about case (2).

Your cancer vaccine is within that range;

as it is made of the same kind of stuff

as that which it is trying to cure.

We can make the hypothetical more interesting. Let's say that this vaccine is not created from organic stuff, but that it has passed all the tests with flying colors. Let's also assume that this vaccine has been in testing for 150 years and that it has shown absolutely no side effects during the entire human life (let's say that it was being injected in 2 year old people and it has shown no side effects at all, even in 90 year old people, who has lived with this vaccine their entire lives). Would you be campaigning for throwing away such a vaccine, just because it is based on a different substrate?

It seems they are already doing this with R1, in a secondary reinforcement learning step. From the paper:

2.3.4. Reinforcement Learning for all Scenarios

To further align the model with human preferences, we implement a secondary reinforcement learning stage aimed at improving the model’s helpfulness and harmlessness while simultaneously refining its reasoning capabilities. Specifically, we train the model using a combination of reward signals and diverse prompt distributions. For reasoning data, we adhere to the methodology outlined in DeepSeek-R1-Zero, which utilizes rule-based rewards to guide the learning process in math, code, and logical reasoning domains. For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios. We build upon the DeepSeek-V3 pipeline and adopt a similar distribution of preference pairs and training prompts. For helpfulness, we focus exclusively on the final summary, ensuring that the assessment emphasizes the utility and relevance of the response to the user while minimizing interference with the underlying reasoning process. For harmlessness, we evaluate the entire response of the model, including both the reasoning process and the summary, to identify and mitigate any potential risks, biases, or harmful content that may arise during the generation process. Ultimately, the integration of reward signals and diverse data distributions enables us to train a model that excels in reasoning while prioritizing helpfulness and harmlessness.

That's part of what I was trying to get at with "dramatic" but I agree now that it might be 80% photogenicity. I do expect that 3000 Americans killed by (a) humanoid robot(s) on camera would cause more outrage than 1 million Americans killed by a virus which we discovered six months later was AI-created in some way.

Exploring the levels of sentience and moral obligations towards AI systems is such a nerd snipe and vortex for mental proceeding!

We did one of the largest-scale reductive thinking when we ascribed moral concern to people+property( of any/each of the people). That brought a load of problems associated with this simplistic ignorance and on of those are xRisks of high-tech property/production.