No offense, but I feel you're not engaging with my argument here. Like if I were to respond to your comment I would just write the arguments from the above post again.

Implications of a Brain Scan Revolution

Suppose we were able to gather large amounts of brain-scans, lets say w/ millions of wearable helmets w/ video and audio as well,^[1] then what could we do with that? I'm assuming a similar pre-training stage where models are used to predict next brain-states (possibly also video and audio), and then can be finetuned or prompted for specific purposes.

Jhana helmets

Jhana is a non-addicting high pleasure state. If we can scan people entering this state, we might drastically reduce the time it takes to learn to enter this state. I predict (maybe just hope) that being able to easily enter this state will lead to an increase in psychological well-being and a reduction in power-seeking behavior. 

Jhana is great but is only temporary. It is a great foundation to then investigate the causes of suffering and remove it.

End Suffering

I currently believe the cause of suffering is an unnecessary mental motion which should show up in high fidelity brain scans. Again, eliminating suffering would go a long way to make decision-makers more selfless (they can still make poor decisions, just likely not out of selfish intent, unless it's habitual maybe? idk)

Additionally, this tech would also enable seeing the brain state of enlightened vs not, which might look like Steve Byrne's model here. This might not be a desirable state to be in, and understanding the structure of it could give insight. 

Lie Detectors

If we can reliably detect lying, this could be used for better coordination between labs & governments, as well as providing trust in employees and government staff (that they won't e.g. steal algorithmic/state secrets).

It could also be used to maintain power in authoritarian regimes (e.g. repeatedly test for faithfulness of those in power under you), and I agree w/ this comment that it could destroy relationships that would've otherwise been fine. 

Additionally this is useful for...

Super Therapy

Look! PTSD is just your brain repeating this same pattern in response to this trigger. CBT-related, negative thought spirals could be detected and brought up immediately. 

Beyond helping w/ existing brands of therapy, I expect larger amounts of therapeutic progress could be made in short amounts of time, which would also be useful for coordination between labs/governments.

Super Stimuli & Hacking Brains

You could also produce extreme levels of stimulation of images/audio that most saliently capture attention and lead to continued engagement. 

I'm unsure if there exist mind-state independent jail-breaks (e.g. a set of pixels that hacks a person's brains); however, I could see different mind states (e.g. just waking up, sleep/food/water deprivation) that, when combined w/ images/sounds could lead to crazy behavior.

Super Pursuasion

If you could simulate someone's brain state given different inputs, you could try to figure out the different arguments (or the surrounding setting or their current mood) that would be required to get them to do what you want. Using this for manipulating others is clear, but also could be used for better coordinating.

A more ethical way to use a simulation for pursuasion is to not decieve/leave-out-important truths. You could even run many simulations to figure out what those important factors that they find important.

However, simulating human brains for optimizing is also an ethical concern in general due to simulated suffering, especially if you do figure out what specific mental motion suffering is. 

Reverse Engineering Social Instincts

Part of Steven Byrne's agenda. This should be pretty easy to reverse engineer given this tech (epistemic note: I know nothing in this field) since you could find what the initial reward components are and how specific image/audio/thoughts led to our values. 

Reverse Engineering the secret sauce of human learning

With the good of learning human values comes the bad of figuring out how humans are so sample efficient!

Uploads

Simulating a full human brain is already an upload; however, I don't expect there to be optimized hardware ready for this. Additionally, running a human brain would be more complicated than just running normal NNs but w/ algorithmic insights from our brains. So this is still possible, but would have a large alignment tax. 

[I believe I first heard this argument from Steve Byrnes]

Everything Else in Medical Literature

Beyond trying to understand how the brain works (which is covered already), likely studying diseases and analyzing the brain's reaction to drugs. 

...

These are clearly dual-use; however, I do believe some of these are potentially pivotal acts (e.g. uploads, solving value alignment, sufficient coordination involving human actors), but would definitely require a lot of trust in the pursuers of this technology (e.g. existing power-seekers in power, potentially unaligned TAI, etc).

Hopefully things like jhana (if the benefits pan out) don't require a high fidelity helmet. The current SOTA is (AFAIK) 50% of people learning how to get into jhana given 1 week of a silent retreat, which is quite good for pedagogy alone! But maybe not sufficient to get e.g. all relevant decision-makers on board. Additionally, I think this percentage is only for getting into it once, as opposed to being able to enter that state at well. 

1. ^{^}

   Why would many people give up their privacy in this way? Well, when the technology actually becomes feasable, I predict most people will be out of jobs, so this would equivalent to a new job. 

If this risk is in the ballpark of a 5% chance in the next couple of years, then it seems to me entirely dominated by AI doom.

I think Abram is saying the following:

• Currently, AIs are lacking capabilities that would meaningfully speed up AI Safety research.
• At some point, they are gonna get those capabilities.
• However, by default, they are gonna get those AI Safety-helpful capabilities roughly at the same time as other, dangerous capabilities (or at least, not meaningfully earlier).
  • In which case, we're not going to have much time to use the AI Safety-helpful capabilities to speed up AI Safety research sufficiently for us to be ready for those dangerous capabilities.
• Therefore, it makes sense to speed up the development of AIS-helpful capabilities now. Even if it means that the AIs will acquire dangerous capabilities sooner, it gives us more time to use AI Safety-helpful capabilities to prepare for dangerous capabilities.

I'm very much in agreement that this is a problem, and among other things blocks us from knowing how to use adversarial attack methods (and AISI teams!) from helping here. Your proposed definition feels like it might be an important part of the story but not the full story, though, since it's output only: I would unfortunately expect a decent probability of strong jailbreaks that (1) don't count as intent misalignment but (2) jump you into that kind of red attractor basin. Certainly ending up in that kind of basin could cause a catastrophe, and I would like to avoid it, but I think there is a meaningful notion of "the AI is unlikely to end up in that basin of its own accord, under nonadversarial distributions of inputs".

Have you seen good attempts at input-side definitions along those lines? Perhaps an ideal story here would be a combination of an input-side definition and the kind of output-side definition you're pointing at.

Also, the solution is obviously to friendship is optimal the system that humans and AI coordinate in. Create an opt-in secure system that allows more resources if you cooperate and you will be able to outperform those silly defectors.

When it comes to solutions I think that humans versus AI axis doesn't make sense for the systems that we're in, it is rather about desirable system properties such as participation, exploration and caring for the participants in the system. 

If we can foster a democratic, caring, open-ended decision making process where humans and AI can converge towards optimal solutions then I think our work is done. 

Human disempowerment is okay as long as it is replaced by a better and smarter system so whilst I think the solutions are pointing in the right direction, the main axis of validation should rather be around system properties and not power distribution. 

Good summary though, it is great that we finally have a great paper to point towards for these problems.

Well, the government could take all your property away at a whim; millions of people have experienced that. Plus there are things like eminent domain.

Also, the government can tax your property (and punish you if you don't pay e.g. by taking away that thing), which is kinda weird philosophically... on one hand, something already belongs to you, on the other hand, you need to keep paying to keep it so -- but this is how renting is supposed to work, not owning.

A more obvious example is the intellectual property. There are no Schelling points; twenty years of copyright versus hundred years, there is no number that seems obviously correct.

And if after Singularity, the superintelligent AIs decide that they won't respect human property claims (but may respect each other's property), then... that's just the new reality. Just like a human ignores how the ants in his garden have distributed the territory among themselves.

So the fact that you have a property at all is downstream from "there is a consensus among those with power that this definition of 'property' is generally a good thing to have".

There is the way these things are generally done in the Western civilization, but if you asked some people living in tribes, they probably would have wildly different intuitions about what a person can own.

I can only own something if I am strong enough to defend it, or if there is a consensus among the others that they should let me keep it.

I interpreted it as: not by "usual means", but rather something like suicide or murder.

Logic simply preserves truth. You can arrive to a valid conclusion that one should act altruistically if you start from some specific premises, and can't if you start from some other presimes.

What are the premises you start from?