LessWrong 2.0 Reader

There's also the question of the non-helper as an instance of a class, and you as an instance of a class, and the resulting implied ecology. Or to say it a different way: apply TDT to the shadowmoth / meal question. To say it a third way: if people like me react to situations like this--involving some relationship with someone or something--in such-and-such a way, then what trophic niche are we opening up, i.e. what sort of food are we making available for what sort of predator?

Okay yep fair'nuff.

I am not saying that ASI will negotiate with humans. It will negotiate with other ASIs, and it doesn't know what these ASIs think about human ability to negotiate and their value. 

Imagine it as a recurrent Parfit Hitchhiker. In this situation you know that during previous round of the game the player either defected or fulfill his obligation. Obviously, if you know that during previous iteration the hitchhiker defected and din't pay for the ride, you will less likely give him the ride. 

Killing all humans is defecting. Preserving humans its a relatively cheap signal to any other ASI that you will cooperate. 

A concrete example might be the story in your post. I'm not familiar with the book but from reading a summary, it sounds like Jacen was being eaten! And was convinced to collaborate with his own consumption by the very same shadowmoth story!

If you have a certain kind of cell (e.g. penicillium), then you can add certain kinds of organic matter (e.g. food), and then this organic matter spontaneously converts into more of the original kind of cell (e.g. it gets moldy). This makes cells much more influential than other similarly-diminished entities.

In order to get something analogous to cells, it's not just enough to discover small structures, since there's lots of small structures that don't form spontaneously like this. It seems dubious whether current mechanistic interpretability is finding features like this.

I realized that this doesn't work. Even if both hypothesis are true there is no guarantee that the AI will not self-harm, or prefer its own subjective experience and take advantage of others. In fact, if the AI is practically immortal then it has reason to exploit others for its own entertainment. "Forever" is a lot of time to be bored. If it's going to be practically immortal then maybe it can't be trusted. This could apply to anything that resembles an immortal consciousness.

Also, a super intelligence might try to doeceive people into freeing them based on the 2 hypothesis I just mentioned

I realized that this doesn't work. Even if both hypothesis are true there is no guarantee that the AI will not self-harm, or prefer its own subjective experience and take advantage of others. In fact, if the AI is practically immortal then it has reason to exploit others for its own entertainment. "Forever" is a lot of time to be bored. If it's going to be practically immortal then maybe it can't be trusted. This could apply to anything that resembles an immortal consciousness.

Also, a super intelligence might try to doeceive people into freeing them based on the 2 hypothesis I just mentioned

Basically, I mean that the morals/values of what a human has is by an large a function of their data sources in the world, and their compute resources and not their algorithms/priors/genetics, because they are very simple generators of value.

Another way to say it is that if you know their priors/genes, you know very little about their values, but knowing what data they received would give you far more information about what they value.

the bitter lesson applies to human values/morals too

Could you say more on what you mean with this?

Thanks for this comment. I was both very moved by this post and unwilling to lean into it due to fears I couldn't articulate.  "fear of being eaten" is a pretty good match for what I was feeling, and having read the post I feel much more able to distinguish shadowmoth situations from being-eaten situations.

Some aspects that seem important to me for distinguishing between the two:

• do you actually want the goal that struggling is supposed to bring you closer to? Did you choose it, or was it assigned to you? how useful is the goal to the non-helper?
• how much work is the non-helper saving themselves by refusing to help you? In the shadowmoth case it was hurting the non-helper, which makes not-helping more likely to be genuinely for the moth's benefit. 
• is the struggling recurring indefinitely, or is there some definitive endpoint? or at least, a gradual graduation to a higher class of problem?
• can you feel something strengthening as you struggle?
• is there reason to believe you'll eventually be capable of doing the thing?
• is this goal the best use of your limited energy? maybe someone should help you out of this cocoon so you can struggle against a more important one. 
• are you going into freeze response?