I would observe that any HEC computronium planet could be destroyed and replaced with a similar amount of computronium running more efficient non-HEC computations, supporting a much greater amount of flourishing and well-being. So the real question is, why suffer a huge utility hit to preserve a blackbox, which at its best is still much worse than your best, and at its worst is possibly truly astronomically dreadful?

There's a game-theoretic component here as well: the choice to hide both encryption/decryption keys is not a neutral one. Any such civilization could choose to preserve at least limited access, and could also possibly provide verifiable proofs of what is going on inside (/gestures vaguely towards witness/functional encryption, PCP, and similar concepts). Since this is possible to some degree, choosing not to do so conveys information.

So, this suggests to me an unraveling argument: any such civilization which thinks its existence is ethically acceptable to all other civs will provide such proofs; any blackbox civ is then inferred to be one of the rest, with low average acceptability and so may be destroyed/replaced, so the civs which are ethically acceptable to almost all other civs will be better off providing the proof too; now the average blackbox civ is going to be even worse, so now the next most acceptable civ will want to be transparent and provide proof... And so on down to the point of civs so universally abhorrent that they are better off taking their chances as a blackbox rather than provide proof they should be burnt with fire. So you would then have good reason to expect any blackbox HEC civs you encounter to probably be one of the truly abominable ones.

As I understand the current FHE schemes, the computer evaluating the encrypted result must know both the circuit and the public key to continue evaluating the computation, which allows some intentional analysis and modification. There's even a proof that no completely obfuscated computation can exist. https://en.wikipedia.org/wiki/Black-box_obfuscation

That leaves the AGI with a fairly simple solution: produce a (much) larger circuit that contains both the original circuit and its values along with an encrypted copy of the AGI which will allow the copy to examine the plaintext values under the FHE and modify the original circuit's outcome as desired. To be safe, the new circuit can be made arbitrarily large enough to allow e.g. nested recursion of a FHE-turing machine running an embedded FHE computation which could be similarly modified the next FHE-level down. The physical AGI could trust its FHE instantiations to make the same (compute and space bounded) decisions it would, although it might never be able to break the FHE cipher and observe the results.

I am not a physicist so I don't know if a similar operation could be done to entangle a working copy.of the AGI with an ongoing unknown quantum computation.

[comment quality may be reduced by my frustration with conflicting philosophy. please help me convert my irritation-induced phrasing into useful insight if possible, but you don't owe me agreement or read-through. citations and references to synonyms needed if my comment is to reach peer-review grade.]

or at least when you have no other choice genociding them to avoid sufficiently bad suffering from being instantiated.

genocide is always wrong without exception; there's no such thing as a nonviable moral patient species, because suffering is inefficient and would waste energy. you do not have the negotiative right to demand an agent who values generating pain or other forms of control-feedback error in themselves stop doing so. you can only coherently demand they not use more negentropy on this than their universal allotment, and they may not take others' soul-structure, instantiate it, and torture it. if necessary, negotiation between species could imaginably turn forceful in response to potential genocide, and this violent response is one of the core large scale signals that genocide violates agency, if agency is defined by [[discovering agents]]. the only way to define valuable physical trajectories is by respecting agency - as far as I can tell, only preference utilitarianism can be mechanized and evaluated algorithmically.

to be clear, I argue this in part to prevent the error in your reasoning from being cited unchallenged in support of groups who would "mercy kill" beings alive today; the information defining a self preserving agent must not be lost into entropy, and any attempt to reduce suffering by ending a life when that life would have continued to try to survive is fundamentally a violation that any safe ai system would try to prevent. of course sometimes a being has an injury or disease that results in damage accumulation that they try and fail to prevent, thereby turning the damage into suffering; if this is severe enough, sometimes the agent will choose to be destroyed quickly so as to not waste negentropy creating life-history-patterns-aka-experiences they do not want to become.

but you have no right to impose your hedonic utility function on another agent. claim: preference utilitarianism iterated through coprotection/mutual-aid-and-defence games is how we got morality in the first place.

in terms of homomorphic encryption - it depends on whether the species contains an unencrypted component defending the world from invaders, even if a weak one. if there's any form of defense, it's likely the species just wants to be left alone. therefore all that can be done morally is say hi and move on. of course the HEC is converting negentropy into structures written into the entropy of the universe, aka experiences; making them unrecognizable outside the HEC system cannot make them nonphysical. but by demonstrating agency of defense, the species also demonstrates a preference for the HEC continuing - and therefore it is a form of life that cannot reasonably be interrupted.

contrast to stars, which we are confident do not contain HECs, and which instead contain maximum dissatisfaction at all times - if ever an agent begins to form at all, the temperature is far too high for that agent to retain control of its allocated matter, and thus suffers maximum death amount immediately. in other words, my view is that positive emotions are all forms of intended-state satisfaction and their value depends on how much computation goes into recognizing and verifying the satisfaction. a star cannot satisfy any agent's values directly with its reaction, and thus we should put the stars out. in comparison a civilization living as an HEC is, worst case, relatively trivial negentropy waste.

[many citations needed in this comment, feel free to tear it apart. but oh boy do I have opinions and insufficient literature references about them!]

This thought experiment really gets to a core disagreement I have with this form of ethics. I can't really formulate a justification for my view, but I have a reaction that I'll call "cosmic libertarianism". It seems to me that the only logical way in which this HEC civilization can come to exist is as some sort of defense against an adversary, and that the civ is essentially turtling. (They might be defending against those who are quick to exterminate civilizations that don't meet some standards of pleasure/pain balance.)

It also seems to me that if civilizations or the beings within them have any fundamental rights, they should have the right to go about their business. (The only exception would be a state of war.) If we were able to communicate with the HEC aliens, then we could get their consent to do... whatever. But otherwise they should be left alone.

(one example of why you might care whether moral patients in black holes "have experiences" is if you can influence what will happen in a black hole — for example, imagine a rocket with moral patients on board is headed for a black hole, and before it gets there, you get to influence how much suffering will happen on board after the rocket passes the event horizon)

Random aside:

I hear a ton of explanations of black holes that focus on the extreme cases of "literal light can't literally escape", sometime saying words like "you would experience X before getting squashed into spaghetti". But the discussion always seems to be doing some kind of spherical cow abstraction over what actual happens to literal humans (or, at least, literal computronium running simulations of human) as they get near, and then cross over, the event horizon. 

Your ship probably isn't moving at literal lightspeed. According to this site a 10-solar-mass black hole has a 30 km radius (diameter?) event horizon. If you pass nearby at near-lightspeed with literal humans on board at distance of, say, 60 km are the humans already super F'd up? Are they basically fine as long as the ship doesn't literally veer into the event horizon? Once they veer into the event horizon do they immediately start to get distorted by gravity forces before hitting the singularity at the center? Do they get to experience anything at all? Does the near-lightspeed ship spiral around the singularity once it crosses the event horizon or just sorta plunge right in? I assume it's all basically over Very Very Fast at that point (my intuition is guessing less than a second and maybe orders of magnitude faster).

If you build a really robust robot designed specifically to survive black holes, with a really high clockspeed for observing/thinking, what's the most conscious experience it would even hypothetically make sense to consider them having in the moments before and after crossing the event horizon at near-but-not-literal-lightspeed, before you start getting into ways that black hole physics is deeply weird? 

Presumably the superintelligence has more confidence than I in it's estimate of the value distribution of that opaque computation, and has some experience in creating new moral patients on that kind of substrate, so can make the call whether to preserve or wipe-and-reuse the system. 

With my limited knowledge of this far-distant situation, I'd expect mostly it's better to terminate unknowns in order to create known-positives.  However, a whole lot depends on specifics, and the value the AI places on variety and striving or other multi-dimensional aspects, rather than a simple hedonistic metric.  

What if you had some computation that could be interpreted (e.g. decrypted with two different keys) as either a simulation full of happy people, or a simulation full of depressed people? I think an adequate theory of experience is able to look at the encrypted computation (or any computation) and decide directly if there is suffering happening there.

Also, what is the difference between normal computation and encrypted computation? I feel like looking at a process that you haven't programmed yourself is not really that different than looking at an encrypted version of that. In either case, we don't have a clue about what's going on. And if we have a theory that lets us figure it out, it should work on both a normal and an encrypted version.

Whether computer-simulated minds or people from other universes (or beyond the event horizon in this post) have subjective experiences is essentially the reference class problem, a category of observers that "I could be" in anthropic arguments: Whether the reference class should include them.

I have a major problem with this "observation selection" type of anthropic reasoning, which pretty much is all that ever gets discussed such as SSA, SIA and their variants. In my opinion, there isn't any valid reference class. Each person's perspective, e.g. who I am, when is now etc, is primitive. Not something to be explained by reasoning.  There is no logical explanation or deduction for it. The first-person is unique, and subjective experience is intrinsic to the first-person perspective only. 

We can all imagine thinking from other people's perspectives. Do you think it is ethically relevant to reason from the perspective of a simulated mind? If so, then you should consider them conscious. Otherwise they are not. But as perspectives are primitive, these types of questions can only be answered by stipulations. Not as a conclusion from some carefully conducted reasoning. Rationality cannot provide any answer here. 

Criticism of one of your links:

those can all be ruled out with a simple device: if any of these things were the case, could that causate onto whether such an intuition fires? for all of them, the answer is no: because they are immaterial claims, the fact of them being true or false cannot have causated my thoughts about them. therefore, these intuitions must be discarded when reasoning about them.

Causation, which cannot be observed, can never overrule data. The attempted comparison involves incompatible types. Causation is not evidence, but a type of interpretation.

You draw a distinction between "material" and "immaterial" claims, without explaining how that distinction is grounded in neutral evidence. Neutral evidence here could mean graphical data like "seeing a red line moving". Such data can become interpreted, as e.g. "the pressure is increasing", leading to predictions, like "the boiler is going to explode". Under this view, illusions are possible: our interpretation of the graphical data may be wrong, and there may not actually be any moving, red, line-shaped object there. The interpretation is necessarily under-determined.

For the convenience of the current iteration of physics, some people would prefer to begin in medias res, starting with the physical interpretation as the fundamental fact, and reasoning backwards to a sense impression. But this is not the order the evidence presents, even if it is the order that is most convenient for our world-model.

P.S. I like your lowercase style.

As a variation of your thought experiment, I've pondered: How do you morally evaluate a life of a human who lives with some mental suffering during a day, but thrives in vivid and blissful dreams during their sleep time? 
  In a hypothetical adversary case one may even have dreams formed by their desires and the desires be made stronger by the daytime suffering. Intuitively it seems dissociative disorders might arise with a mechanism like this.

Very interesting, carado! I'll have to mull this one over.