TsviBT's Shortform

I also kinda thought this. I actually thought it sounded sufficiently academic that I didn't realize at first it was your org, instead of some other thing you were supporting.

You might complain that the reason it doesn't solve stability is just that the thing doesn't have goal-pursuits.

Not so - I'd just call it the trivial case and implore us to do better literally at all!

Apart from that, thanks - I have a better sense of what you meant there. "Deep change" as in "no, actually, whatever you pointed to as the architecture of what's Really Going On... can't be that, not for certain, not forever."

Agreed we would have to talk more. I think I mostly get the homunculi objection. Don't have time now to write an actual response, so here are some signposts:
- part of what you call agency is explained by roughly active inference style of reasoning
-- some type of "living" system is characteristic by having boundaries between them and the environment (boundaries mostly in sense of separation of variables)
-- maintaining the boundary leads to need to model the environment
-- modelling the environment introduces a selection pressure toward approximating Bayes
- other critical ingredient is boundedness
-- in this universe, negentropy isn't free
-- this introduces fundamental tradeoff / selection pressure [LW · GW] for any cognitive system: length isn't free, bitflips aren't free, etc.
(--- downstream of that is compression everywhere, abstractions)
-- empirically, the cost/returns function for scaling cognition usually hits diminishing returns, leading to minds where it's not effective to grow the single mind further
--- this leads to the basin of convergent evolution [LW · GW] I call "specialize and trade"
-- empirically, for many cognitive systems, there is a general selection pressure toward modularity
--- I don't know what are all the reasons for that, but one relatively simple is 'wires are not free'; if wires are not free, you get colocation of computations like brain regions or industry hubs
--- other possibilities are selection pressures from CAP theorem, MVG, ...
(modularity also looks a bit like box-inverted [LW · GW] specialize and trade)

So, in short, I think where I agree with the spirit of If humans didn't have a fixed skull size, you wouldn't get civilization with specialized members and my response is there seems to be extremely general selection pressure in this direction. If cells were able to just grow in size and it was efficient, you wouldn't get multicellulars. If code bases were able to just grow in size and it was efficient, I wouldn't get a myriad of packages on my laptop, it would all be just kernel. (But even if it was just kernel, it seems modularity would kick in and you still get the 'distinguishable parts' structure.)

It's just more homunculi.

It's a bit annoying to me that "it's just more homunculi" [? · GW] is both kind of powerful for reasoning about humans, but also evades understanding agentic things. I also find it tempting because it gives a cool theoretical foothold to work off, but I wonder whether the approach is hiding most of the complexity of understanding agency.

Human values are meta and open--part of the core argument of my OP (the bullet point about communing).

Unless the human, on reflection, doesn't want some specific subset of their current values to be open to change / has meta-level preferences to freeze some object-level values. Which I think is common. (Source: I have meta-preferences to freeze some of my object-level values at "eudaimonia", and I take specific deliberate actions to avoid or refuse value-drift on that.)

Not implausible, like they could for some reason specifically ask the AGI to do this, but IDK why they would.

Callousness. "We probably need to do something about the rest of humanity, probably shouldn't just wipe them all out, lemme draft some legislation, alright looks good, rubber-stamp it and let's move on". Tons of bureaucracies and people in power seem to act this way today, including decisions that impact the fates of millions.

Right, certainly they could. Who actually would? (Not rhetorical.)

I don't know that Genghis Khan or Stalin wouldn't have. Some clinical psychopaths or philosophical extremists (e. g., the human successionists) certainly would.

What do you think would actually happen if a random person were put in the corrigible AGI control seat?

Mm...

First, I think "corrigibility to a human" is underdefined [LW(p) · GW(p)]. A human is not, themselves, a coherent agent with a specific value/goal-slot to which an AI can be corrigible.

Like, is it corrigible to a human's momentary impulses? Or to the command the human would give if they thought for five minutes? For five days? Or perhaps to the command they'd give if the AI taught them more wisdom? But then which procedure should the AI choose for teaching them more wisdom? The outcome is likely path-dependent on that: on the choice between curriculum A and curriculum B. And if so, what procedure should the AI use to decide what curriculum to use? Or should the AI perhaps basically ignore the human in front of them, and simply interpret them as a rough pointer to CEV? Well, that assumes the conclusion, and isn't really "corrigibility" at all, is it?

The underlying issue here is that "a human's values" are themselves underdefined. They're derived in a continual, path-dependent fashion, by a unstable process [LW · GW] with lots of recursions and meta-level interference. There's no unique ground-true set of values which the AI should take care not to step onto. This leaves three possibilities:

1. The AI acts as a tool that does what the human knowingly instructs it to do, with the wisdom by-default outsourced to the human.
   • But then it is possible to use it unwisely. For example, if the human operator is smart enough to foresee issues with self-modification, they could ask the AI to watch out for that. They could also ask it to watch out for that whole general class of unwise-on-the-part-of-the-human decisions. But they can also fail to do so, or unwisely ignore a warning in a fit of emotion, or have some beliefs about how decisions Ought to be Done that they're unwilling to even discuss with the AI.
2. The AI never does anything, because it knows that any of its actions can step onto one of the innumerable potential endpoints of a human's self-reflection process.
   • But then it is useless.
3. The AI isn't corrigible at all, it just optimizes for some fixed utility function, if perhaps with an indirect pointer to it ("this human's happiness", "humanity's CEV", etc.).

(1) is the only possibility worth examining here, I think.

And what I expect to happen if an untrained, philosophically median human is put in control of a tool ASI, is some sort of catastrophe. They would have various cached thoughts about how the story ought to go, what the greater good is, who the villains are, how the society ought to be set up. These thoughts would be endorsed at the meta-level, and not open to debate. The human would not want to ask the ASI to examine those; if the ASI attempts to challenge them as part of some other request, the human would tell it to shut up.^[1]

In addition, the median human is not, really, a responsible person. If put in control of an ASI, they would not suddenly become appropriately responsible. It wouldn't by default occur to them to ask the ASI for making them more responsible, either, because that's itself a very responsible thing to do. The way it would actually go, they are going to be impulsive, emotional, callous, rash, unwise, cognitively lazy.

Some sort of stupid and callous outcome is likely to result. Maybe not specifically "self-modifying into a monster/zombie and trapping humanity in a dystopian prison", but something in that reference class of outcomes.

Not to mention if the human has some extant prejudices: racism or any other manner of "assigning different moral worth to different sapient beings based on arbitrary features". The stupid-callous-impulsive process would spit out some not-very-pleasant fate for the undesirables, and this would be reflectively endorsed on some level, so a genuine tool-like corrigible ASI^[2] wouldn't say a word of protest.

Maybe I am being overly cynical about this, that's definitely possible. Still, that's my current model.

1. ^{^}

   Source: I would not ask the ASI to search for arguments against eudaimonia-maximization, or ask it to check if there's something else that "I" "should" be pursuing instead, because I do not want to be argued out of that even if there's some coherent, true, and compelling sense in which it is not what "I" "actually" "want". If the ASI asks whether it should run that check as part of some other request, I would tell it to shut up.

   (Note that it's different from examining whether my idea of eudaimonia/human flourishing/the-thing-I-mean-when-I-say-human-flourishing is correct/good, or whether my fundamental assumptions about how the world works are correct, etc.)

2. ^{^}

   As opposed to a supposedly corrigible but secretly eudaimonic ASI which, in one's imagination, always happens to gently question the human's decisions when the human orders it to do something bad, and then happens to pick the specific avenues of questioning that make the human "realize" they wanted good things all along.

So it's certainly not a claim that could be verified empirically by looking at any individual humans because there aren't yet any millenarians or megaannumarians.

If its not a conclusion which could be disproven empirically, then I don’t know how you came to it.

(I wrote my quick take quickly and therefore very elliptically, and therefore it would require extra charity / work on the reader's part (like, more time spent asking "huh? this makes no sense? ok what could he have meant, which would make this statement true?").)

I mean, I did ask myself about counter-arguments you could have with my objection, and came to basically your response. That is, something approximating “well they just don’t have enough information, and if they had way way more information then they’d love each other again” which I don’t find satisfying.

Namely because I expect people in such situations get stuck in a negative-reinforcement cycle, where the things which used to be fun which the other did lose their novelty over time as they get repetitive, which leads to the predicted reward of those interactions overshooting the actual reward, which in a TD learning sense is just as good (bad) as a negative reinforcement event. I don’t see why this would be fixed with more knowledge, and it indeed does seem likely to be exacerbated with more knowledge as more things the other does become less novel & more boring, and worse, fundamental implications of their nature as a person, rather than unfortunate accidents they can change easily.

I also think intuitions in this area are likely misleading. It is definitely the case now that marginally more understanding of each other would help with coordination problems, since people love making up silly reasons to hate each other. I do also think this is anchoring too much on our current bandwidth limitations, and generalizing too far. Better coordination does not always imply more love.

Sounds to me like wishful thinking. You basically assume that in 1 000 000 years people will get bored of doing the wrong thing, and start doing the right thing. My perspective is that "good" is a narrow target in the possibility space, and if someone already keeps missing it now, if we expand their possibility space by making them a God-emperor, the chance of converging to that narrow target only decreases.

Basically, for your model to work, kindness would need to be the only attractor in the space of human (actually, post-human) psychology.

A simple example of how things could go wrong is for Genghis Khan to set up an AI to keep everyone else in horrible conditions forever, and then (on purpose, or accidentally) wirehead himself.

Another example is the God-emperor editing their own brain to remove all empathy, e.g. because they consider it a weakness at the moment. Once all empathy is uninstalled, there is no incentive to reinstall it.

EDIT: I see that Thane Ruthenis already made this argument, and didn't convince you.

Major sleep debt?

Probably either one of (or some combination of): (1) "g" is the next consonant after "c" in "cognitive"; (2) leakage from "g-factor"; (3) leakage from "general(ly good at thinking)"

Also,

Background music: -6 to -2 CQ points.

That might generalize to "minimizing sound good", in which case I'd suggest trying these earplugs.

Generalizing to sensory deprivation in general, an easy way to do that is to lay in bed with eyes closed and lights off (not to sleep). (I've found this helpful, but maybe it's a side effect of not being distracted by a computer)

Fine, but it still seems like a reason one could give for death being net good (which is your chief criterion for being a deathist).

I do think it's a weaker reason than the second one. The following argument in defence of it is mainly for fun:

I slightly have the feeling that it's like that decision theory problem where the devil offers you pieces of a poisoned apple one by one. First half, then a quarter, then an eighth, than a sixteenth... You'll be fine unless you eat the whole apple, in which case you'll be poisoned. Each time you're offered a piece it's rational to take it, but following that policy means you get poisoned.

The analogy is that I consider living for eternity to be scary, and you say, "well, you can stop any time". True, but it's always going to be rational for me to live for one more year, and that way lies eternity.