Posts

Likely AGI architectures? 2020-07-29T03:49:47.822Z · score: 3 (2 votes)
Most probable AGI scenarios? 2020-07-06T17:20:27.188Z · score: 3 (2 votes)
‘Maximum’ level of suffering? 2020-06-20T14:05:14.423Z · score: 7 (6 votes)
Likelihood of hyperexistential catastrophe from a bug? 2020-06-18T16:23:41.608Z · score: 11 (7 votes)
Pessimism over AGI/ASI causing psychological distress? 2020-06-02T18:28:19.466Z · score: 7 (4 votes)

Comments

Comment by anirandis on Open & Welcome Thread - July 2020 · 2020-07-25T14:37:14.748Z · score: 3 (2 votes) · LW · GW

Thanks for your response, just a few of my thoughts on your points:

If you *can* stop doing philosophy and futurism

To be honest, I've never really *wanted* to be involved with this. I only really made an account here *because* of my anxieties and wanted to try to talk myself through them.

If an atom-for-atom identical copy of you, *is* you, and an *almost* identical copy is *almost* you, then in a sufficiently large universe where all possible configurations of matter are realized, it makes more sense to think about the relative measure of different configurations rather than what happens to "you".

I don't buy that theory of personal-identity personally. It seems to me that if the biological me that's sitting here right now isn't *feeling* the pain, that's not worth worrying about as much. Like, I can *imagine* that a version of me might be getting tortured horribly or experiencing endless bliss, but my consciousness doesn't (as far as I can tell) "jump" over to those versions. Similarly, were *I* to get tortured it'd be unlikely that I care about what's happening to the "other" versions of me. The "continuity of consciousness" theory *seems* stronger to me, although admittedly it's not something I've put a lot of thought into. I wouldn't want to use a teleporter for the same reasons.

*And* there are evolutionary reasons for a creature like you to be *more* unable to imagine the scope of the great things.

Yes, I agree that it's possible that the future could be just as good as an infinite torture future would be bad. And that my intuitions are somewhat lopsided. But I do struggle to find that comforting. Were an infinite-torture future realised (whether it be a SignFlip error, an insane neuromorph, etc.) the fact that I could've ended up in a utopia wouldn't console me one bit.

Comment by anirandis on Open & Welcome Thread - July 2020 · 2020-07-25T02:46:46.236Z · score: 3 (2 votes) · LW · GW

As anyone could tell from my posting history, I've been obsessing & struggling psychologically recently when evaluating a few ideas surrounding AI (what if we make a sign error on the utility function, malevolent actors creating a sadistic AI, AI blackmail scenarios, etc.) It's predominantly selfishly worrying about things like s-risks happening to me, or AI going wrong so I have to live in a dystopia and can't commit suicide. I don't worry about human extinction (although I don't think that'd be a good outcome, either!)


I'm wondering if anyone's gone through similar anxieties and have found a way to help control them? I'm diagnosed ASD and I wouldn't consider it unlikely that I've got OCD or something similar on top of it, so it's possibly just that playing up.

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-07-23T14:36:58.410Z · score: 1 (1 votes) · LW · GW
Not really, because it takes time to train the cognitive skills necessary for deception.

Would that not be the case with *any* form of deceptive alignment, though? Surely it (deceptive alignment) wouldn't pose a risk at all if that were the case? Sorry in advance for my stupidity.

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-07-23T02:34:34.109Z · score: 1 (1 votes) · LW · GW

Sorry for the dumb question a month after the post, but I've just found out about deceptive alignment. Do you think it's plausible that a signflipped AGI could fake being an FAI in the training stage, just to take a treacherous turn at deployment?

Comment by anirandis on ‘Maximum’ level of suffering? · 2020-06-22T08:28:58.559Z · score: 1 (1 votes) · LW · GW

It’s more a selfish worry, tbh. I don’t buy that pleasure being unlimited can cancel it out though - even if I were promised a 99.9% chance of Heaven and 0.1% chance of Hell, I still wouldn’t want both pleasure and pain to be potentially boundless.

Comment by anirandis on ‘Maximum’ level of suffering? · 2020-06-21T13:07:28.434Z · score: 1 (1 votes) · LW · GW

I do agree that they’re symmetrical. I just find it worrying that I could potentially experience such enormous amounts of pain, even when the opposite is also a possibility.

Comment by anirandis on ‘Maximum’ level of suffering? · 2020-06-20T23:49:02.503Z · score: 1 (1 votes) · LW · GW
I'd still expect a reasonable utility function to *cap* the (dis)utility of pain. If it didn't, the (possible) torture of just one creature capable of experiencing arbitrary amounts/degrees/levels of pain would effectively be 'Pascal's hostage'

I suppose I never thought about that, but I'm not entirely sure how it'd work in practice. Since the AGI could never be 100% certain that the pain it's causing is at its maximum, it might further increase pain levels, just to *make sure* that it's hitting the maximum level of disutility.


It also seems unclear why evolution would result in creatures able to experience pain more intensely than such a maximum.

I think part of what worries me is that, even if we had a "maximum" amount of pain, it'd be hypothetically possible for humans to be re-wired to remove that maximum. I'd think that I'd still be the same person experiencing the same consciousness *after* being rewired, which is somewhat troubling.


If the pain a superintelligence can cause scales linearly or better with computational power, then the thought is even more terrifying.


Overall, you make some solid points that I wouldn't have considered otherwise.

Comment by anirandis on ‘Maximum’ level of suffering? · 2020-06-20T20:19:56.410Z · score: 1 (1 votes) · LW · GW

I think it's the modifying humans to experience pain part that's the most terrifying, to be honest.

Comment by anirandis on ‘Maximum’ level of suffering? · 2020-06-20T19:40:56.328Z · score: 1 (1 votes) · LW · GW
Real-life animals can and do die of shock, which seems *like* it might be some maximum 'pain' threshold being exceeded.

In theory, would it not be possible for, say, a malevolent superintelligence to "override" any possibility of a "shock" reaction, and prevent the brain from shutting down? Wouldn't that allow for ridiculous amounts of agony?


It seems plausible to me that a sufficiently powerful agent could create some form of ever-growing agony by expanding subjects' pain centres to maximise pain; and the limit being the point where most of the matter in the universe is part of someone's pain centre seems incredibly scary. I sincerely hope there's good reason to believe that a hypothetical "evil" superintelligence would get diminishing returns quite quickly.

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-19T21:27:03.975Z · score: 1 (1 votes) · LW · GW

I think it's also a case of us (or at least me) not yet being convinced that the probability is <= 10^-6. Especially with something as uncertain as this. My credence in such a scenario happening has, too, decreased a fair bit with this thread but I remain unconvinced overall.


And even then, 1 in a million isn't *that* unlikely - it's massive compared to the likelihood that a mugger is actually a God. I'm not entirely sure how low it would have to be for me to dismiss it as "Pascalian", but 1 in a million still feels far too high.

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-19T18:39:44.188Z · score: 1 (1 votes) · LW · GW

I think a probability of ~1/30,000 is still way too high for something as bad as this (with near-infinite negative utility). I sincerely hope that it’s much lower.

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-19T13:41:21.449Z · score: 1 (1 votes) · LW · GW

Everything about the AGI, loosely speaking, has to be near-perfect except for that one bit.

Isn’t this exactly what happened with the GPT-2 bug, which led to maximally ‘bad’ output? Would that not suggest that the probability of this occurring with an AGI is non-negligible?

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-18T22:02:20.635Z · score: 1 (1 votes) · LW · GW

All of these worry me as well. It simply doesn't console me enough to think that we "will probably notice it".

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-18T21:36:05.138Z · score: 1 (1 votes) · LW · GW

Can we be sure that we'd pick it up during the training process, though? And would it be possible for it to happen after the training process?

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-18T21:22:47.093Z · score: 1 (1 votes) · LW · GW

What do you think the difference would be between an AGI's reward function, and that of GPT-2 during the error it experienced?

Comment by anirandis on Likelihood of hyperexistential catastrophe from a bug? · 2020-06-18T20:00:23.910Z · score: 3 (1 votes) · LW · GW

Surely with a sufficiently hard take-off it would be possible for the AI to prevent its turning off? If not, couldn’t the AI just deceive its creators into thinking that no signflip has occurred (e.g. making it look like it’s gaining utility from doing something beneficial to human values when it’s actually losing it). How would we be able to determine that it’s happened before it’s too late?

Further to that, what if this fuck-up happens during an arms race when its creators haven’t put enough time into safety to prevent this type of thing from happening?

Comment by anirandis on Pessimism over AGI/ASI causing psychological distress? · 2020-06-04T22:14:46.342Z · score: 1 (1 votes) · LW · GW

The American government is shit, don't get me wrong, but in modern times it's much better than some others. The US government isn't currently ethnically cleansing and hasn't done for a while now. I'm a lot less worried about them doing anything than certain other governments.

Comment by anirandis on Pessimism over AGI/ASI causing psychological distress? · 2020-06-04T22:06:24.619Z · score: 1 (1 votes) · LW · GW

I suppose that my view of the Chinese and Russian governments are a little bit worse than that of the American one (imo those three are the most likely to win an AGI arms race so I don't consider others so much), but I think that's justified.


The American government isn't ethnically cleansing right now in the same way that the Chinese one is for Uighar Muslims. And to me, it doesn't seem vanishingly unlikely that such a government would want to maximise suffering for a particular group that it doesn't happen to be very fond of.


If the Nazis had access to this type of technology, how sure would you be that we wouldn't have certain groups being infinitely tortured as we speak?


That's not to mention the perhaps more likely near-miss scenarios in which we fuck up AI alignment in a way that we're unable to die nor live a meaningful life. Like, say, if we all ended up getting locked underground by Omega because of some black swan possibility that nobody thought of. It just seems to me that human extinction and dystopia are the most likely outcomes.