What To Do: Environmentalism vs Friendly AI (John Baez)

Speaking as someone whose introduction to transhumanist ideas was the mind-altering idea shotgun titled Permutation City, I've been pretty disappointed with his take on AI and the existential risks crowd.

A reoccurring theme in Egan's fiction is that "all minds face the same fundamental computing bottlenecks", serving to establish the non-existence of large-scale intrinsic cognitive disparities. I always figured this was the sort of assumption that was introduced for the sake of telling a certain class of story - the kind that need only be plausible (e.g., "an asteroid is on course to hit us"), and didn't think much more about it.

But from what I recall of Egan's public comments on the issue of foom (I lack links, sorry) he appears to have a firm intuition that it's impossible, grounded by handwaving "halting problem is unsolvable"-style arguments. Which in turn seemingly forms the basis of his estimation of uFAI scenarios as "immensely unlikely". With no defense on offer for his initial "cognitive universality" assumption, he takes the only remaining course of argumentation...

but frankly, their attempt to prescribe what rational altruists should be doing with their time and money is just laughable

...derision.

This spring...

Egan, musing: Some people apparently find this irresistible

Greg Egan is...

Egan, screaming: The probabilities are approaching epsilon!!!

Above the Argument.

Egan, grimly: Yudkowsky is off the air.

Previous arguments by Egan:

http://metamagician3000.blogspot.com/2009/09/interview-with-greg-egan.html

Sept. 2009, from an interview in Aurealis.

http://metamagician3000.blogspot.com/2008/04/transhumanism-still-at-crossroads.html

From April 2008. Only in the last few comments does Egan actually express an argument for the key intuition that has been driving the entire rest of his reasoning.

(To my eyes, this intution of Egan's refers to a completely irrelevant hypothetical, in which humans somehow magically and reliably are always able to acquire possession of and make appropriate use of any insentient software tools that will be required, at any given moment, in order for humans to maintain hypothetical strategic parity with any contemporary AI's.)

Greg Egan's view was discussed here a few months ago.

I think Greg Egan makes an important point there that I have mentioned before and John Baez seems to agree:

I agree that multiplying a very large cost or benefit by a very small probability to calculate the expected utility of some action is a highly unstable way to make decisions.

Actually this was what I had in mind when I voiced my first attempt at criticizing the whole endeavour of friendly AI, I just didn't know what exactly was causing my uneasiness.

I am still confused about it but think that it isn't much of a problem as long as friendly AI research is not being funded at the cost of other risks that are more thoroughly based on empirical evidence rather than the observation of logically valid arguments.

To be clear, as I wrote in the post above, I think that there are very strong arguments in support of friendly AI research. I believe that it is currently the most important cause one could support, but I also think that there is a limit to what one should do in the name of mere logical implications. Therefore I partly agree with Greg Egan.

ETA

There's now another comment by Greg Egan:

All of Yudkowsky’s arguments about the dangers and benefits of AI are just appeals to intuition of various kinds, as indeed are the counter-arguments. So I wouldn’t hold your breath waiting for that to be settled. If he wants to live his own life based on his own hunches, that’s fine, but I see no reason for anyone else to take his land-grabs on terms like “rationality” and “altruism” at all seriously, merely because it’s not currently possible to provide mathematically rigorous proofs that his assignments of probabilities to various scenarios are incorrect. There’s an almost limitless supply of people who believe that their ideas are of Earth-shattering importance, and that it’s incumbent on the rest of the world to either follow them or spend their life proving them wrong.

But clearly you’re showing no signs of throwing in productive work to devote your life to “Friendly AI” — or of selling a kidney in order to fund other people’s research in that area — so I should probably just breathe a sigh and relief, shut up and go back to my day job, until I have enough free time myself to contribute something useful to the Azimuth Project, get involved in refugee support again, or do any of the other “Rare Disease for Cute Kitten” activities on which the fate of all sentient life in the universe conspicuously does not hinge.

Surely not ... Does Greg Egan understand how "a small chance every year" can build into "almost certain by this date"? Because that was convincing for me:

I can easily see humans building work-arounds or stop-gaps for most major problems, and continuing business mostly as usual. We run out of fossil fuels, so we get over our distrust of nuclear energy because it's the only way. We don't slow environmental damage enough, so agriculture suffers, so we get over our distrust of genetically modified plants because it's the only way. And so on.

Then some article somewhere reminded me that business as usual includes repeated attempts at artificial intelligence. And runaway AI is not something we can build a work-around for; given a long enough timespan and faith in human ingenuity, we'll push through all the other non-instant-game-over events until we finally succeed at making the game end instantly.

Yep, Egan created a Yudkowsky and Overcoming Bias/LessWrong stand-ins for mockery in his most recent novel, Zendegi. There was a Less Wrong discussion at the time.

Carl, you hit the biggest nail on the head. But I think there's another nail there. If not for XiXiDu, then for many others. Working on fooming AI issues makes one a weirdo. Wearing a tinfoil hat would only be slightly less embarrassing. Working on environmental problems is downright normal, at least within some (comfortably large) social circles.

Back to that biggest nail - it needs another whack. AI threatens to dramatically worsen the world within our children's lifetimes. Robin Hanson, sitting next to his son, will feel significantly less comfortable upon thinking such thoughts. That provides a powerful motive to rationalize the problem away - or to worry at it, I suppose, depending on one's personality, but I find denial to be more popular than worrywarting.

The idea of moral demandingness is really a separate issue.

I don't see what difference it makes if you are selfish or not regarding friendly AI research. I believe that being altruistic is largely instrumental in maximizing egoistic satisfactions.

Thanks for the post by Nick Bostrom. But it only adds to the general confusion.

There seem to be dramatic problems with both probability-utility calculations and moral decision making. Taking those problems into account makes it feel like one could as well throw a coin to decide.

Michael Anissimov recently wrote:

For instance, you must have made decisions for your children that were more in alignment with what they would want if they were smarter. If you made judgments in alignment with their actual preferences (like wanting to eat candy all day — I don’t know your kids but I know a lot of kids would do this), they would suffer for it in the longer term.

This sounds good but seems to lead to dramatic problems. In the end it is merely an appeal to intuition without any substance.

If you don't try to satisfy your actual preferences, what else?

In the example, stated by Anissimov, what actually happens is that parents try to satisfy their own preferences by not allowing their children to die of candy-intoxication.

If we were going to disregard our current preferences and postpone having fun in favor of gathering more knowledge then we would eventually end up as perfectly rational agents in static game theoretic equilibria.

The problem with the whole utility-maximization heuristic is that it eventually does deprive us of human nature by reducing our complex values to mere game theoretic models.

Part of human nature, what we value, is the way we like to decide. It won't work to just point at hyperbolic discounting and say it is time-inconsistent and therefore irrational.

Human preferences are always actual, we do not naturally divide decisions according to instrumental and terminal goals.

I don't want a paperclip maximizer to burn the cosmic commons. I don't want to devote most of my life to mitigate that risk. This is not a binary decision, that is not how human nature seems to work.

If you try to force people into a binary decision between their actual preferences and some idealistic far mode then you cause them to act according to academic considerations rather than their complex human values they are supposed protect.

Suppose you want to eat candies all day and are told that you can eat a lot more candies after the Singularity, if only you work hard enough right now. The problem is that there is always another Singularity that promises more candies. At what point are you going to actually eat candies? But that is a rather academic problem. There is a more important problem concerning human nature, as demonstrated by extreme sport. Humans care much more about living their life's according to their urges than about maximizing utility.

What does it even mean to "maximize utility"? Many sportsmen and sportswomen are aware of the risks associated with their favorite activity. Yet they take the risk.

It seems that humans are able to assign infinite utility to pursuing a certain near-mode activity.

Deliberately risking your life doesn't seem to be maximizing experience utility, as you would be able to experience a lot more of the same or similar experience differently. And how can one "maximize" terminal decision utility?

Finally, some quantification!

Here's a sequence of interpretations of this passage, in decreasing order of strength:

1. The odds of us being wiped out by badly done AI are easily larger than 10%
2. The odds of us being wiped out by badly done AI are larger than or equal to 10%
3. There can be no compelling qualitative argument that the probability of us being wiped out by badly done AI is less that 1%
4. There is a compelling argument that the probability of us being wiped out by badly done AI is greater than or equal to 1%

I would be very grateful to see the weakest of these claims, number 4, supported with some calculations.

Of course I wish that there was a date attached to these claims. Easily greater than 10% chance that we'll be wiped out in the next 50 years?

From E.Y on the same page, we have this:

I’m generally reluctant to assign exact probabilities to topics like these; I consider it a sin, like giving five many significant digits on something you cannot calculate to 1 part in 10,000 precision.

We should be assigning error bars to our probabilities? A cute idea - but surely life is too short for that.

There is a "discussion" unleashing between Eliezer Yudkowsky and Greg Egan, scroll down here. It yielded this highly interesting comment by Eliezer Yudkowsky:

I don’t think the odds of us being wiped out by badly done AI are small. I think they’re easily larger than 10%. [...]

I would take the other side of a bet on that at 1000:1 odds.

:-) <--

The theory here seems to be that if someone believes preserving the environment is the most important thing, you can explain to them why preserving the environment is not the most important thing, and they will stop believing that preserving the environment is the most important thing. But is there any precedent for this result actually happening?

This runs into strongly believed theoretical comp sci limits such as the likelyhood that P != NP.

Does it? There are certainly situations (breaking encryption) where the problem statement looks something like "I'd like my program to be able to get the single exact solution to this problem in polynomial time", but for optimization we're often perfectly happy with "I'd like my program to be able to get close to the exact solution in polynomial time", or even just "I'd like my program to be able to get a much better solution than people's previous intuitive guesses".

that was unworthy of a transhumanist magazine. it read like an AI column for a fashion magazine.