M. Y. Zuo's Shortform

post by M. Y. Zuo · 2021-11-07T01:42:46.261Z · LW · GW · 27 comments


Comments sorted by top scores.

comment by M. Y. Zuo · 2021-11-07T01:42:46.686Z · LW(p) · GW(p)

Hi everyone, I’m fairly new to the community, though I’ve been lurking on and off for a few years, and I would like to hear the opinions on a key question I am unsure about.

What is the ultimate goal of the rationalist enterprise? 

I understand there are clear goals to establish ‘Friendly AI’, to realize intelligence ‘upgrading’, if achievable, life extension, and so on. But what is unclear to me is what comes next in the ideal case where all these goals have been achieved, and to what ultimate end.

For example, 

I’ve encountered  discussions about eudaimonia scenarios (private galaxies, etc.), though I’m not sure how seriously to take those, as surely the possibilities of the co-moving light cone that is within our capacity to inhabit are exhaustible in finite time, especially if all these designs reach their ultimate fruition? 

Replies from: ChristianKl, Viliam, maximkazhenkov
comment by ChristianKl · 2021-11-07T08:49:02.453Z · LW(p) · GW(p)

There are no shared ultimate goals of the rationalist enterprise. Different rationalist have different goals. 

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-16T15:28:27.829Z · LW(p) · GW(p)

+1 on this

Rationality is instrumental.

comment by Viliam · 2021-11-07T23:50:48.832Z · LW(p) · GW(p)

I think the idea is to have as much fun as possible [LW · GW], and keep doing science (which might expand our opportunities to have fun).

In the very long term, if the universe runs out of energy and nothing in the new science allows us to overcome this issue, then, sadly, we die.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-08T13:31:26.384Z · LW(p) · GW(p)

Well that is an article that, although interesting, seems to miss a key factor in presenting the eudaemonia scenario ('maximizing fun'). Because it does not define 'fun'. e.g. a paperclip maximizer would consider more paperclips brought into existence as more 'fun'. 

And we know from game theory that when there is more than 1 player in any game... the inter-player dynamics ultimately decide their actions as rational agents.

So I cannot see how an individual's aspirations ('fun') are relevant to determining a future state without considering the total sum of all aspirations (sum of all 'funs') as well. Unless there is only 1 conscious entity remaining, which to be fair is not out of the realm of possibility in some very distant future. 

Also, this section of the article:

 Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil).  Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance.  Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization.

Is not convincing, because it does not actually refute Leibniz's old argument, that only an omniscient and omnipresent being could 'clearly' see whether the world is benevolently designed or not, whether it has been optimized, along all dimensions, to the greatest extent possible or not, even whether it even has any flaws on a total net basis or not.

And I've not yet seen a convincing disproof of those arguments. 

Now of course I personally am leery of believing those claims to be true, but then I also cannot prove with 100% certainty that they are false. And the 'Fun Theory' article is certainly presented as if there was such proof.

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-16T15:32:50.087Z · LW(p) · GW(p)

Because it does not define 'fun'.

Because you need to figure out or intuit what is fun for you. It may not be identical to what is fun for Yudkowsky. It beingvery arbitrary all the more means you can't use some rationalist discourse to arrive at an understanding of what 'fun' is.

only an omniscient and omnipresent being could 'clearly' see whether the world is benevolently designed or not

We can meaningfully move towards goals we know we can never attain, forever.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-16T16:46:35.305Z · LW(p) · GW(p)

So why must we prevent paperclip optimizers from bringing about their own ‘fun’?

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-16T19:13:29.545Z · LW(p) · GW(p)

If you really feel comfortable with that, you can do that. I wondered this too [EA · GW].

Most people are not keen on sacrificing current human civilisation at the altar of a different one.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-18T05:35:08.665Z · LW(p) · GW(p)

What’s the rational basis for preferring all mass-energy consuming grey goo created by humans over all mass-energy consuming grey goo created by a paperclip optimizer? The only possible ultimate end in both scenarios is heat death anyways.

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-18T06:50:09.430Z · LW(p) · GW(p)

Ideal rational agents can be programmed with arbitrary goals. Such an agent won't ask questions like "why is my terminal goal X not Y", cause "terminal goal = X" is an axiom. You can't prove axioms out of anything more fundmental. If you could, those more fundamental things would be the axioms then. And two different agents can have different sets of axioms.

Non-ideal agents can have all sorts of behaviours, some of which are goal-seeking, some of which are rational, some of which are not. Humans are in this category.

Human brains are programmed to release neurochemicals on knowing that other humans are thriving. But since we're neither perfectly rational, nor perfectly goal-seeking, we end up asking questions like "why are my goals X not Y", which would look weird to an ideal agent.

In short you can't use rationalist discourse to "solve" what the ideal goals are. Your goals are roughly - the things you care about. Most people care about preventing human extinction.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-20T03:07:04.237Z · LW(p) · GW(p)

If no one’s goals can be definitely proven to be better than anyone else’s goals, then it doesnt seem like we can automatically conclude the majority of present or future humans, or our descendants, will prioritize maximizing fun, happiness, etc.

If some want to pursue that then fine, if others want to pursue different goals, even ones that are deleterious to overall fun, happiness, etc., then there doesn’t seem to be a credible argument to dissuade them?

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-20T03:22:39.972Z · LW(p) · GW(p)

there doesn’t seem to be a credible argument to dissuade them

There can be still be arguments like:

 - "You will be unhappy if you resist what you are biologically hardwired to do. You probably don't want that."

 - "People will socially isolate from you if you want human extinction. You don't want the former."

Basically vaguely point at inconsistency within your wants, rather than which wants are universally correct or wrong in some metaphysical frame. Most people don't want their wants to be inconsistent. (So much so that some people come up with elaborate moral frameworks and theories in an attempt to find or create consistency.)

But I agree with you broadly, arguments over ethics can become harder to relate to once you are aware of this.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-20T17:16:08.007Z · LW(p) · GW(p)

Those appear to be examples of arguments from consequences, a logical fallacy. How could similar reasoning be derived from axioms, if at all?

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-20T17:53:15.754Z · LW(p) · GW(p)

Don't know.

"Axioms plus their consequences" is a toy/ideal model that may look very different from how desires and reasoning are actually wired in the brain. You can check out Coherentism for some intuitions on an alternate model. Deeper understanding over which model best describes how human brains work or are constrained, is an open problem. Someone besides me might have better resources.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-28T17:28:19.751Z · LW(p) · GW(p)

Let’s think about it another way. Consider the thought experiment where a single normal cell is removed from the body of any randomly selected human. Clearly they would still be human.

If you keep on removing normal cells though eventually they would die. And if you keep on plucking away cells eventually the entire body would be gone and only cancerous cells would be left, i.e. only a ‘paperclip optimizer’ would remain from the original human, albeit inefficient and parasitic ‘paperclips’ that need a organic host.

(Due to the fact that everyone has some small number of cancerous cells at any given time that are taken care of by regular processes)

At what point does the human stop being ‘human’ and starts being a lump of flesh? And at what point does the lump of flesh become a latent ‘paperclip optimizer’?

Without a sharp cutoff, which I don’t think there is, there will inevitably be inbetween cases where your proposed methods cannot be applied consistently. 

The trouble is if we, or the decision makers of the future, accept even one idea that is not internally consistent then it hardly seems like anyone will be able to refrain from accepting other ideas that are internally contradictory too. Nor will everyone err in the same way. There is no rational basis to accept one or another as a contradiction can imply anything at all, as we know from basic logic.

Then the end result will appear quite like monkey tribes fighting each other, agitating against each and all based on which inconsistencies they accept or not. Regardless of what they call each other, humans, aliens, AI, machines, organism, etc…

Replies from: samuel-shadrach
comment by acylhalide (samuel-shadrach) · 2021-11-29T05:51:30.441Z · LW(p) · GW(p)

I agree with most of this, and my intuitions are towards AI alignment being impossible for these very reasons. Humans not being capable of consistency doesn't seem to me like something we can change through sheer willpower alone. We have entire cognitive modules that are not designed for rational thinking in the first place. Perhaps only neurosurgery can change that.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-12-01T18:16:51.617Z · LW(p) · GW(p)

It does seem like alignment for all intents and purposes is impossible. Creating an AI truly beyond us then is really creating future, hopefully doting, parents to live under.

comment by maximkazhenkov · 2021-11-07T13:38:50.719Z · LW(p) · GW(p)

But what is unclear to me is what comes next in the ideal case where all these goals have been achieved

You live happily ever after.

I’ve encountered  discussions about eudaimonia scenarios (private galaxies, etc.), though I’m not sure how seriously to take those, as surely the possibilities of the co-moving light cone that is within our capacity to inhabit are exhaustible in finite time, especially if all these designs reach their ultimate fruition? 

Where is the contradiction here?

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-11-07T14:34:09.080Z · LW(p) · GW(p)

What occurs after the finite period of expansion and enjoyment ends?

Replies from: maximkazhenkov
comment by maximkazhenkov · 2021-11-08T00:17:01.803Z · LW(p) · GW(p)

Death, presumably.

(the finiteness is actually far from certain, but that's neither here nor there)

comment by M. Y. Zuo · 2021-12-12T05:15:50.207Z · LW(p) · GW(p)
  • Some of the people on death row today might not be there if the courts had not been so lenient on them when they were first offenders. Thomas Sowell

The problem of ‘justice’. Over what time scales do we refer to when we say ‘justice’? Near ‘justice’ can be in reality a far tyranny. And vice versa. Thoughts?

Replies from: ChristianKl
comment by ChristianKl · 2021-12-12T12:38:18.881Z · LW(p) · GW(p)

Some of them are also on death row because the courts were too harsh and put them together with other criminals and got them to join a gang to survive in a harsh enviroment.

Replies from: frontier64, M. Y. Zuo
comment by frontier64 · 2021-12-13T17:52:56.805Z · LW(p) · GW(p)

This happens exceedingly rarely. The thing you should understand about the American court system before judging it is that non-murderers rarely get sent to prison for more than a year for a first-time offense. If it's truly non-violent drug possession or dealing then in all likelihood a first-time offender won't get more than probation. The prison system is not creating felons by punishing people too harshly.

Look at the kid sent to prison for 7 years for threatening to shoot up his school on runescape. That's one of those examples of a way over-harsh punishment only given to send a message. The kid didn't fall in with a bad crowd because he was never a bad person. Prison doesn't force good people to become murderers when they get out. This kid did his unjust sentence, got out, and he's still a nice a dude.

It's fairly easy to picture violent criminals as similar to you but unlucky to have grown up in a worse situation or been screwed by the system. It's hard to understand them as they are: people who are incredibly different from you and have a value system that is not aligned with society.

comment by M. Y. Zuo · 2021-12-12T15:02:58.931Z · LW(p) · GW(p)

Sounds like a possible scenario as well.  Are they both just, both unjust, one or the other, variable?

And what period of time should we use as a standard? 

The same for both scenarios, lenient punishments encouraging more severe crimes later on, and onerous punishments also encouraging more severe crimes. since both lead to the same outcome?

Or different?

Replies from: ChristianKl
comment by ChristianKl · 2021-12-12T21:26:52.595Z · LW(p) · GW(p)

When it comes to interacting with complex systems expecting them to work according to your own preconceptions is generally a bad idea. You want policy to be driven by evidence about effects of interventions and not just based on thought experiments. You want to build feedback system into your system to optimize it's actions. 

You want to produce institutions that assume that they can't know the answer to question like this just by thinking about but that think about how to gather the evidence to make informed policy choices.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2021-12-12T23:00:49.283Z · LW(p) · GW(p)

Well that’s all well and good but all organizations, including all conceivable institutions, will eventually seek to optimize towards goals that we, present day people, cannot completely control. 

i.e. they will carry out their affairs using whatever is at hand through their own preconceptions, regardless of how perfect our initial designs are or what we want their behaviours to be or how much we wish for them to lack preconceptions. They will seek answers to similar questions, perhaps with the same or different motivations.

So then if we proceed along such a path the same problem appears at the meta level. How do we take into account what future actors will consider what period of time they should use as a standard? (In order to build the ‘feedback system’ for them to operate in)