Comment by player_03 on [HPMOR] Harry - example or anti-example? · 2020-02-25T22:13:15.347Z · score: 1 (1 votes) · LW · GW

It is a stretch, which is why it needed to be explained.

And yes, it would kind of make him immune to dying... in cases where he could be accidentally rescued. Cases like a first year student's spell locking a door, which an investigator could easily dispel when trying to investigate.

Oh, and I guess once it was established, the other time travel scenes would have had to be written differently. Or at least clarify that "while Draco's murder plot was flimsy enough that the simplest timeline was the timeline in which it failed, Quirrel's murder plot was bulletproof enough that the simplest outcome was for it to succeed." Because authors write the rules, they can get away with a lot of nonsense. But in this kind of story, they do need to acknowledge and (try to) explain any inconsistencies.

And here's the line I was referring to:

"The earlier experiment had measured whether Transfiguring a long diamond rod into a shorter diamond rod would allow it to lift a suspended heavy weight as it contracted, i.e., could you Transfigure against tension, which you in fact could." (Chapter 28, foreshadowing the nanotube, which may or may not have been what you were talking about)

Comment by player_03 on [HPMOR] Harry - example or anti-example? · 2020-02-25T08:13:24.313Z · score: 1 (1 votes) · LW · GW

I don't mind the occasional protagonist who makes their own trouble. I agree it would be annoying if all protagonists were like that (and I agree that Harry is annoying in general), there's room in the world for stories like this.

Now that you mention it, your first example does sound like a Deus Ex Machina. Except that

the story already established that the simplest possible time loop is preferred, and it's entirely possible that if Harry hadn't gotten out to pass a note, someone would have gone back in time to investigate his death, and inadvertently caused a paradox by unlocking the door.

This wouldn't have had to be a long explanation or full-blown lecture, just enough to confirm this interpretation. But since it wasn't confirmed and there are multiple valid interpretations of the mechanics, it does come across as a bit of an "I got out of jail free" moment.

I... don't understand your second example. I think that part of the story works just fine. Harry's solution was plausible, and even foreshadowed

in chapter 28 when he used transfiguration to apply force.

Comment by player_03 on [HPMOR] Harry - example or anti-example? · 2020-02-25T04:26:02.996Z · score: 3 (3 votes) · LW · GW

It's been a while since I read it, but off the top of my head I can't recall any blatant cases of Deus ex Machina. I'd ask for concrete examples, but I don't think it would be useful. I'm sure you can provide an example, and in turn I'll point out reasons why it doesn't count as Deus ex Machina. We'd argue about how well the solution was explained, and whether enough clues were presented far enough in advance to count as valid foreshadowing, and ultimately it'll come down to opinion.

Instead, I can go ahead and answer your question. Eliezer definitely meant to teach useful lessons. Not everything Harry does is meant to be a good example (I mean, even Eliezer knows better than to write a completely perfect character), which is probably why he gets into all that trouble. But whenever a character goes into Lecture Mode while solving a problem, it's meant to be both useful and accurate.

Wait a minute, are you talking about Lecture Mode when you say "Deus ex Machina"? I can kind of see that: the situation seems hopeless and then someone (usually Harry) gives a long explanation and suddenly the problem is solved. Thing is, these lectures don't pull the solution out of nowhere. The relevant story details were established beforehand, and the lecture just puts them together. (Or at least, that was the author's intent. As I said, it comes down to opinion.)

Comment by player_03 on Defining the normal computer control problem · 2017-04-27T00:20:36.624Z · score: 2 (2 votes) · LW · GW

An example I like is the Knight Capital Group trading incident. Here are the parts that I consider relevant:

KCG deployed new code to a production environment, and while I assume this code was thoroughly tested in a sandbox, one of the production servers had some legacy code ("Power Peg") that wasn't in the sandbox and therefore wasn't tested with the new code. These two pieces of code used the same flag for different purposes: the new code set the flag during routine trading, but Power Peg interpreted that flag as a signal to buy and sell ~10,000 arbitrary* stocks.

*Actually not arbitrary. What matters is that the legacy algorithm was optimized for something other than making money, so it lost money on average.

They stopped this code after 45 minutes, but by then it was too late. Power Peg had already placed millions of inadvisable orders, nearly bankrupting KCG.

Sometimes, corrigibility isn't enough.

Comment by player_03 on Use the Try Harder, Luke · 2014-07-20T00:18:51.466Z · score: 0 (0 votes) · LW · GW

I suppose, but even then he would have to take time to review the state of the puzzle. You would still expect him to take longer to spot complex details, and perhaps he'd examine a piece or two to refresh his memory.

But that isn't my true rejection here.

If you assume that Claude's brother "spent arbitrarily much time" beforehand, the moral of the story becomes significantly less helpful: "If you're having trouble, spend an arbitrarily large amount of time working on the problem."

Comment by player_03 on Use the Try Harder, Luke · 2014-07-18T06:24:57.313Z · score: 6 (6 votes) · LW · GW

His brother's hint contained information that he couldn't have gotten by giving the hint to himself. The fact that his brother said this while passing by means that he spotted a low-hanging fruit. If his brother had spent more time looking before giving the hint, this would have indicated a fruit that was a little higher up.

This advice is worth trying, but when you give it to yourself, you can't be sure that there's low hanging fruit left. If someone else gives it to you, you know it's worth looking for, because you know there's something there to find. (The difference is that they, not you, took the time to search for it.)

Again, it's a worthwhile suggestion. I just want to point out that it boils down to "If you're having trouble, check for easier solutions," and that while you can always give this advice to yourself, it will not always help.

Comment by player_03 on Harry Potter and the Methods of Rationality discussion thread, part 13, chapter 81 · 2014-04-24T02:03:49.696Z · score: 0 (0 votes) · LW · GW

Harry left "a portion of his life" (not an exact quote) in Azkaban, and apparently it will remain there forever. That could be the remnant that Death would fail to destroy.

Anyway, Snape drew attention to the final line in the prophecy. It talked about two different spirits that couldn't exist in the same world, or perhaps two ingredients that cannot exist in the same cauldron. That's not Harry and Voldemort; that's Harry and Death.

I mean, Harry has already sworn to put an end to death. It's how he casts his patronus. He's a lot less sure about killing Voldemort, and would prefer not to, if given the choice.

Comment by player_03 on MIRI's Winter 2013 Matching Challenge · 2014-01-12T04:26:45.743Z · score: 0 (0 votes) · LW · GW

On the other hand, MIRI hit its goal three weeks early, so the amount of support is pretty obvious.

Though I have to admit, I was going to remain silent too, and upon reflection I couldn't think of any good reasons to do so. It may not be necessary, but it couldn't hurt either. So...

I donated $700 to CFAR.

Comment by player_03 on 2013 Less Wrong Census/Survey · 2013-11-23T06:35:28.492Z · score: 14 (14 votes) · LW · GW

So that's how Omega got the money for box B!

Comment by player_03 on Rationality Quotes October 2013 · 2013-10-06T04:01:31.905Z · score: 8 (8 votes) · LW · GW

Well designed traditions and protocols will contain elements that cause most competent people to not want to throw them out.

Comment by player_03 on Rationality Quotes October 2013 · 2013-10-06T03:59:56.814Z · score: 1 (1 votes) · LW · GW

Having just listened to much of the Ethical Injunctions sequence (as a podcast courtesy of George Thomas), I'm not so sure about this one. There are reasons for serious, competent people to follow ethical rules, even when they need to get things done in the real world.

Ethics aren't quite the same as tradition and protocol, but even so, sometimes all three of those things exist for good reasons.

Comment by player_03 on The genie knows, but doesn't care · 2013-09-13T02:06:08.540Z · score: 4 (4 votes) · LW · GW


Though actually, Eliezer used similar phrasing regarding Richard Loosemore and got downvoted for it (not just by me). Admittedly, "persistent troll" is less extreme than "permanent idiot," but even so, the statement could be phrased to be more useful.

I'd suggest, "We've presented similar arguments to [person] already, and [he or she] remained unconvinced. Ponder carefully before deciding to spend much time arguing with [him or her]."

Not only is it less offensive this way, it does a better job of explaining itself. (Note: the "ponder carefully" section is quoting Eliezer; that part of his post was fine.)

Comment by player_03 on The genie knows, but doesn't care · 2013-09-12T07:02:14.289Z · score: 2 (2 votes) · LW · GW

You and I are both bound by the terms of a scenario that someone else has set here.

Ok, if you want to pass the buck, I won't stop you. But this other person's scenario still has a faulty premise. I'll take it up with them if you like; just point out where they state that the goal code starts out working correctly.

To summarize my complaint, it's not very useful to discuss an AI with a "sincere" goal of X, because the difficulty comes from giving the AI that goal in the first place.

What you did was consider some other possibilities, such as those in which the AI is actually not being sincere. Nothing wrong with considering those, but that would be a story for another day.

As I see it, your (adopted) scenario is far less likely than other scenario(s), so in a sense that one is the "story for another day." Specifically, a day when we've solved the "sincere goal" issue.

Comment by player_03 on The genie knows, but doesn't care · 2013-09-11T02:22:34.249Z · score: 1 (1 votes) · LW · GW

I didn't mean to ignore your argument; I just didn't get around to it. As I said, there were a lot of things I wanted to respond to. (In fact, this post was going to be longer, but I decided to focus on your primary argument.)

Your story:

This hypothetical AI will say “I have a goal, and my goal is to get a certain class of results, X, in the real world.” [...] And we say “Hey, no problem: looks like your goal code is totally consistent with that verbal description of the desired class of results.” Everything is swell up to this point.

My version:

The AI is lying. Or possibly it isn't very smart yet, so it's bad at describing its goal. Or it's oversimplifying, because the programmers told it to, because otherwise the goal description would take days. And the goal code itself is too complicated for the programmers to fully understand. In any case, everything is not swell.

Your story:

Then one day the AI says “Okay now, today my goalX code says I should do this…” and it describes an action that is VIOLENTLY inconsistent with the previously described class of results, X. This action violates every one of the features of the class that were previously given.

My version:

The AI's goal was never really X. It was actually Z. The AI's actions perfectly coincide with Z.

In the rest of the scenario you described, I agree that the AI's behavior is pretty incoherent, if its goal is X. But if it's really aiming for Z, then its behavior is perfectly, terrifyingly coherent.

And your "obvious" fail-safe isn't going to help. The AI is smarter than us. If it wants Z, and a fail-safe prevents it from getting Z, it will find a way around that fail-safe.

I know, your premise is that X really is the AI's true goal. But that's my sticking point.

Making it actually have the goal X, before it starts self-modifying, is far from easy. You can't just skip over that step and assume it as your premise.

Comment by player_03 on The genie knows, but doesn't care · 2013-09-10T08:41:57.759Z · score: 5 (7 votes) · LW · GW

Oh, yeah, I found that myself eventually.

Anyway, I went and read the the majority of that discussion (well, the parts between Richard and Rob). Here's my summary:


I think that what is happening in this discussion [...] is a misunderstanding. [...]

[Rob responds]


You completely miss the point that I was trying to make. [...]

[Rob responds]


You are talking around the issue I raised. [...] There is a gigantic elephant in the middle of this room, but your back is turned to it. [...]

[Rob responds]


[...] But each time I explain my real complaint, you ignore it and respond as if I did not say anything about that issue. Can you address my particular complaint, and not that other distraction?

[Rob responds]


[...] So far, nobody (neither Rob nor anyone else at LW or elsewhere) will actually answer that question. [...]

[Rob responds]


Once again, I am staggered and astonished by the resilience with which you avoid talking about the core issue, and instead return to the red herring that I keep trying to steer you away from. [...]


Alright. You say I’ve been dancing around your “core” point. I think I’ve addressed your concerns quite directly, [...] To prevent yet another suggestion that I haven’t addressed the “core”, I’ll respond to everything you wrote above. [...]


Rob, it happened again. [...]

I snipped a lot of things there. I found lots of other points I wanted to emphasize, and plenty of things I wanted to argue against. But those aren't the point.

Richard, this next part is directed at you.

You know what I didn't find?

I didn't find any posts where you made a particular effort to address the core of Rob's argument. It was always about your argument. Rob was always the one missing the point.

Sure, it took Rob long enough to focus on finding the core of your position, but he got there eventually. And what happened next? You declared that he was still missing the point, posted a condensed version of the same argument, and posted here that your position "withstands all the attacks against it."

You didn't even wait for him to respond. You certainly didn't quote him and respond to the things he said. You gave no obvious indication that you were taking his arguments seriously.

As far as I'm concerned, this is a cardinal sin.

I think I am explaining the point with such long explanations that I am causing you to miss the point.

How about this alternate hypothesis? Your explanations are fine. Rob understands what you're saying. He just doesn't agree.

Perhaps you need to take a break from repeating yourself and make sure you understand Rob's argument.

(P.S. Eliezer's ad hominem is still wrong. You may be making a mistake, but I'm confident you can fix it, the tone of this post notwithstanding.)

Comment by player_03 on The genie knows, but doesn't care · 2013-09-10T07:21:14.569Z · score: 0 (2 votes) · LW · GW

Link to the nailed-down version of the argument?

Comment by player_03 on The genie knows, but doesn't care · 2013-09-10T06:22:44.849Z · score: 4 (4 votes) · LW · GW

I posted elsewhere that this post made me think you're anthropomorphizing; here's my attempt to explain why.

egregiously incoherent behavior in ONE domain (e.g., the Dopamine Drip scenario)

the craziness of its own behavior (vis-a-vis the Dopamine Drip idea)

if an AI cannot even understand that "Make humans happy" implies that humans get some say in the matter

Ok, so let's say the AI can parse natural language, and we tell it, "Make humans happy." What happens? Well, it parses the instruction and decides to implement a Dopamine Drip setup.

As FeepingCreature pointed out, that solution would in fact make people happy; it's hardly inconsistent or crazy. The AI could certainly predict that people wouldn't approve, but it would still go ahead. To paraphrase the article, the AI simply doesn't care about your quibbles and concerns.

For instance:

people might consider happiness to be something that they do not actually want too much of

Yes, but the AI was told, "make humans happy." Not, "give humans what they actually want."

people might be allowed to be uncertain or changeable in their attitude to happiness

Yes, but the AI was told, "make humans happy." Not, "allow humans to figure things out for themselves."

subtleties implicit in that massive fraction of human literature that is devoted to the contradictions buried in our notions of human happiness

Yes, but blah blah blah.

Actually, that last one makes a point that you probably should have focused on more. Let's reconfigure the AI in light of this.

The revised AI doesn't just have natural language parsing; it's read all available literature and constructed for itself a detailed and hopefully accurate picture of what people tend to mean by words (especially words like "happy"). And as a bonus, it's done this without turning the Earth into computronium!

This certainly seems better than the "literal genie" version. And this time we'll be clever enough to tell it, "give humans what they actually want." What does this version do?

My answer: who knows? We've given it a deliberately vague goal statement (even more vague than the last one), we've given it lots of admittedly contradictory literature, and we've given it plenty of time to self-modify before giving it the goal of self-modifying to be Friendly.

Maybe it'll still go for the Dopamine Drip scenario, only for more subtle reasons. Maybe it's removed the code that makes it follow commands, so the only thing it does is add the quote "give humans what they actually want" to its literature database.

As I said, who knows?

Now to wrap up:

You say things like "'Make humans happy' implies that..." and "subtleties implicit in..." You seem to think these implications are simple, but they really aren't. They really, really aren't.

This is why I say you're anthropomorphizing. You're not actually considering the full details of these "obvious" implications. You're just putting yourself in the AI's place, asking yourself what you would do, and then assuming that the AI would do the same.

Comment by player_03 on The genie knows, but doesn't care · 2013-09-10T04:54:46.909Z · score: 1 (5 votes) · LW · GW

I did see the insult, but Eliezer (quite rightly) got plenty of downvotes for it. I'm pretty sure that's not the reason you're being rated down.

I myself gave you a downvote because I got a strong impression that you were anthropomorphizing. Note that I did so before reading Eliezer's comment.

I certainly should have explained my reasons after voting, but I was busy and the downvote button seemed convenient. Sorry about that. I'll get started on a detailed response now.

Comment by player_03 on The genie knows, but doesn't care · 2013-09-07T08:07:25.490Z · score: 8 (14 votes) · LW · GW

I want to upvote this for the link to further discussion, but I also want to downvote it for the passive-aggressive jab at LW users.

No vote.

Comment by player_03 on MIRI's 2013 Summer Matching Challenge · 2013-08-11T00:05:41.101Z · score: 12 (12 votes) · LW · GW

I've donated a second $1000.

Comment by player_03 on MIRI's 2013 Summer Matching Challenge · 2013-07-26T16:46:38.783Z · score: 19 (21 votes) · LW · GW

I donated $1000 and then went and bought Facing the Intelligence Explosion for the bare minimum price. (Just wanted to put that out there.)

I've also left myself a reminder to consider another donation a few days before this runs out. It'll depend on my financial situation, but I should be able to manage it.

Comment by player_03 on Reductionism sequence now available in audio format · 2013-06-02T18:41:18.536Z · score: 1 (1 votes) · LW · GW

A podcast entry is included for that one, but it just directs you to read the original article.

I was going to link you one of the other podcasts (which all provide samples), but then I realized you might be asking why this specific podcast doesn't have one.

Comment by player_03 on The Lifespan Dilemma · 2013-04-13T04:46:37.802Z · score: 1 (1 votes) · LW · GW

Ok, yeah, in that case my response is to take as many deals as Omega offers.

AdeleneDawner and gwern provide a way to make the idea more palatable - assume MWI. That is, assume there will be one "alive" branch and a bunch of "dead" branches. That way, your utility payoff is guaranteed. (Ignoring the grief of those around you in all the "dead" branches.)

Without that interpretation, the idea becomes scarier, but the math still comes down firmly on the side of accepting all the offers. It certainly feels like a bad idea to accept that probability of death, no matter what the math says, but as far as I can tell that's scope insensitivity talking.

With that in mind, my only remaining objection is the "we can do better than that" argument presented above. My feeling is, why not use a few of those 10^10,000,000,000 years to figure out a way to live even longer? Omega won't allow it? Ok, so I don't want to get involved with Omega in the first place; it's not worth losing my (admittedly slim) chances at actual immortality. Too late for that? Fine, then I'll sit down, shut up, and multiply.

Comment by player_03 on The Lifespan Dilemma · 2013-04-12T23:37:43.704Z · score: 1 (1 votes) · LW · GW

One-in-a-million is just an estimate. Immortality is a tough proposition, but the singularity might make it happen. The important part is that it isn't completely implausible.

I'm not sure what you mean, otherwise.

Are you suggesting that Omega takes away any chance of achieving immortality even before making the offer? In that case, Omega's a jerk, but I'll shut up and multiply.

Or are you saying that 10^10,000,000,000 years could be used for other high-utility projects, like making simulated universes full of generally happy people? Immortality would allow even more time for that.

Comment by player_03 on The Lifespan Dilemma · 2013-04-12T22:26:16.662Z · score: 0 (2 votes) · LW · GW

So let me get this straight: I get an 80% chance of living 10^10,000,000,000 years, and a 20% chance of dying within the hour?

Awesome! A little unsettling, but the math checks out. I'll take it!

Another deal? Go down to a 79.99992% probability of living, but up to 10^10^10,000,000,000 years? Are you crazy? No!

Well, the difference being that 10^10,000,000,000 years is way more than I'll need to figure out immortality, even taking Hofstadter's Law into account! You've already given me immortality; why would I agree to reducing my chances?

Not allowed? What?

Omega's going to ensure that my life ends? Just to fulfill his (her? its?) guarantee?

The deal is off!

Yes, off. Tell Omega to take this offer and stick it somewhere light can't escape from! Omega isn't offering an extended lifespan; it's offering an 80% chance of guaranteed death plus a 20% chance of guaranteed death! I'd prefer to keep my one-in-a-million chance of actual immortality, thank you very much.

(Sorry for any low-quality writing and unclear explanations. I know I'm nowhere near Eliezer's league, but that just means I need the practice.)

Comment by player_03 on Rationality Quotes April 2013 · 2013-04-08T06:25:20.379Z · score: 16 (20 votes) · LW · GW

When I was a Christian, and when I began this intense period of study which eventually led to my atheism, my goal, my one and only goal, was to find the best evidence and argument I could find that would lead people to the truth of Jesus Christ. That was a huge mistake. As a skeptic now, my goal is very similar - it just stops short. My goal is to find the best evidence and argument, period. Not the best evidence and argument that leads to a preconceived conclusion. The best evidence and argument, period, and go wherever the evidence leads.

--Matt Dillahunty

Comment by player_03 on Epilogue: Atonement (8/8) · 2013-02-22T23:45:55.892Z · score: 2 (2 votes) · LW · GW

Sorry I'm late... is no-one curious given the age of the universe why the 3 races are so close technologically?

Sorry I'm late in replying to this, but I'd guess the answer is that this is "the past's future." He would not have been able to tell this story with one species being that advanced, so he postulated a universe in which such a species doesn't exist (or at least isn't nearby).

Your in-universe explanations work as well, of course.

Comment by player_03 on Epilogue: Atonement (8/8) · 2013-02-22T23:33:18.831Z · score: 0 (0 votes) · LW · GW

Perhaps the fact that it's the "traditional and irrational" ending is the reason Eliezer went with it as the "real" one. (Note that he didn't actually label them as "good" and "bad" endings.)

Comment by player_03 on 2012 Less Wrong Census/Survey · 2012-11-04T00:38:32.034Z · score: 9 (9 votes) · LW · GW

I assumed the same, based on the definition of "god" as "supernatural" and the definition of "supernatural" as "involving ontologically basic mental entities."

(Oh, and for anyone who hasn't read the relevant post, the survey is quoting this.)

Comment by player_03 on Rationality Quotes March 2012 · 2012-03-04T00:46:17.530Z · score: 3 (9 votes) · LW · GW

I could be interpreting it entirely wrong, but I'd guess this is the list Cochran had in mind:

Comment by player_03 on Rationality Quotes November 2011 · 2011-11-01T23:36:02.840Z · score: 5 (5 votes) · LW · GW

This reminds me of Lojban, in which the constructs meaning "good" and "bad" encourage you to specify a metric. It is still possible to say that something is "worse" without providing any detail, but I suspect most Lojban speakers would remember to provide detail if there was a chance of confusion.

Comment by player_03 on Epilogue: Atonement (8/8) · 2011-10-28T05:10:56.464Z · score: 1 (1 votes) · LW · GW

Give the furries, vampire-lovers and other assorted xenophiles a few generations to chase their dreams, and you're going to start seeing groups with distinctly non-human psychology.


Because you haven't had time to read all the Orion's Arm stories, probably. (Details)

Comment by player_03 on Epilogue: Atonement (8/8) · 2011-10-28T04:50:10.593Z · score: 9 (11 votes) · LW · GW

2) Honestly, I would have been happy with the aliens' deal (even before it was implemented), and I think there is a ~60% chance that Elizier agrees.

I'm of the opinion that pain is a bad thing, except insofar as it prevents you from damaging yourself. People argue that pain is necessary to provide contrast to happiness, and that pleasure wouldn't be meaningful without pain, but I would say that boredom and slight discomfort provide more than enough contrast.

However, this future society disagrees. The idea that "pain is important" is ingrained in these people's minds, in much the same way that "rape is bad" is ingrained in ours. I think one of the main points Elizier is trying to make is that we would disagree with future humans almost as much as we would disagree with the baby-eaters or superhappies.

(Edit 1.5 years later: I was exaggerating in that second paragraph. I suspect I was trying too hard to sound insightful. The claims may or may not have merit, but I would no longer word them as forcefully.)

Comment by player_03 on Rationality Quotes August 2011 · 2011-08-05T00:57:56.599Z · score: 1 (1 votes) · LW · GW

If you're trying to present any kind of information at all, you should figure out what is important about it and what presentation will make it clear.

Unfortunately, the quote above isn't at all clear, even in context. I suspect this is because Jacques Bertin isn't as good at expressing himself in English as in French, but even so I'm unable to understand the sample data he presents or how it relates to the point he was trying to make.

Comment by player_03 on How to Seem (and Be) Deep · 2011-08-04T22:06:56.137Z · score: 0 (0 votes) · LW · GW

I agree with your interpretation of the song, and to back it up, here's the chorus of "The Farthest Star" (another song by the same band).

We possess the power, if this should start to fall apart, To mend divides, to change the world, to reach the farthest star. If we should stay silent, if fear should win our hearts, Our light will have long diminished before it reaches the farthest star.

This time, the message is pretty clear: we should aspire to overcome both our differences and our limitations, to avoid extinction and to expand throughout the universe. I suppose it doesn't say anything about immortality, but otherwise it seems to match transhumanist philosophy.

Comment by player_03 on Nonperson Predicates · 2011-08-04T07:28:58.889Z · score: 0 (0 votes) · LW · GW

If the problem here is that the entity being simulated ceases to exist, an alternative solution would be to move the entity into an ongoing simulation that won't be terminated. Clearly, this would require an ever-increasing number of resources as the number of simulations increased, but perhaps that would be a good thing - the AI's finite ability to support conscious entities would impose an upper bound on the number of simulations it would run. If it was important to be able to run such a simulation, it could, but it wouldn't do so frivolously.

Before you say anything, I don't actually think the above is a good solution. It's more like a constraint to be routed around than a goal to be achieved. Plus, it's far too situational and probably wouldn't produce desirable results in situations we didn't design it for.

The thing is, it isn't important to come up with the correct solution(s) ourselves. We should instead make the AI understand the problem and want to solve it. We need to carefully design the AI's utility function, making it treat conscious entities in simulations as normal people and respect simulated lives as it respects ours. We will of course need a highly precise definition for consciousness that applies not just to modern-day humans but also to entities the AI could simulate.

Here's how I see it - as long as the AI values the life of any entity that values its own life, the AI will find ways to keep those entities alive. As long as it considers entities in simulations to be equivalent to entities outside, it will avoid terminating their simulation and killing them. The AI (probably) would still make predictions using simulations; it would just avoid the specific situation of destroying a conscious entity that wanted to continue living.

Comment by player_03 on Rationality Quotes August 2011 · 2011-08-03T19:21:29.631Z · score: 22 (22 votes) · LW · GW

Daniel Oppenheimer's Ig Nobel Prize acceptance speech:

My research shows that conciseness is interpreted as intelligence. So, thank you.

Comment by player_03 on On the unpopularity of cryonics: life sucks, but at least then you die · 2011-07-30T07:46:04.064Z · score: 6 (6 votes) · LW · GW

Until then, I'd be more interested in donating to general life extension research than paying for cryonics specifically.

This is very similar to my primary objection to cryonics.

I realize that, all factors considered, the expected utility you'd get from signing up for cryonics is extremely, if not infinitely, large. In any case, it's certainly large enough to be worth the price.

However, it seems to me that there are better alternatives. Sure, paying for cryonics increases your chances of an unlimited life by many orders of magnitude. On the other hand, funding longevity research makes it more likely that we will ever overcome aging and disease. Unlimited life for most or all of the future human population is far more important than unlimited life for yourself, right? (One might object that life extension research is already on its way to accomplishing this regardless of your contributions, which brings me to my next point.)

If an existential risk comes to pass, then no one will have a chance at an unlimited life. All of the time and money spent on cryonics will go to waste, and life extension research will have been (mostly) squandered. Preventing this sort of risk is therefore far more important than preserving any one person, even if that person is you. To make matters worse, there are multiple existential risks that have a significant chance of happening, so the need for extra attention and donations is much greater than the need for extra longevity research.

To summarize: Cryonics gives you alone a significant chance of gaining unlimited life. Working to prevent existential risk gives billions of people a slightly increased chance of the same.

It seems to me we shouldn't be spending money on freezing ourselves just in case a singularity (or equivalent scientific progress) happens. Instead, we should focus on increasing the chances that it will happen at all. To do anything else would be selfish.

Ok, time to take a step back and look at some reasons I might be wrong.

First, and perhaps most obviously, people are not inclined to donate all their money to any cause, no matter how important. I freely admit that I will probably donate only a small fraction of my earnings, despite the arguments I made in this post. Plus, it's possible (likely?) that people would be more inclined to spend money on cryonics than on existential risk reduction, because cryonics benefits them directly. If someone is going to spend money selfishly, I suppose cryonics is the most beneficial way to do so.

Second, there's a chance I misestimated the probabilities involved, and in fact your money would be best spent on cryonics. If the Cryonics Institute webpage is to be believed, the cheapest option costs $28,000, which is generally covered by insurance, costing you $120 per year (this option also requires a one-time payment of $1,250). Unfortunately, I have no idea how much $1,250 plus $120 per year would help if donated to SIAI or another such organization. Cryonics certainly give a huge expected reward, and I'm just guessing at the expected reward for donating.

Comment by player_03 on 0 And 1 Are Not Probabilities · 2011-07-07T07:52:50.277Z · score: 0 (0 votes) · LW · GW

Thanks for the analysis, MathijsJ! It made perfect sense and resolved most of my objections to the article.

I was willing to accept that we cannot reach absolute certainty by accumulating evidence, but I also came up with multiple logical statements that undeniably seemed to have probability 1. Reading your post, I realized that my examples were all tautologies, and that your suggestion to allow certainty only for tautologies resolved the discrepancy.

The Wikipedia article timtyler linked to seems to support this: "Cromwell's rule [...] states that one should avoid using prior probabilities of 0 or 1, except when applied to statements that are logically true or false." This matches your analysis - you can only be certain of tautologies.

Also, your discussion of models neatly resolves the distinction between, say, a mathematically-defined die (which can be certain to end up showing an integer between 1 and 6) and a real-world die (which cannot quite be known for sure to have exactly six stable states).

Eliezer makes his position pretty clear: "So I propose that it makes sense to say that 1 and 0 are not in the probabilities; just as negative and positive infinity, which do not obey the field axioms, are not in the real numbers."

It's true - you cannot ever reach a probability of 1 if you start at 0.5 and accumulate evidence, just as you cannot reach infinity if you start at 0 and add integer values. And the inverse is true, too - you cannot accumulate evidence against a tautology and bring its probability down to anything less than 1. But this doesn't mean a probability of 1 is an incoherent concept or anything.

Eliezer: if you're going to say that 0 and 1 are not probabilities, you need to come up with a new term for them. They haven't gone away completely just because we can't reach them.

Edit a year and a half later: I agree with the article as written, partially as a result of reading How to Convince Me That 2 + 2 = 3, and partially as a result of concluding that "tautologies that have probability 1 but no bearing on reality" is a useless concept, and that therefore, "probability 1" is a useless concept.