Posts
Comments
Gary Marcus Yann LeCun describes LLMs as "an off-ramp on the road to AGI," and I'm inclined to agree. LLMs themselves aren't likely to "turn AGI." Each generation of LLMs demonstrates the same fundamental flaws, even as they get better at hiding them.
But I also completely buy the "FOOM even without superintelligence" angle, as well as the argument that they'll speed up AI research by an unpredictable amount.
I agree that it can take a long time to prove simple things. But my claim is that one has to be very stupid to think 1+1=3
Or one might be working from different axioms. I don't know what axioms, and I'd look at you funny until you explained, but I can't rule it out. It's possible (though implausible given its length) that Principia Mathematica wasn't thorough enough, that it snuck in a hidden axiom that - if challenged - would reveal an equally-coherent alternate counting system in which 1+1=3.
I brought up Euclid's postulates as an example of a time this actually happened. It seems obvious that "two lines that are parallel to the same line are also parallel to each other," but in fact it only holds in Euclidean geometry. To quote the Wikipedia article on the topic,
Many other statements equivalent to the parallel postulate have been suggested, some of them appearing at first to be unrelated to parallelism, and some seeming so self-evident that they were unconsciously assumed by people who claimed to have proven the parallel postulate from Euclid's other postulates.
"So self-evident that they were unconsciously assumed." But it turned out, you can't prove the parallel postulate (or any equivalent postulate) from first principles, and there were a number of equally-coherent geometries waiting to be discovered once we started questioning it.
My advice is to be equally skeptical of claims of absolute morality. I agree you can derive human morality if you assume that sentience is good, happiness is good, and so on. And maybe you can derive those from each other, or from some other axioms, but at some point your moral system does have axioms. An intelligent being that didn't start from these axioms could likely derive a coherent moral system that went against most of what humans consider good.
Summary: you're speculating, based on your experience as an intelligent human, that an intelligent non-human would deduce a human-like moral system. I'm speculating that it might not. The problem is, neither of us can exactly test this at the moment. The only human-level intelligences we could ask are also human, meaning they have human values and biases baked in.
We all accept similar axioms, but does that really mean those axioms are the only option?
In other words, when I say that "Murder is bad," that is a fact about the world, as true as 2+2=4 or the Pythagorean theorem.
I like this way of putting it.
In Principia Mathematica, Whitehead and Russell spent over 300 pages laying groundwork before they even attempt to prove 1+1=2. Among other things, they needed to define numbers (especially the numbers 1 and 2), equality, and addition.
I do think "1+1=2" is an obvious fact. If someone claimed to be intelligent and also said that 1+1=3, I'd look at them funny and press for clarification. Given all the assumptions about how numbers work I've absorbed over the course of my life, I'd find it hard to conceive of anything else.
Likewise, I find it hard to conceive of any alternative to "murder is bad," because over the course of my life I've absorbed a lot of assumptions about the value of sentient life. But the fact that I've absorbed these assumptions doesn't mean every intelligent entity would agree with them.
In this analogy, the assumptions underpinning human morality are like Euclid's postulates. They seem so obvious that you might just take them for granted, as the only possible self-consistent system. But we could have missed something, and one of them might not be the only option, and there might be other self-consistent geometries/moralities out there. (The difference being that in the former case M.C. Escher uses it to make cool art, and in the latter case an alien or AI does something we consider evil.)
Back when you were underestimating Covid, how much did you hear from epidemiologists? Either directly or filtered through media coverage?
I was going to give an answer about how "taking the outside view" should work, but I realized I needed this information first.
I don't think it invalidates the claim that "Without the minimum wage law, lots of people would probably be paid significantly less." (I believe that's one of the claims you were referring to. Let me know if I misinterpreted your post.)
I don't have a whole lot of time to research economies around the world, but I checked out a couple sources with varying perspectives (two struck me as neutral, two as libertarian). One of the libertarian ones made no effort to understand or explain the phenomenon, but all three others agreed that these countries rely on strong unions to keep wages up.
IMO, that means you're both partially right. As you said, some countries can and do function without minimum wages - it's clearly possible. But as the original poster said, if a country has minimum wage laws, removing those laws will in fact tend to reduce wages.
Some countries without minimum wages still have well-paid workers. Other countries without minimum wages have sweatshops. I think that market forces push towards the "sweatshop" end of the scale (for the reasons described by the original poster), and unions are one of the biggest things pushing back.
Most of the research is aware of that limitation. Either they address it directly, or the experiment is designed to work around it, assuming mental state based on actions just as you suggest.
My point here isn't necessarily that you're wrong, but that you can make a stronger point by acknowledging and addressing the existing literature. Explain why you've settled on suicidal behavior as the best available indicator, as opposed to vocalizations and mannerisms.
This is important because, as gbear605 pointed out, most farms restrict animals' ability to attempt suicide. If suicide attempts are your main criterion, that seems likely to skew your results. (The same is true of several other obvious indicators of dissatisfaction, such as escape attempts.)
I'm afraid I don't have time to write out my own views on this topic, but I think it's important to note that several researchers have looked into the question of whether animals experience emotion. I think your post would be a lot stronger if you addressed and/or cited some of this research.
I do want to add - separately - that superrational agents (not sure about EDT) can solve this problem in a roundabout way.
Imagine if some prankster erased the "1" and "2" from the signs in rooms A1 and A2, leaving just "A" in both cases. Now everyone has less information and makes better decisions. And in the real contest, (super)rational agents could achieve the same effect by keeping their eyes closed. Simply say "tails," maximize expected value, and leave the room never knowing which one it was.
None of which should be necessary. (Super)rational agents should win even after looking at the sign. They should be able to eliminate a possibility and still guess "tails." A flaw must exist somewhere in the argument for "heads," and even if I haven't found that flaw, a perfect logician would spot it no problem.
Oh right, I see where you're coming from. When I said "you can't control their vote" I was missing the point, because as far superrational agents are concerned, they do control each other's votes. And in that case, it sure seems like they'll go for the $2, earning less money overall.
It occurs to me that if team 4 didn't exist, but teams 1-3 were still equally likely, then "heads" actually would be the better option. If everyone guesses "heads," two teams are right, and they take home $4. If everyone guesses "tails," team 3 takes home $3 and that's it. On average, this maximizes winnings.
Except this isn't the same situation at all. With group 4 eliminated from the get go, the remaining teams can do even better than $4 or $3. Teammates in room A2 knows for a fact that the coin landed heads, and they automatically earn $1. Teammates in room A1 are no longer responsible for their teammates' decisions, so they go for the $3. Thus teams 1 and 2 both take home $1 while team 3 takes home $3, for a total of $5.
Maybe that's the difference. Even if you know for a fact that you aren't on team 4, you also aren't in a world where team 4 was eliminated from the start. The team still needs to factor into your calculations... somehow. Maybe it means your teammate isn't really making the same decision you are? But it's perfectly symmetrical information. Maybe you don't get to eliminate team 4 unless your teammate does? But the proof is right in front of you. Maybe the information isn't symmetrical because your teammate could be in room B?
I don't know. I feel like there's an answer in here somewhere, but I've spent several hours on this post and I have other things to do today.
I'm going to rephrase this using as many integers as possible because humans are better at reasoning about those. I know I personally am.
Instead of randomness, we have four teams that perform this experiment. Teams 1 and 2 represent the first flip landing on heads. Team 3 is tails then heads, and team 4 is tails then tails. No one knows which team they've been assigned to.
Also, instead of earning $1 or $3 for both participants, a correct guess earns that same amount once. They still share finances so this shouldn't affect anyone's reasoning; I just don't want to have to double it.
Team 1 makes 2 guesses. Each "heads" guess earns $1, each "tails" guess earns nothing.
Team 2 makes 2 guesses. Each "heads" guess earns $1, each "tails" guess earns nothing.
Team 3 makes 1 guess. Guessing "heads" earns nothing, guessing "tails" earns $3.
Team 4 makes 1 guess. Guessing "heads" earns nothing, guessing "tails" earns $3.
If absolutely everyone guesses "heads," teams 1 and 2 will earn $4 between them. If absolutely everyone guesses "tails," teams 3 and 4 will earn $6 between them. So far, this matches up.
Now let's look at how many people were sent to each room.
Three people visit room A1: one from team 1, one from team 2, and one from team 3. 2/3 of them are there because the first "flip" was heads.
Three people visit room A2: one from team 1, one from team 2, and one from team 4. 2/3 of them are there because the first "flip" was heads.
Two people visit room B: one from team 3 and one from team 4. They don't matter.
The three visitors to A1 know they aren't on team 4, thus they can subtract that team's entire winnings from their calculations, leaving $4 vs. $3.
The three visitors to A2 know they aren't on team 3, thus they can subtract that team's entire winnings from their calculations, leaving $4 vs. $3.
Do you see the error? Took me a bit.
If you're in room A1, you need to subtract more than just team 4's winnings. You need to subtract half of team 1 and team 2's winnings. Teams 1 and 2 each have someone in room A2, and you can't control their vote. Thus:
Three people visit room A1: one from team 1, one from team 2, and one from team 3. If all three guess "heads" they earn $2 in all. If all three guess "tails" they earn $3 in all.
Three people visit room A1: one from team 1, one from team 2, and one from team 4. If all three guess "heads" they earn $2 in all. If all three guess "tails" they earn $3 in all.
Guessing "tails" remains the best way to maximize expected value.
---
The lesson here isn't so much to do with EDT agents, it's to do with humans and probabilities. I didn't write this post because I'm amazing and you're a bad math student, I wrote this post because without it, I wouldn't have been able to figure it out either.
Whenever this sort of thing comes up, try to rephrase the problem. Instead of 85%, imagine 100 people in a room, with 85 on the left and 15 on the right. Instead of truly random experiments, imagine the many-worlds interpretation, where each outcome is guaranteed to come up in a different branch. (And try to have an integer number of branches, each representing an equal fraction.) Or use multiple teams like I did above.
It is a stretch, which is why it needed to be explained.
And yes, it would kind of make him immune to dying... in cases where he could be accidentally rescued. Cases like a first year student's spell locking a door, which an investigator could easily dispel when trying to investigate.
Oh, and I guess once it was established, the other time travel scenes would have had to be written differently. Or at least clarify that "while Draco's murder plot was flimsy enough that the simplest timeline was the timeline in which it failed, Quirrel's murder plot was bulletproof enough that the simplest outcome was for it to succeed." Because authors write the rules, they can get away with a lot of nonsense. But in this kind of story, they do need to acknowledge and (try to) explain any inconsistencies.
And here's the line I was referring to:
"The earlier experiment had measured whether Transfiguring a long diamond rod into a shorter diamond rod would allow it to lift a suspended heavy weight as it contracted, i.e., could you Transfigure against tension, which you in fact could." (Chapter 28, foreshadowing the nanotube, which may or may not have been what you were talking about)
I don't mind the occasional protagonist who makes their own trouble. I agree it would be annoying if all protagonists were like that (and I agree that Harry is annoying in general), there's room in the world for stories like this.
Now that you mention it, your first example does sound like a Deus Ex Machina. Except that
the story already established that the simplest possible time loop is preferred, and it's entirely possible that if Harry hadn't gotten out to pass a note, someone would have gone back in time to investigate his death, and inadvertently caused a paradox by unlocking the door.
This wouldn't have had to be a long explanation or full-blown lecture, just enough to confirm this interpretation. But since it wasn't confirmed and there are multiple valid interpretations of the mechanics, it does come across as a bit of an "I got out of jail free" moment.
I... don't understand your second example. I think that part of the story works just fine. Harry's solution was plausible, and even foreshadowed
in chapter 28 when he used transfiguration to apply force.
It's been a while since I read it, but off the top of my head I can't recall any blatant cases of Deus ex Machina. I'd ask for concrete examples, but I don't think it would be useful. I'm sure you can provide an example, and in turn I'll point out reasons why it doesn't count as Deus ex Machina. We'd argue about how well the solution was explained, and whether enough clues were presented far enough in advance to count as valid foreshadowing, and ultimately it'll come down to opinion.
Instead, I can go ahead and answer your question. Eliezer definitely meant to teach useful lessons. Not everything Harry does is meant to be a good example (I mean, even Eliezer knows better than to write a completely perfect character), which is probably why he gets into all that trouble. But whenever a character goes into Lecture Mode while solving a problem, it's meant to be both useful and accurate.
Wait a minute, are you talking about Lecture Mode when you say "Deus ex Machina"? I can kind of see that: the situation seems hopeless and then someone (usually Harry) gives a long explanation and suddenly the problem is solved. Thing is, these lectures don't pull the solution out of nowhere. The relevant story details were established beforehand, and the lecture just puts them together. (Or at least, that was the author's intent. As I said, it comes down to opinion.)
An example I like is the Knight Capital Group trading incident. Here are the parts that I consider relevant:
KCG deployed new code to a production environment, and while I assume this code was thoroughly tested in a sandbox, one of the production servers had some legacy code ("Power Peg") that wasn't in the sandbox and therefore wasn't tested with the new code. These two pieces of code used the same flag for different purposes: the new code set the flag during routine trading, but Power Peg interpreted that flag as a signal to buy and sell ~10,000 arbitrary* stocks.
*Actually not arbitrary. What matters is that the legacy algorithm was optimized for something other than making money, so it lost money on average.
They stopped this code after 45 minutes, but by then it was too late. Power Peg had already placed millions of inadvisable orders, nearly bankrupting KCG.
Sometimes, corrigibility isn't enough.
I suppose, but even then he would have to take time to review the state of the puzzle. You would still expect him to take longer to spot complex details, and perhaps he'd examine a piece or two to refresh his memory.
But that isn't my true rejection here.
If you assume that Claude's brother "spent arbitrarily much time" beforehand, the moral of the story becomes significantly less helpful: "If you're having trouble, spend an arbitrarily large amount of time working on the problem."
His brother's hint contained information that he couldn't have gotten by giving the hint to himself. The fact that his brother said this while passing by means that he spotted a low-hanging fruit. If his brother had spent more time looking before giving the hint, this would have indicated a fruit that was a little higher up.
This advice is worth trying, but when you give it to yourself, you can't be sure that there's low hanging fruit left. If someone else gives it to you, you know it's worth looking for, because you know there's something there to find. (The difference is that they, not you, took the time to search for it.)
Again, it's a worthwhile suggestion. I just want to point out that it boils down to "If you're having trouble, check for easier solutions," and that while you can always give this advice to yourself, it will not always help.
Harry left "a portion of his life" (not an exact quote) in Azkaban, and apparently it will remain there forever. That could be the remnant that Death would fail to destroy.
Anyway, Snape drew attention to the final line in the prophecy. It talked about two different spirits that couldn't exist in the same world, or perhaps two ingredients that cannot exist in the same cauldron. That's not Harry and Voldemort; that's Harry and Death.
I mean, Harry has already sworn to put an end to death. It's how he casts his patronus. He's a lot less sure about killing Voldemort, and would prefer not to, if given the choice.
On the other hand, MIRI hit its goal three weeks early, so the amount of support is pretty obvious.
Though I have to admit, I was going to remain silent too, and upon reflection I couldn't think of any good reasons to do so. It may not be necessary, but it couldn't hurt either. So...
I donated $700 to CFAR.
So that's how Omega got the money for box B!
Well designed traditions and protocols will contain elements that cause most competent people to not want to throw them out.
Having just listened to much of the Ethical Injunctions sequence (as a podcast courtesy of George Thomas), I'm not so sure about this one. There are reasons for serious, competent people to follow ethical rules, even when they need to get things done in the real world.
Ethics aren't quite the same as tradition and protocol, but even so, sometimes all three of those things exist for good reasons.
Agreed.
Though actually, Eliezer used similar phrasing regarding Richard Loosemore and got downvoted for it (not just by me). Admittedly, "persistent troll" is less extreme than "permanent idiot," but even so, the statement could be phrased to be more useful.
I'd suggest, "We've presented similar arguments to [person] already, and [he or she] remained unconvinced. Ponder carefully before deciding to spend much time arguing with [him or her]."
Not only is it less offensive this way, it does a better job of explaining itself. (Note: the "ponder carefully" section is quoting Eliezer; that part of his post was fine.)
You and I are both bound by the terms of a scenario that someone else has set here.
Ok, if you want to pass the buck, I won't stop you. But this other person's scenario still has a faulty premise. I'll take it up with them if you like; just point out where they state that the goal code starts out working correctly.
To summarize my complaint, it's not very useful to discuss an AI with a "sincere" goal of X, because the difficulty comes from giving the AI that goal in the first place.
What you did was consider some other possibilities, such as those in which the AI is actually not being sincere. Nothing wrong with considering those, but that would be a story for another day.
As I see it, your (adopted) scenario is far less likely than other scenario(s), so in a sense that one is the "story for another day." Specifically, a day when we've solved the "sincere goal" issue.
I didn't mean to ignore your argument; I just didn't get around to it. As I said, there were a lot of things I wanted to respond to. (In fact, this post was going to be longer, but I decided to focus on your primary argument.)
Your story:
This hypothetical AI will say “I have a goal, and my goal is to get a certain class of results, X, in the real world.” [...] And we say “Hey, no problem: looks like your goal code is totally consistent with that verbal description of the desired class of results.” Everything is swell up to this point.
My version:
The AI is lying. Or possibly it isn't very smart yet, so it's bad at describing its goal. Or it's oversimplifying, because the programmers told it to, because otherwise the goal description would take days. And the goal code itself is too complicated for the programmers to fully understand. In any case, everything is not swell.
Your story:
Then one day the AI says “Okay now, today my goalX code says I should do this…” and it describes an action that is VIOLENTLY inconsistent with the previously described class of results, X. This action violates every one of the features of the class that were previously given.
My version:
The AI's goal was never really X. It was actually Z. The AI's actions perfectly coincide with Z.
In the rest of the scenario you described, I agree that the AI's behavior is pretty incoherent, if its goal is X. But if it's really aiming for Z, then its behavior is perfectly, terrifyingly coherent.
And your "obvious" fail-safe isn't going to help. The AI is smarter than us. If it wants Z, and a fail-safe prevents it from getting Z, it will find a way around that fail-safe.
I know, your premise is that X really is the AI's true goal. But that's my sticking point.
Making it actually have the goal X, before it starts self-modifying, is far from easy. You can't just skip over that step and assume it as your premise.
Oh, yeah, I found that myself eventually.
Anyway, I went and read the the majority of that discussion (well, the parts between Richard and Rob). Here's my summary:
Richard:
I think that what is happening in this discussion [...] is a misunderstanding. [...]
[Rob responds]
Richard:
You completely miss the point that I was trying to make. [...]
[Rob responds]
Richard:
You are talking around the issue I raised. [...] There is a gigantic elephant in the middle of this room, but your back is turned to it. [...]
[Rob responds]
Richard:
[...] But each time I explain my real complaint, you ignore it and respond as if I did not say anything about that issue. Can you address my particular complaint, and not that other distraction?
[Rob responds]
Richard:
[...] So far, nobody (neither Rob nor anyone else at LW or elsewhere) will actually answer that question. [...]
[Rob responds]
Richard:
Once again, I am staggered and astonished by the resilience with which you avoid talking about the core issue, and instead return to the red herring that I keep trying to steer you away from. [...]
Rob:
Alright. You say I’ve been dancing around your “core” point. I think I’ve addressed your concerns quite directly, [...] To prevent yet another suggestion that I haven’t addressed the “core”, I’ll respond to everything you wrote above. [...]
Richard:
Rob, it happened again. [...]
I snipped a lot of things there. I found lots of other points I wanted to emphasize, and plenty of things I wanted to argue against. But those aren't the point.
Richard, this next part is directed at you.
You know what I didn't find?
I didn't find any posts where you made a particular effort to address the core of Rob's argument. It was always about your argument. Rob was always the one missing the point.
Sure, it took Rob long enough to focus on finding the core of your position, but he got there eventually. And what happened next? You declared that he was still missing the point, posted a condensed version of the same argument, and posted here that your position "withstands all the attacks against it."
You didn't even wait for him to respond. You certainly didn't quote him and respond to the things he said. You gave no obvious indication that you were taking his arguments seriously.
As far as I'm concerned, this is a cardinal sin.
I think I am explaining the point with such long explanations that I am causing you to miss the point.
How about this alternate hypothesis? Your explanations are fine. Rob understands what you're saying. He just doesn't agree.
Perhaps you need to take a break from repeating yourself and make sure you understand Rob's argument.
(P.S. Eliezer's ad hominem is still wrong. You may be making a mistake, but I'm confident you can fix it, the tone of this post notwithstanding.)
Link to the nailed-down version of the argument?
I posted elsewhere that this post made me think you're anthropomorphizing; here's my attempt to explain why.
egregiously incoherent behavior in ONE domain (e.g., the Dopamine Drip scenario)
the craziness of its own behavior (vis-a-vis the Dopamine Drip idea)
if an AI cannot even understand that "Make humans happy" implies that humans get some say in the matter
Ok, so let's say the AI can parse natural language, and we tell it, "Make humans happy." What happens? Well, it parses the instruction and decides to implement a Dopamine Drip setup.
As FeepingCreature pointed out, that solution would in fact make people happy; it's hardly inconsistent or crazy. The AI could certainly predict that people wouldn't approve, but it would still go ahead. To paraphrase the article, the AI simply doesn't care about your quibbles and concerns.
For instance:
people might consider happiness to be something that they do not actually want too much of
Yes, but the AI was told, "make humans happy." Not, "give humans what they actually want."
people might be allowed to be uncertain or changeable in their attitude to happiness
Yes, but the AI was told, "make humans happy." Not, "allow humans to figure things out for themselves."
subtleties implicit in that massive fraction of human literature that is devoted to the contradictions buried in our notions of human happiness
Yes, but blah blah blah.
Actually, that last one makes a point that you probably should have focused on more. Let's reconfigure the AI in light of this.
The revised AI doesn't just have natural language parsing; it's read all available literature and constructed for itself a detailed and hopefully accurate picture of what people tend to mean by words (especially words like "happy"). And as a bonus, it's done this without turning the Earth into computronium!
This certainly seems better than the "literal genie" version. And this time we'll be clever enough to tell it, "give humans what they actually want." What does this version do?
My answer: who knows? We've given it a deliberately vague goal statement (even more vague than the last one), we've given it lots of admittedly contradictory literature, and we've given it plenty of time to self-modify before giving it the goal of self-modifying to be Friendly.
Maybe it'll still go for the Dopamine Drip scenario, only for more subtle reasons. Maybe it's removed the code that makes it follow commands, so the only thing it does is add the quote "give humans what they actually want" to its literature database.
As I said, who knows?
Now to wrap up:
You say things like "'Make humans happy' implies that..." and "subtleties implicit in..." You seem to think these implications are simple, but they really aren't. They really, really aren't.
This is why I say you're anthropomorphizing. You're not actually considering the full details of these "obvious" implications. You're just putting yourself in the AI's place, asking yourself what you would do, and then assuming that the AI would do the same.
I did see the insult, but Eliezer (quite rightly) got plenty of downvotes for it. I'm pretty sure that's not the reason you're being rated down.
I myself gave you a downvote because I got a strong impression that you were anthropomorphizing. Note that I did so before reading Eliezer's comment.
I certainly should have explained my reasons after voting, but I was busy and the downvote button seemed convenient. Sorry about that. I'll get started on a detailed response now.
I want to upvote this for the link to further discussion, but I also want to downvote it for the passive-aggressive jab at LW users.
No vote.
I've donated a second $1000.
I donated $1000 and then went and bought Facing the Intelligence Explosion for the bare minimum price. (Just wanted to put that out there.)
I've also left myself a reminder to consider another donation a few days before this runs out. It'll depend on my financial situation, but I should be able to manage it.
A podcast entry is included for that one, but it just directs you to read the original article.
I was going to link you one of the other podcasts (which all provide samples), but then I realized you might be asking why this specific podcast doesn't have one.
Ok, yeah, in that case my response is to take as many deals as Omega offers.
AdeleneDawner and gwern provide a way to make the idea more palatable - assume MWI. That is, assume there will be one "alive" branch and a bunch of "dead" branches. That way, your utility payoff is guaranteed. (Ignoring the grief of those around you in all the "dead" branches.)
Without that interpretation, the idea becomes scarier, but the math still comes down firmly on the side of accepting all the offers. It certainly feels like a bad idea to accept that probability of death, no matter what the math says, but as far as I can tell that's scope insensitivity talking.
With that in mind, my only remaining objection is the "we can do better than that" argument presented above. My feeling is, why not use a few of those 10^10,000,000,000 years to figure out a way to live even longer? Omega won't allow it? Ok, so I don't want to get involved with Omega in the first place; it's not worth losing my (admittedly slim) chances at actual immortality. Too late for that? Fine, then I'll sit down, shut up, and multiply.
One-in-a-million is just an estimate. Immortality is a tough proposition, but the singularity might make it happen. The important part is that it isn't completely implausible.
I'm not sure what you mean, otherwise.
Are you suggesting that Omega takes away any chance of achieving immortality even before making the offer? In that case, Omega's a jerk, but I'll shut up and multiply.
Or are you saying that 10^10,000,000,000 years could be used for other high-utility projects, like making simulated universes full of generally happy people? Immortality would allow even more time for that.
Summary of this retracted post:
Omega isn't offering an extended lifespan; it's offering an 80% chance of guaranteed death plus a 20% chance of guaranteed death. Before this offer was made, actual immortality was on the table, with maybe a one-in-a-million chance.
When I was a Christian, and when I began this intense period of study which eventually led to my atheism, my goal, my one and only goal, was to find the best evidence and argument I could find that would lead people to the truth of Jesus Christ. That was a huge mistake. As a skeptic now, my goal is very similar - it just stops short. My goal is to find the best evidence and argument, period. Not the best evidence and argument that leads to a preconceived conclusion. The best evidence and argument, period, and go wherever the evidence leads.
Sorry I'm late... is no-one curious given the age of the universe why the 3 races are so close technologically?
Sorry I'm late in replying to this, but I'd guess the answer is that this is "the past's future." He would not have been able to tell this story with one species being that advanced, so he postulated a universe in which such a species doesn't exist (or at least isn't nearby).
Your in-universe explanations work as well, of course.
Perhaps the fact that it's the "traditional and irrational" ending is the reason Eliezer went with it as the "real" one. (Note that he didn't actually label them as "good" and "bad" endings.)
I assumed the same, based on the definition of "god" as "supernatural" and the definition of "supernatural" as "involving ontologically basic mental entities."
(Oh, and for anyone who hasn't read the relevant post, the survey is quoting this.)
I could be interpreting it entirely wrong, but I'd guess this is the list Cochran had in mind:
•
This reminds me of Lojban, in which the constructs meaning "good" and "bad" encourage you to specify a metric. It is still possible to say that something is "worse" without providing any detail, but I suspect most Lojban speakers would remember to provide detail if there was a chance of confusion.
Give the furries, vampire-lovers and other assorted xenophiles a few generations to chase their dreams, and you're going to start seeing groups with distinctly non-human psychology.
WHY HAVEN'T I READ THIS STORY?
Because you haven't had time to read all the Orion's Arm stories, probably. (Details)
2) Honestly, I would have been happy with the aliens' deal (even before it was implemented), and I think there is a ~60% chance that Elizier agrees.
I'm of the opinion that pain is a bad thing, except insofar as it prevents you from damaging yourself. People argue that pain is necessary to provide contrast to happiness, and that pleasure wouldn't be meaningful without pain, but I would say that boredom and slight discomfort provide more than enough contrast.
However, this future society disagrees. The idea that "pain is important" is ingrained in these people's minds, in much the same way that "rape is bad" is ingrained in ours. I think one of the main points Elizier is trying to make is that we would disagree with future humans almost as much as we would disagree with the baby-eaters or superhappies.
(Edit 1.5 years later: I was exaggerating in that second paragraph. I suspect I was trying too hard to sound insightful. The claims may or may not have merit, but I would no longer word them as forcefully.)
If you're trying to present any kind of information at all, you should figure out what is important about it and what presentation will make it clear.
Unfortunately, the quote above isn't at all clear, even in context. I suspect this is because Jacques Bertin isn't as good at expressing himself in English as in French, but even so I'm unable to understand the sample data he presents or how it relates to the point he was trying to make.
I agree with your interpretation of the song, and to back it up, here's the chorus of "The Farthest Star" (another song by the same band).
We possess the power, if this should start to fall apart, To mend divides, to change the world, to reach the farthest star. If we should stay silent, if fear should win our hearts, Our light will have long diminished before it reaches the farthest star.
This time, the message is pretty clear: we should aspire to overcome both our differences and our limitations, to avoid extinction and to expand throughout the universe. I suppose it doesn't say anything about immortality, but otherwise it seems to match transhumanist philosophy.
If the problem here is that the entity being simulated ceases to exist, an alternative solution would be to move the entity into an ongoing simulation that won't be terminated. Clearly, this would require an ever-increasing number of resources as the number of simulations increased, but perhaps that would be a good thing - the AI's finite ability to support conscious entities would impose an upper bound on the number of simulations it would run. If it was important to be able to run such a simulation, it could, but it wouldn't do so frivolously.
Before you say anything, I don't actually think the above is a good solution. It's more like a constraint to be routed around than a goal to be achieved. Plus, it's far too situational and probably wouldn't produce desirable results in situations we didn't design it for.
The thing is, it isn't important to come up with the correct solution(s) ourselves. We should instead make the AI understand the problem and want to solve it. We need to carefully design the AI's utility function, making it treat conscious entities in simulations as normal people and respect simulated lives as it respects ours. We will of course need a highly precise definition for consciousness that applies not just to modern-day humans but also to entities the AI could simulate.
Here's how I see it - as long as the AI values the life of any entity that values its own life, the AI will find ways to keep those entities alive. As long as it considers entities in simulations to be equivalent to entities outside, it will avoid terminating their simulation and killing them. The AI (probably) would still make predictions using simulations; it would just avoid the specific situation of destroying a conscious entity that wanted to continue living.
Daniel Oppenheimer's Ig Nobel Prize acceptance speech:
My research shows that conciseness is interpreted as intelligence. So, thank you.
Until then, I'd be more interested in donating to general life extension research than paying for cryonics specifically.
This is very similar to my primary objection to cryonics.
I realize that, all factors considered, the expected utility you'd get from signing up for cryonics is extremely large. Certainly large enough to be worth the price.
However, it seems to me that there are better alternatives. Sure, paying for cryonics increases your chances of nigh-immortality by orders of magnitude. On the other hand, funding longevity research makes it more likely that we will ever overcome aging and disease. Unlimited life for most or all of the future human population is far more important than unlimited life for yourself, right? (One might object that life extension research is already on its way to accomplishing this regardless of your contributions, which brings me to my next point.)
If an existential risk comes to pass, then no one will have a chance at an unlimited life. All of the time and money spent on cryonics will go to waste, and life extension research will have been (mostly) squandered. Preventing this sort of risk is therefore far more important than preserving any one person, even if that person is you. To make matters worse, there are multiple existential risks that have a significant chance of happening, so the need for extra attention and donations is much greater than the need for extra longevity research.
To summarize: Cryonics gives you alone a far bigger chance of nigh-immortality. Working to prevent existential risk gives billions of people a slightly increased chance of the same.
It seems to me we shouldn't be spending money on freezing vitrifying ourselves just in case a singularity (or equivalent scientific progress) happens. Instead, we should focus on increasing the chances that it will happen at all. To do anything else would be selfish.
Ok, time to take a step back and look at some reasons I might be wrong.
First, and perhaps most obviously, people are not inclined to donate all their money to any cause, no matter how important. I freely admit that I will probably donate only a small fraction of my earnings, despite the arguments I made in this post. Plus, it's possible (likely?) that people would be more inclined to spend money on cryonics than on existential risk reduction, because cryonics benefits them directly. If someone is going to spend money selfishly, I suppose cryonics is the most beneficial way to do so.
Second, there's a chance I misestimated the probabilities involved, and in fact your money would be best spent on cryonics. If the Cryonics Institute webpage is to be believed, the cheapest option costs $28,000, which is generally covered by insurance, costing you $120 per year (this option also requires a one-time payment of $1,250). Unfortunately, I have no idea how much $1,250 plus $120 per year would help if donated to SIAI or another such organization. Cryonics certainly give a huge expected reward, and I'm just guessing at the expected reward for donating.
Thanks for the analysis, MathijsJ! It made perfect sense and resolved most of my objections to the article.
I was willing to accept that we cannot reach absolute certainty by accumulating evidence, but I also came up with multiple logical statements that undeniably seemed to have probability 1. Reading your post, I realized that my examples were all tautologies, and that your suggestion to allow certainty only for tautologies resolved the discrepancy.
The Wikipedia article timtyler linked to seems to support this: "Cromwell's rule [...] states that one should avoid using prior probabilities of 0 or 1, except when applied to statements that are logically true or false." This matches your analysis - you can only be certain of tautologies.
Also, your discussion of models neatly resolves the distinction between, say, a mathematically-defined die (which can be certain to end up showing an integer between 1 and 6) and a real-world die (which cannot quite be known for sure to have exactly six stable states).
Eliezer makes his position pretty clear: "So I propose that it makes sense to say that 1 and 0 are not in the probabilities; just as negative and positive infinity, which do not obey the field axioms, are not in the real numbers."
It's true - you cannot ever reach a probability of 1 if you start at 0.5 and accumulate evidence, just as you cannot reach infinity if you start at 0 and add integer values. And the inverse is true, too - you cannot accumulate evidence against a tautology and bring its probability down to anything less than 1. But this doesn't mean a probability of 1 is an incoherent concept or anything.
Eliezer: if you're going to say that 0 and 1 are not probabilities, you need to come up with a new term for them. They haven't gone away completely just because we can't reach them.
Edit a year and a half later: I agree with the article as written, partially as a result of reading How to Convince Me That 2 + 2 = 3, and partially as a result of concluding that "tautologies that have probability 1 but no bearing on reality" is a useless concept, and that therefore, "probability 1" is a useless concept.