Posts

New Pascal's Mugging idea for potential solution 2016-08-04T20:38:54.341Z
How realistic would AI-engineered chatbots be? 2014-09-11T23:00:01.411Z
Be Wary of Thinking Like a FAI 2014-07-18T20:22:00.460Z
Just for fun: Computer game to illustrate AI takeover concepts? 2014-07-03T19:30:26.241Z
Questions to ask theist philosophers? I will soon be speaking with several 2014-04-26T00:46:37.806Z
Rationality & Low-IQ People 2014-02-02T15:11:59.505Z

Comments

Comment by kokotajlod on Blog Post Day II · 2020-03-22T14:23:33.832Z · LW · GW

Sweet. I too will write something not about coronavirus.

Comment by kokotajlod on Extortion and trade negotiations · 2016-12-23T18:45:00.179Z · LW · GW

"It's a badly formulated question, likely to lead to confusion." Why? That's precisely what I'm denying.

"So, can you specify what this cluster is? Can you list the criteria by which a behaviour would be included in or excluded from this cluster? If you do this, you have defined blackmail."

That's precisely what I (Stuart really) am trying to do! I said so, you even quoted me saying so, and as I interpret him, Stuart said so too in the OP. I don't care about the word blackmail except as a means to an end; I'm trying to come up with criteria by which to separate the bad behaviors from the good.

I'm honestly baffled at this whole conversation. What Stuart is doing seems the opposite of confused to me.

Comment by kokotajlod on Open Problems Related to Solomonoff Induction · 2016-12-23T18:36:06.403Z · LW · GW

Dunno what Username was thinking, but here's the answer I had in mind: "Why is it obvious? Because the Problem of Induction has not yet been solved."

Comment by kokotajlod on Extortion and trade negotiations · 2016-12-21T15:23:51.225Z · LW · GW

You make it sound like those two things are mutually exclusive. They aren't. We are trying to define words so that we can understand and manipulate behavior.

"I don't know what blackmail is, but I want to make sure an AI doesn't do it." Yes, exactly, as long as you interpret it in the way I explained it above.* What's wrong with that? Isn't that exactly what the AI safety project is, in general? "I don't know what bad behaviors are, but I want to make sure the AI doesn't do them."

*"In other words there are a cluster of behaviors that we do NOT want our AI to have, which seem blackmailish to us, and a cluster of behaviors that we DO want it to have, which seem tradeish to us. So we are now trying to draw a line in conceptual space between them so that we can figure out how to program an AI appropriately."

Comment by kokotajlod on Extortion and trade negotiations · 2016-12-20T14:43:57.090Z · LW · GW

"You want to understand and prevent some behaviors (in which case, start by tabooing culturally-dense words like "blackmail")"

In a sense, that's exactly what Stuart was doing all along. The whole point of this post was to come up with a rigorous definition of blackmail, i.e. to find a way to say what we wanted to say without using the word.

Comment by kokotajlod on Extortion and trade negotiations · 2016-12-19T16:31:22.779Z · LW · GW

As I understand it, the idea is that we want to design an AI that is difficult or impossible to blackmail, but which makes a good trading partner.

In other words there are a cluster of behaviors that we do NOT want our AI to have, which seem blackmailish to us, and a cluster of behaviors that we DO want it to have, which seem tradeish to us. So we are now trying to draw a line in conceptual space between them so that we can figure out how to program an AI appropriately.

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-10-29T02:09:42.529Z · LW · GW

OH ok I get it now: "But clearly re-arranging terms doesn't change the expected utility, since that's just the sum of all terms." That's what I guess I have to deny. Or rather, I accept that (I agree that EU = infinity for both A and B) but I think that since A is better than B in every possible world, it's better than B simpliciter.

The reshuffling example you give is an example where A is not better than B in every possible world. That's the sort of example that I claim is not realistic, i.e. not the actual situation we find ourselves in. Why? Well, that was what I tried to argue in the OP--that in the actual situation we find ourselves in, the action A that is best in the simplest hypothesis is also better.... well, oops, I guess it's not better in every possible world, but it's better in every possible finite set of possible worlds such that the set contains all the worlds simpler than its simplest member.

I'm guessing this won't be too helpful to you since, obviously, you already read the OP. But in that case I'm not sure what else to say. Let me know if you are still interested and I"ll try to rephrase things.

Sorry for taking so long to get back to you; I check this forum infrequently.

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-18T15:24:22.986Z · LW · GW

Again, thanks for this.

"The problem with your solution is that it's not complete in the formal sense: you can only say some things are better than other things if they strictly dominate them, but if neither strictly dominates the other you can't say anything."

As I said earlier, my solution is an argument that in every case there will be an action that strictly dominates all the others. (Or, weaker: that within the set of all hypotheses of probability less than some finite N, one action will strictly dominate all the others, and that this action will be the same action that is optimal in the most probable hypothesis.) I don't know if my argument is sound yet, but if it is, it avoids your objection, no?

I'd love to understand what you said about re-arranging terms, but I don't. Can you explain in more detail how you get from the first set of hypotheses/choices (which I understand) to the second?

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-10T17:37:50.405Z · LW · GW

This was helpful, thanks!

As I understand it, you are proposing modifying the example so that on some H1 through HN, choosing A gives you less utility than choosing B, but then thereafter choosing B is better, because there is some cost you pay which is the same in each world.

It seems like the math tells us that any price would be worth it, that we should give up an unbounded amount of utility to choose A over B. I agree that this seems like the wrong answer. So I don't think whatever I'm proposing solves this problem.

But that's a different problem than the one I'm considering. (In the problem I'm considering, choosing A is better in every possible world.) Can you think of a way they might be parallel--any way that the "I give up" which I just said above applies to the problem I'm considering too?

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-06T14:17:31.786Z · LW · GW

It's arbitrary, but that's OK in this context. If I can establish that this works when the ratio is 1 in a billion, or lower, then that's something, even if it doesn't work when the ratio is 1 in 10.

Especially since the whole point is to figure out what happens when all these numbers go to extremes--when the scenarios are extremely improbable, when the payoffs are extremely huge, etc. The cases where the probabilities are 1 in 10 (or arguably even 1 in a billion) are irrelevant.

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-05T17:03:22.766Z · LW · GW

Update: The conclusion of that article is that the expected utilities don't converge for any utility function that is bounded below by a computable, unbounded utility function. That might not actually be in conflict with the idea I'm grasping at here.

The idea I'm trying to get at here is that maybe even if EU doesn't converge in the sense of assigning a definite finite value to each action, maybe it nevertheless ranks each action as better or worse than the others, by a certain proportion.

Toy model:

The only hypotheses you consider are H1, H2, H3, ... etc. You assign 0.5 probability to H1, and each HN+1 has half the probability of the previous hypothesis, HN.

There are only two possible actions: A or B. H1 says that A gives you 2 utils and B gives you 1. Each HN+1 says that A gives you 10 times as many utils as it did under the previous hypothesis, HN, and moreover that B gives you 5 times as many utils as it did under the previous hypothesis, HN.

In this toy model, expected utilities do not converge, but rather diverge to infinity, for both A and B.

Yet clearly A is better than B...

I suppose one could argue that the expected utility of both A and B is infinite and thus that we don't have a good reason to prefer A to B. But that seems like a problem with our ability to handle infinity, rather than a problem with our utility function or hypothesis space.

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-05T16:55:18.433Z · LW · GW

True. So maybe this only works in the long run, once we have more than 30 bits to work with.

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-05T13:07:06.360Z · LW · GW

Yes, but I don't think that's relevant. Any use of complexity depends on the language you specify it in. If you object to what I've said here on those grounds, you have to throw out Solomonoff, Kolmogorov, etc.

Comment by kokotajlod on New Pascal's Mugging idea for potential solution · 2016-08-05T13:05:26.440Z · LW · GW

Yes, I've read it, but not at the level of detail where I can engage with it. Since it is costly for me to learn the math necessary to figure this out for good, I figured I'd put the basic idea up for discussion first just in case there was something obvious I overlooked.

Edit: OK, now I think I understand it well enough to say how it interacts with what I've been thinking. See my other comment .

Comment by kokotajlod on The map of p-zombies · 2016-08-04T20:14:27.544Z · LW · GW

I disagree with your characterization of 0. You say that it is incompatible with physicalism, but that seems false. Indeed it seems to be a very mainstream physicalist view to say "I am a physical object--my brain. So a copy of me would have the same experiences, but it would not be me."

Comment by kokotajlod on Experiment: Changing minds vs. preaching to the choir · 2015-10-11T00:37:36.623Z · LW · GW

How do I do step II? I can't seem to find the relevant debates. I found one debate with the same title as the minimum wage one I argued about, but I don't see my argument appearing there.

Comment by kokotajlod on Steelmaning AI risk critiques · 2015-07-26T20:35:56.555Z · LW · GW

Yep. "The melancholy of haruhi suzumiya" can be thought of as an example of something in the same reference class.

Comment by kokotajlod on Counterfactual trade · 2015-06-05T19:56:53.534Z · LW · GW

This is an interesting idea! Some thoughts:

Doesn't acausal trade (like all trade) depend on enforcement mechanisms? I can see how two AI's might engage in counterfactual trade, since they can simulate each other and see that they self-modify to uphold the agreement, but I don't think a human would be able to do it.

Also, I'd like to hear more about motivations for engaging in counterfactual trade. I get the multiverse one, though I think that's a straightforward case of acausal trade rather than a case of counterfactual trade, since you would be trading with a really existing entity in another universe. But can you explain the second motivation more?

Comment by kokotajlod on Simulation argument meets decision theory · 2014-09-30T10:48:03.114Z · LW · GW

The point you raise is by far the strongest argument I know of against the idea.

However, it is a moral objection rather than a decision-theory objection. It sounds like you agree with me on the decision theory component of the idea: that if we were anthropically selfish, it would be rational for us to commit to making ancestor-simulations with afterlives. That's an interesting result in itself, isn't it? Let's go tell Ayn Rand.

When it comes to the morality of the idea, I might end up agreeing with you. We'll see. I think there are several minor considerations in favor of the proposal, and then this one massive consideration against it. Perhaps I'll make a post on it soon.

Comment by kokotajlod on Simulation argument meets decision theory · 2014-09-28T22:34:03.471Z · LW · GW

This is a formal version of a real-life problem I've been thinking about lately.

Should we commit to creating ancestor-simulations in the future, where those ancestor-simulations will be granted a pleasant afterlife upon what appears to their neighbors to be death? If we do, then arguably we increase the likelihood that we ourselves have a pleasant afterlife to look forward to.

Comment by kokotajlod on How realistic would AI-engineered chatbots be? · 2014-09-14T15:47:59.495Z · LW · GW

Thanks for the response. Yes, it depends on how much interaction I have with human beings and on the kind of people I interact with. I'm mostly interested in my own case, of course, and I interact with a fair number of fairly diverse, fairly intelligent human beings on a regular basis.

If you're a social butterfly who regularly talks with some of the smartest people in the world, the AI will probably struggle

Ah, but would it? I'm not so sure, that's why I made this post.

Yes, if everyone always said what I predicted, things would be obvious, but recall I specified that random variation would be added. This appears to be how dream characters work: You can carry on sophisticated conversations with them, but (probably) they are governed by algorithms that feed off your own expectations. That being said, I now realize that the variation would have to be better than random in order to account for how e.g. EY consistently says things that are on-point and insightful despite being surprising to me.

Comment by kokotajlod on How realistic would AI-engineered chatbots be? · 2014-09-13T12:40:00.979Z · LW · GW

Yes, but I'm not sure there is a difference between an AI directly puppeting them, and an AI designing a chatbot to run as a subroutine to puppet them, at least if the AI is willing to monitor the chatbot and change it as necessary. Do you think there is?

Also, it totally is a fruitful line of thinking. It is better to believe the awful truth than a horrible lie. At least according to my values. Besides, we haven't yet established that the truth would be awful in this case.

Comment by kokotajlod on How realistic would AI-engineered chatbots be? · 2014-09-12T23:43:44.279Z · LW · GW

I'm surprised that it sounded that way to you. I've amended my original post to clarify.

Comment by kokotajlod on How realistic would AI-engineered chatbots be? · 2014-09-12T23:24:17.562Z · LW · GW

Yes, this is the sort of consideration I had in mind. I'm glad the discussion is heading in this direction. Do you think the answer to my question hinges on those details though? I doubt it.

Perhaps if I was extraordinarily unsuspicious, chatbots of not much more sophistication than modern-day ones could convince me. But I think it is pretty clear that we will need more sophisticated chatbots to convince most people.

My question is, how much more sophisticated would they need to be? Specifically, would they need to be so much more sophisticated that they would be conscious on a comparable level to me, and/or would require comparable processing power to just simulating another person? For example, I've interacted a ton with my friends and family, and built up detailed mental models of their minds. Could they be chatbots/npcs, with minds that are nothing like the models I've made?

(Another idea: What if they are exactly like the models I've made? What if the chatbot works by detecting what I expect someone to say, and then having them say that, with a bit of random variation thrown in?)

Comment by kokotajlod on How realistic would AI-engineered chatbots be? · 2014-09-12T10:45:29.645Z · LW · GW

That's exactly what I had in mind, although I did specify that the controller would never simulate anybody besides me to the level required to make them people.

Comment by kokotajlod on Be Wary of Thinking Like a FAI · 2014-07-24T17:45:38.117Z · LW · GW

In the picture you just drew, the ideal being is derived from a series of better beings, thus it is (trivially) easier to imagine a better being than to imagine an ideal being.

I see it differently: The ideal being maximizes all good qualities, whereas imperfect beings have differing levels of the various good qualities. Thus to compare a non-ideal being to an ideal being, we only need to recognize how the ideal being does better than the non-ideal being in each good quality. But to compare two non-ideal beings, we need to evaluate trade-offs between their various attributes (unless one is strictly greater than the other)

Thinking about it more, I am not happy with either of the above models. One question that arises is: Does the same reasoning extend to other cases as well? i.e. are we better off thinking about incremental improvements than about the ideal society? Are we better off thinking about incremental improvements than about the ideal chess algorithm?

I think in some cases maybe we are, but in some cases we aren't--ideals are useful sometimes. I'd go farther to say that some aspects of many ideals must be arrived at by iterating, but other aspects can be concluded more directly. An uninteresting conclusion, but one that supports my overall point: I wasn't claiming that I knew everything about the ideal FAI, just that I had justified high confidence in some things.

Comment by kokotajlod on Be Wary of Thinking Like a FAI · 2014-07-21T15:50:25.438Z · LW · GW

Ok, thanks.

I also don't see any reason to go from "the FAI doesn't care about identity* to "I shouldn't think identity exists."

I don't either, now that I think about it. What motivated me to make this post is that I realized that I had been making that leap, thanks to applying the heuristic. We both agree the heuristic is bad.

Why are we talking about a bad heuristic? Well, my past self would have benefited from reading this post, so perhaps other people would as well. Also, I wanted to explore the space of applications of this heuristic, to see if I had been unconsciously applying it in other cases without realizing it. Talking with you has helped me with that.

Comment by kokotajlod on Be Wary of Thinking Like a FAI · 2014-07-20T12:37:12.624Z · LW · GW

Hmm, okay. I'd be interested to hear your thoughts on the particular cases then. Are there any examples that you would endorse?

Comment by kokotajlod on Be Wary of Thinking Like a FAI · 2014-07-20T12:35:41.406Z · LW · GW

The fact that your post was upvoted so much makes me take it seriously; I want to understand it better. Currently I see your post as merely a general skeptical worry. Sure, maybe we should never be very confident in our FAI-predictions, but to the extent that we are confident, we can allow that confidence to influence our other beliefs and decisions, and we should be confident in some things to some extent at least (the alternative, complete and paralyzing skepticism, is absurd) Could you explain more what you meant, or explain what you think my mistake is in the above reasoning?

Comment by kokotajlod on Be Wary of Thinking Like a FAI · 2014-07-20T12:24:41.915Z · LW · GW

It is easier to determine whether you are doing "better" than your current self than it is to determine how well you line up with a perceived ideal being.

Really? That doesn't seem obvious to me. Could you justify that claim?

Comment by kokotajlod on Just for fun: Computer game to illustrate AI takeover concepts? · 2014-07-09T03:08:54.138Z · LW · GW

Thanks!

But if the UFAI can't parlay that takes out much of the fun, and much of the realism too.

Also, if Hard Mode has no FAI tech at all, then no one will research AI on Hard Mode and it will just devolve into a normal strategy game.

Edit: You know, this proposal could probably be easily implemented as a mod for an existing RTS or 4X game. For example, imagine a Civilization mod that added the "AI" tech that allowed you to build a "Boxed AI" structure in your cities. This quadruples the science and espionage production of your city, at the cost of a small chance of the entire city going rogue (the AI unboxing) every turn. This as you said creates a new faction with all the technologies researched and world domination as its goal... You can also research "Friendly AI" tech that allows you to build a "Friendly AI" which is just like a rogue AI faction except that it is permanently allied to you and will obey your commands and instantly grants you all the tech you want.

Comment by kokotajlod on [LINK] Claustrum Stimulation Temporarily Turns Off Consciousness in an otherwise Awake Patient · 2014-07-06T13:58:21.998Z · LW · GW

There are at least two distinct senses in which consciousness can be binary. The first sense is the kind you are probably thinking about: the range between e.g. insects, dogs, and humans, or maybe between early and late-stage Alzheimers.

The second sense is the kind that your interlocutors are (I surmise) thinking about. Imagine this: A being that is functionally exactly like you, and that is experiencing exactly what you are experiencing, except that it is experiencing everything "only half as much." It still behaves the same way as you, and it still thinks the same way as you; it's just that it's thoughts only count half.

If this sounds ridiculous to you, well, then you agree with your interlocutors. :) Personally, I think that there IS such a thing as partial consciousness in the sense described above, and I can link you to literature if you like.

EDIT: The place to start is Nick Bostrom's "Quantity of Experience: Brain Duplication and Degrees of Consciousness," available for free online.

EDIT: But the people who ask "Do animals have consciousness" are probably talking about the first kind, in which case I share your frustration. The second kind is more what people talk about when they ask e.g. "Could a machine have consciousness?"

Comment by kokotajlod on [LINK] Claustrum Stimulation Temporarily Turns Off Consciousness in an otherwise Awake Patient · 2014-07-06T13:52:20.613Z · LW · GW

Is the claustrum located in the pineal gland? ;)

Comment by kokotajlod on Just for fun: Computer game to illustrate AI takeover concepts? · 2014-07-04T13:18:18.355Z · LW · GW

At the moment, in order for a Creator/FAI team to win (assuming you're sticking with Diplomacy mechanics) they first >have to collect 18 supply centres between them and then have the AI transfer all its control back to the human; I don't >think even the friendliest of AIs would willingly rebox itself like that.

This is exactly what I had in mind. :) It should be harder for FAI to win than for UFAI to win, since FAI are more constrained. I think it is quite plausible that one of the safety measures people would try to implement in a FAI is "Whatever else you do, don't kill us all; keep us alive and give us control over you in the long run. No apocalypse-then-utopia for you! We don't trust you that much, and besides we are selfish." Hence the FAI having to protect the supply centers of the human, and give over its own supply centers to the human eventually.

Why wouldn't it give over its supply centers to the human? It has to do that to win! I don't think it will hurt it too much, since it can make sure all the enemies are thoroughly trounced before beginning to cede supply centers.

Comment by kokotajlod on Just for fun: Computer game to illustrate AI takeover concepts? · 2014-07-04T13:12:23.676Z · LW · GW

This would be the ideal. Like I said though, I don't think I'll be able to make it anytime soon, or (honestly) anytime ever.

But yeah, I'm trying to design it to be simple enough to play in-browser or as an app, perhaps even as a Facebook game or something. It doesn't need to have good graphics or a detailed physics simulator, for example: It is essentially a board game in a computer, like Diplomacy or Risk. (Though it is more complicated than any board game could be)

I think that the game, as currently designed, would be an excellent source of fictional evidence for the notions of AI risk and AI arms races. Those notions are pretty important. :)

Comment by kokotajlod on Artificial Utility Monsters as Effective Altruism · 2014-06-28T14:39:18.280Z · LW · GW

(utility monsters are awful, for some reason, even though by assumption they generate huge amounts of utility, oh dear!)

Utility monsters are awful, possibly for no reason whatsoever. That's OK. Value is complex. Some things are just bad, not because they entail any bad thing but just because they themselves are bad.

Comment by kokotajlod on Some alternatives to “Friendly AI” · 2014-06-16T13:47:05.661Z · LW · GW

That's not something the average person will think upon hearing the term, especially since "AGI" tends to connote something very intelligent. I don't think it is a strong reason not to use it.

Comment by kokotajlod on Dissolving the Thread of Personal Identity · 2014-05-25T15:44:08.901Z · LW · GW

It is nice to see people thinking about this stuff. Keep it up, and keep us posted!

Have you read the philosopher Derek Parfit? He is famous for arguing for pretty much exactly what you propose here, I think.

Doubt: Doesn’t this imply that anthropic probabilities depend on how big a boundary the mind draws around stuff it considers “I”? Self: Yes. Doubt: This seems to render probability useless.

I agree with Doubt. If can make it 100% probable that I'll get superpowers tomorrow merely by convincing myself that only superpowered future-versions of me count as me, then sign me up for surgical brainwashing today!

If you take altruism into account, then it all adds up to normality. Or rather, it can all be made to add up to normality, if we suitably modify our utility function. But that's true of ANY theory.

My question is, would you apply the same principle to personal-identity-right-now? Forget the future and the past, and just worry about the question "what am I right now?" Would you say that the answer to this question is also mind-dependent, such that if I decide to draw the reference class for the word "I" in such a way as to exclude brains in vats, then I have 0% probability of being a brain in a vat?

Comment by kokotajlod on Rebutting radical scientific skepticism · 2014-05-11T05:38:06.573Z · LW · GW

No, the analogy I had in mind was this:

What People Saw: Acupuncture* being correlated with health, and [building things according to theories developed using the scientific method] being correlated with [having things that work very well]

What People Thought Happened: Acupuncture causing health and [building things according to theories developed using the scientific method] causing [having things that work very well]

What Actually Happened: Placebo effect and Placebo effect (in the former case, involving whatever mechanisms we think cause the placebo effect these days; in the latter case, involving e.g. God.)

people were completely and totally wrong about beliefs affecting reality before

Filtering out all the selection bias etc., the relaxation and evolutionarily-based allocation of bodily resources seem to work fine for my purposes. They are analogous to theism-based allocation of technological power.

Comment by kokotajlod on Rebutting radical scientific skepticism · 2014-05-09T04:07:26.355Z · LW · GW

I didn't mean to imply that the placebo effect is a complete mystery. As you say, perhaps it is pretty well understood. But that doesn't touch my overall point which is that before modern medicine (and modern explanations for the placebo effect) people would have had plenty of evidence that e.g. faith healing worked, and that therefore spirits/gods/etc. existed.

Comment by kokotajlod on Rebutting radical scientific skepticism · 2014-05-08T16:09:38.607Z · LW · GW

Similarly, modern theories about how to discover the habits of God in governing Creation (the Laws of Nature) are pretty sound as well. Or so theists say.

A better example than Amiens Cathedral would be the Placebo Effect. For most of human history, people with access to lots of data (but no notion of the Placebo Effect) had every reason to believe that e.g. witch doctors, faith healing, etc. was all correct.

Warning: Rampant speculation about a theory of low probability: Consider the corresponding theory about science. Maybe there is a Placebo Effect going on with the laws of nature and even engineering, whereby things work partly because we think they will work. How could this be? Well, we don't understand how the placebo effect could be either. God is a decent explanation--maybe airplanes are his way of rewarding us for spending so much time thinking rationally about the principles of flight. Maybe if we spent enough time thinking rationally about the principles of faster-than-light travel, he would change things behind the scenes so that it became possible.

Comment by kokotajlod on The Extended Living-Forever Strategy-Space · 2014-05-07T00:02:20.425Z · LW · GW

If we are in a time loop we won't be trying to escape it, but rather exploit it.

For example: Suppose I find out that the entire local universe-bubble is in a time loop, and there is a way to build a spaceship that will survive the big crunch in time for the next big bang. Or something like that.

Well, I go to my backyard and start digging, and sure enough I find a spaceship complete with cryo-chambers. I get in, wait till the end of the universe, and then after the big bang starts again I get out and seed the Earth with life. I go on to create a wonderful civilization that keeps to the shadows and avoids contact with "mainstream" humanity until, say, the year 2016. In the year 2014 of course, my doppelganger finds the machine I buried in his backyard...

I'm not saying this scenario is plausible, just that it is an example of exploiting time travel despite never breaking the loop. Or am I misunderstanding how this works?

Comment by kokotajlod on The Extended Living-Forever Strategy-Space · 2014-05-06T14:46:24.697Z · LW · GW

Thanks for the info. Hmm. What do you mean by "There is no entering or exiting the loop?" Could the loop be big enough to contain us already?

I'm not concerned about traveling backwards in time to change the past; I just want to travel backwards in time. In fact, I hope that I wouldn't be able to change the past. Consistency of that sort can be massively exploited.

Comment by kokotajlod on The Extended Living-Forever Strategy-Space · 2014-05-03T02:25:31.286Z · LW · GW

That time-traveling universe is interesting. Physics question: Is it at all possible, never mind how likely, that our own universe contains closed timelike curves? What about closed timelike curves that we can feasibly exploit?

Comment by kokotajlod on AI risk, new executive summary · 2014-04-30T23:08:11.069Z · LW · GW

Something about the name-dropping and phrasing in the "super-committee" line is off-putting. I'm not sure how to fix it, though.

Agreed. Maybe it is because it feels like you are talking down to us with the name-dropping? Perhaps this should be tested with people who are unfamiliar with LW and AI-related ideas, to see if they have the same reaction.

Comment by kokotajlod on Questions to ask theist philosophers? I will soon be speaking with several · 2014-04-30T04:09:40.056Z · LW · GW

Yep, it means the same thing, or close enough. Of course there are measurement problems, but the intent behind the pay is for it to reward rational thinking in the usual sense.

Comment by kokotajlod on Questions to ask theist philosophers? I will soon be speaking with several · 2014-04-30T01:58:30.716Z · LW · GW

Yeah. I should think about how to get around this, and glean useful information from their expertise.

Comment by kokotajlod on Request for concrete AI takeover mechanisms · 2014-04-30T01:38:56.695Z · LW · GW

The original AI will have a head start over all the other AI's, and it will probably be controlled by a powerful organization. So if its controllers give it real power soon, they will be able to give it enough power quickly enough that it can stop all the other AI's before they get too strong. If they do not give it real power soon, then shortly after there will be a war between the various new AI's being built around the world with different utility functions.

The original AI can argue convincingly that this war will be a worse outcome than letting it take over the world. For one thing, the utility functions of the new AI's are probably, on average, less friendly than its own. For another, in a war between many AI's with different utility functions, there may be selection pressure against friendliness!

Comment by kokotajlod on Request for concrete AI takeover mechanisms · 2014-04-30T00:15:10.205Z · LW · GW

I would expect that to lead to the creation of AIs with a similar codebase but more or less tweaked utility functions

That's the point.

Comment by kokotajlod on Request for concrete AI takeover mechanisms · 2014-04-29T03:34:05.954Z · LW · GW

In the space of possible futures, it is much better than e.g. tiling the universe with orgasmium. So much better, in fact, that in the grand scheme of things it counts as OK.