**chris_leong**on Debate AI and the Decision to Release an AI · 2019-01-18T16:06:27.069Z · score: 2 (1 votes) · LW · GW

A considers A' to be a different agent so it won't help A' for nothing. But there could be some issues with acausal cooperation that I haven't really thought about enough to have a strong opinion on.

**chris_leong**on Debate AI and the Decision to Release an AI · 2019-01-18T12:24:21.509Z · score: 2 (1 votes) · LW · GW

"And we hope that gives an advantage to the side of truth" - we aren't even relying on that. We're handicapping the AI that wants to be released in terms of message length.

**chris_leong**on Debate AI and the Decision to Release an AI · 2019-01-18T12:22:15.691Z · score: 2 (1 votes) · LW · GW

That's a good point, except you aren't addressing my scheme as explained by Gurkenglas

## Debate AI and the Decision to Release an AI

2019-01-17T14:36:53.512Z · score: 7 (1 votes)**chris_leong**on In SIA, reference classes (almost) don't matter · 2019-01-17T13:26:49.296Z · score: 4 (3 votes) · LW · GW

Thanks, that's helpful. Actually, now that you've put it that way, I recall having known this fact at some point in the past.

**chris_leong**on In SIA, reference classes (almost) don't matter · 2019-01-17T12:15:07.438Z · score: 2 (1 votes) · LW · GW

This result seems strange to me, even though the maths seems to check out. Is there a conceptual explanation of why this should be the case?

**chris_leong**on Buy shares in a megaproject · 2019-01-16T23:19:35.264Z · score: 2 (1 votes) · LW · GW

It isn't clear to me how this resolves the problem of Megaprojects. If the shares fall, then perhaps we can tell that the project is likely to fall behind and be assessed a penalty and knowing that will allow some mitigation, but that's a pretty minor fix.

**chris_leong**on In SIA, reference classes (almost) don't matter · 2019-01-16T13:00:05.418Z · score: 4 (2 votes) · LW · GW

pR(Ui) already had an R(Ui), then you divided by it, but the original factor disappears so you are left with a divided by R(Ui). But I don't see where the original factor of R(Ui) went, which would have resulted in cancelling.

**chris_leong**on What are the open problems in Human Rationality? · 2019-01-16T11:30:34.561Z · score: 3 (2 votes) · LW · GW

"It took someone deciding to make it their fulltime project and getting thousands of dollars in funding, which is roughly what such things normally take" - lots of open source projects get off the ground without money being involved

**chris_leong**on In SIA, reference classes (almost) don't matter · 2019-01-15T21:41:40.257Z · score: 4 (2 votes) · LW · GW

When you calculate pR(Ui|sub), you perform the following transformation pR(Ui)→pR(Ui)×R0(Ui)/R(Ui), but an R(Ui) seems to go missing. Can anyone explain?

**chris_leong**on Sleeping Beauty Not Resolved · 2019-01-15T21:30:33.828Z · score: 4 (2 votes) · LW · GW

I just thought I'd add a note in case anyone stumbles upon this thread: Stuart has actually now changed his views on anthropic probabilities as detailed here.

**chris_leong**on Anthropics: Full Non-indexical Conditioning (FNC) is inconsistent · 2019-01-15T21:26:45.575Z · score: 4 (2 votes) · LW · GW

Full non-indexical conditioning is broken in other ways too. As I argued before, the core of this idea is essentially a cute trick where by precommitting to only guess on a certain sequence, you can manipulate the chance that at least one copy of you guesses and that the guesses of your copies are correct. Except full non-indexical conditioning doesn't precommit so the probabilities calculated are for a completely different situation. Hopefully the demonstration of time inconsistency will make it clearer that this approach is incorrect.

**chris_leong**on What are the open problems in Human Rationality? · 2019-01-14T14:10:37.753Z · score: 15 (5 votes) · LW · GW

What do you consider to be his core insights? Would you consider writing a post on this?

**chris_leong**on What are the open problems in Human Rationality? · 2019-01-14T13:46:39.541Z · score: -1 (5 votes) · LW · GW

Group rationality is a big one. It wouldn't surprise me if rationalists are less good on average at co-ordinating than other group because rationalists tend to be more individualistic and have their own opinions of what needs to be done. As an example, how long did it take for us to produce a new LW forum despite half of the people here being programmers? And rationality still doesn't have its own version of CEA.

**chris_leong**on AlphaGo Zero and capability amplification · 2019-01-11T21:11:53.286Z · score: 2 (1 votes) · LW · GW

I don't suppose you could explain how it uses P and V? Does it use P to decide which path to go down and V to avoid fully playing it out?

**chris_leong**on Which approach is most promising for aligned AGI? · 2019-01-08T10:33:06.810Z · score: 2 (1 votes) · LW · GW

This question is specifically about building it, but that's a worthwhile clarification.

**chris_leong**on Optimizing for Stories (vs Optimizing Reality) · 2019-01-08T02:30:49.233Z · score: 2 (1 votes) · LW · GW

I think it might be interesting to discuss how story analysis differs from signalling analysis since I expect most people on Less Wrong to be extremely familiar with this. One difference is that people are happy to be given a story about you even if it is imperfect so that they can slot you into a box. Another is that signalling analysis focuses on whether something makes you look good or bad, while story analysis focuses on how engaging a narrative is. It also focuses more on how cultural tropes shape perspectives - ie. the romanticisation of bank robbers.

## Which approach is most promising for aligned AGI?

2019-01-08T02:19:50.278Z · score: 6 (2 votes)**chris_leong**on Does anti-malaria charity destroy the local anti-malaria industry? · 2019-01-07T02:33:55.361Z · score: 2 (1 votes) · LW · GW

Seems possible, though malaria nets seems like such a niche industry that it wouldn't result in much additional human or infrastructural capital

## On Abstract Systems

2019-01-06T23:41:52.563Z · score: 14 (8 votes)**chris_leong**on In what ways are holidays good? · 2018-12-28T02:22:31.458Z · score: 6 (3 votes) · LW · GW

You missed conversational and social signalling value. Travel is an excellent conversation topic as almost everyone has some memories that they'd love to share. Or at least I find it more interesting than most other smalltalk topics as you're at least learning about other parts of the world. And people who have travelled a lot are seen as more adventurous.

**chris_leong**on Can dying people "hold on" for something they are waiting for? · 2018-12-27T20:56:34.671Z · score: 5 (3 votes) · LW · GW

Maybe we should differentiate holding off losing consciousness from holding off dying? Because I know that I can definitely hold off on falling asleep and maybe holding onto consciousness is the same?

## On Disingenuity

2018-12-26T17:08:47.138Z · score: 34 (15 votes)**chris_leong**on Boundaries enable positive material-informational feedback loops · 2018-12-23T15:02:58.386Z · score: 4 (4 votes) · LW · GW

If EA focused more on feedback loops, then there'd be less focus on donating money to charity. How would you like these resources to be deployed instead?

## Best arguments against worrying about AI risk?

2018-12-23T14:57:09.905Z · score: 15 (7 votes)**chris_leong**on Anthropic probabilities and cost functions · 2018-12-21T23:07:43.616Z · score: 5 (2 votes) · LW · GW

See also: If a tree falls on Sleeping Beauty.

**chris_leong**on Anthropic paradoxes transposed into Anthropic Decision Theory · 2018-12-21T22:44:39.660Z · score: 2 (1 votes) · LW · GW

I suppose that makes sense if you're a moral non-realist.

Also, you may care about other people for reasons of morality. Or simply because you like them. Ultimately why you care doesn't matter and only the fact that you have a preference matters. The morality aspect is inessential.

**chris_leong**on Anthropic paradoxes transposed into Anthropic Decision Theory · 2018-12-21T13:10:54.288Z · score: 2 (1 votes) · LW · GW

"So you can't talk about anthropic "probabilities" without including how much you care about the cost to your copies" - Yeah, but that isn't anything to do with morality, just individual preferences. And instead of using just a probability, you can define probability and the number of repeats.

**chris_leong**on Anthropic paradoxes transposed into Anthropic Decision Theory · 2018-12-21T12:59:02.286Z · score: 2 (1 votes) · LW · GW

Well, if anything that's about your preferences, not morality.

**chris_leong**on Anthropic paradoxes transposed into Anthropic Decision Theory · 2018-12-21T12:04:53.009Z · score: 2 (1 votes) · LW · GW

The way I see it, your morality defines a preference ordering over situations and your decision theory maps from decisions to situations. There can be some interaction there is that different moralities may want different inputs, ie. consequentialism only cares about the consequences, while others care about the actions that you chose. But the point is that each theory should be capable of standing on its own. And I agree with probability being somewhat ambiguous for anthropic situations, but our decision theory can just output betting outcomes instead of probabilities.

**chris_leong**on Anthropic paradoxes transposed into Anthropic Decision Theory · 2018-12-21T10:07:42.302Z · score: 2 (1 votes) · LW · GW

The point is that ADT is a theory of morality + anthropics. When your core theory of anthropics conceptually shouldn't refer to morality at all, but should be independent.

**chris_leong**on The Mad Scientist Decision Problem · 2018-12-21T00:13:21.813Z · score: 2 (1 votes) · LW · GW

This is an interesting reformulation of Counterfactual Mugging. In the case where the cooperation of the paperclip maximiser is provable I don't see it any different from a Counterfactual Mugging taking place before the AI comes into existence. The only way I see this as becoming more complicated is when the AI tries to blackmail you in the counterfactual world.

**chris_leong**on Anthropic paradoxes transposed into Anthropic Decision Theory · 2018-12-20T18:16:15.464Z · score: 3 (2 votes) · LW · GW

I'm generally against this approach because just because X can be modelled as Y doesn't mean that Y is literally true. It mixes up anthropics and morality when these issues should be solved separately. Obviously, this is a neat trick, but I don't see it as anything more.

**chris_leong**on What are some concrete problems about logical counterfactuals? · 2018-12-20T00:03:15.550Z · score: 2 (1 votes) · LW · GW

So most of the problems seem to be related to making UDT work with logical uncertainty?

**chris_leong**on An environment for studying counterfactuals · 2018-12-17T18:44:02.831Z · score: 2 (1 votes) · LW · GW

"The agent receives an observation O as input, from which it can infer whether exploration will occur." - I'm confused here. What is this observation? Does it purely relate to whether it will explore or does it also provide data about the universe? And is it merely a correlation or a binary yes or no?

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-17T15:07:31.854Z · score: 2 (1 votes) · LW · GW

"Its symmetries are essential to what makes Euclidean space interesting" - Isn't the interesting aspect of Euclidean space its ability to model our world excluding relativity?

Well, I just don't think it's that unusual for functions to have properties that break at their limits. Is this any different from 1/x being definable everywhere except 0? Is there anything that makes the change at the limit particularly concerning.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-17T15:02:16.136Z · score: 2 (1 votes) · LW · GW

Okay, I now understand why closure under those operations is incompatible with being well-ordered. And I'm guessing you believe that well-ordering is necessary for a coherent notion of passing through tomorrow infinitely many times because it's a requirement for transfinite induction?

I'm not so sure that this is important. After all, we can imagine getting from 1 to 2 via passing through an infinite number of infinitesimally small steps even though [1,2] isn't well-ordered on <. Indeed, this is the central point of Zeno's paradox.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-16T11:52:25.056Z · score: 2 (1 votes) · LW · GW

Thanks for that suggestion. The long line looks very interesting. Are you suggesting that the boundary doesn't exist?

## What are some concrete problems about logical counterfactuals?

2018-12-16T10:20:26.618Z · score: 26 (6 votes)**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-16T09:59:45.434Z · score: 2 (1 votes) · LW · GW

I'm happy to bite that bullet and destroy the symmetry. If we pick a random point and line in the universe, are there more unit points to the left or right? Well, that depends on where the point is.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-16T09:52:36.193Z · score: 2 (1 votes) · LW · GW

"But this is *equivalent* to having your set be well-ordered, which is incompatible with the property "closed under division and subtraction by finite integers"" - Why is this incompatible?

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-15T18:19:04.427Z · score: 2 (1 votes) · LW · GW

Well shifting left produces a superset of the original, so of course we shouldn't expect that to preserve measure.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-15T10:00:42.916Z · score: 2 (1 votes) · LW · GW

There's definitely a part of me wondering if infinities exist, but before I even consider tackling that question, I need to figure out the most consistent interpretation of infinities assuming they exist.

"At the very least, volume-preserving transformations like shifting everything 1 meter to the left or rotating everything around some axis cease to be volume-preserving, though I don't know if you'd find this convincing" - Well there are non-measurable sets that do this without surreals, but do surreals add more examples?

I'll have to read more about how surreals apply to volumes. It may be hard getting convergence to the exact infinitesimal, but I don't know if the problems will extent beyond that.

(Also, the ability to integrate is mostly besides the point. Instead of the sphere of suffering, we could have defined the expanding cube of suffering. This will then let us solve some special cases of the sphere of suffering)

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-15T09:48:27.648Z · score: 2 (1 votes) · LW · GW

Because we should be working with labelled sequences rather than just sequences (that is sequences with a length attached). That solves the most obvious issues, though there are some subtleties there

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-15T00:51:40.882Z · score: 2 (1 votes) · LW · GW

Thanks for the suggestion. I took a look at nets, but their purpose seems mainly directed towards generalising limits to topological spaces, rather than adding extra nuance to what it means for a sequence to have infinite length. But perhaps you could clarify why you think that they are relevant?

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-14T23:42:07.933Z · score: 2 (1 votes) · LW · GW

Thanks for your feedback. I'll note that these are only informal hints/thoughts on how surreal numbers could help us here and that I'll be providing a more developed version of some of these thoughts in a future post.

Infinitarian paralysis: I consider Pascal's Mugging to be its own seperate problem. Indeed Pascal's Mugging type issues are already present with the more standard infinities. In any case, the measure theory solution is dependent on an individual being a finite fraction of the agents in the universe. While this is an extremely plausible assumption, there doesn't seem to be any principled reason why our solution to infinite paralysis should depend on this assumption.

Paradox of the Gods, Banach-Tarski: Your complaint is that I'm disallowing the a sequence consisting of all finite inverses/all finite integers. I actually believe that actual and potential infinity models of these problems need to be treated separately, though I've only outlined how I plan to handle actual infinities. Hopefully, you find my next post on this topic more persuasive.

Two-envelopes paradox: "The probability that there is a finite amount of money in the envelope is infinitesimal" - Hmm, you're right. That is a rather significant issue.

Sphere of suffering: "Surreal numbers are not the right tool for measuring the volume of Euclidean space or the duration of forever" - why?

St Petersberg Paradox: Ah, but you have an infinitesimal chance of winning a higher infinity. So it becomes an even more extreme version of Pascal's Mugging, but again that's its own discussion.

Magic Dartboard: Yes, I'm aware that rejecting the existence of magic dartboards could have far-reaching consequences. It's something I hope to look into more.

Parity: See my response to gjm. Ordinal numbers lack resolution and so can't properly describe the length of sequence.

**chris_leong**on An Untrollable Mathematician Illustrated · 2018-12-14T22:55:27.726Z · score: 2 (1 votes) · LW · GW

Since propositional consistency is weaker than consistency our prior may distribute some probability to cases that are contradictory. I guess that's considered acceptable because the aim is to make the prior non-trollable, rather than good.

**chris_leong**on An Untrollable Mathematician Illustrated · 2018-12-14T22:50:55.649Z · score: 2 (1 votes) · LW · GW

This confused me as well. This being true ensures that the ratio P(A):P(not A) doubles at each step. But part of this comic seems to imply that being less than a half stops the trolling, when it should only stop the trolling from proceeding at such a fast-paced rate.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-14T22:44:57.785Z · score: 4 (2 votes) · LW · GW

Maybe I should re-emphasise the caveat at the top of the post: "I will provide informal hints on how surreal numbers could help us solve some of these paradoxes, although the focus on this post is primarily categorisation, so please don't mistake these for formal proofs. I'm also aware that simply noting that a formalisation provides a satisfactory solution doesn't philosophically justify its use, but this is also not the focus of this post."

You wrote that I "*made the entirely arbitrary choice that you are only prepared to understand it in terms of surreal numbers". *This choice isn't arbitrary. I've given some hints as to why I am taking this approach, but a full justification won't occur until future posts.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-14T13:26:26.658Z · score: 2 (1 votes) · LW · GW

I already pointed above to the distinction between absolute and potential infinities. I admit that the surreal solution assumes that we are dealing with an absolute infinity instead of a potential one, so let's just consider this case. You want to conceive of this problem as "a sequence whose order-type is ω", but from the surreal perspective this lacks resolution. Is the number of elements (surreal) ω, ω+1 or ω+1000? All of these are possible given that in the ordinals 1+ω=ω so we can add arbitrarily many numbers to the start of a sequence without changing its order type.

So I don't think the ordinary notion of sequence makes sense. In particular, it doesn't account for the fact that two sequences which appear to be the same in every place can actually be different if they have different lengths. Anyway, I'll try to untangle some of these issues in future posts, in particular I'm leaning towards hyperreals as a better fit for modelling potential infinities, but I'm still uncertain about how things will develop once I manage to look into this more.

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-14T12:53:48.871Z · score: 2 (1 votes) · LW · GW

We can definitely solve this problem for real agents, but the reason why I find this problem so perplexing is because of the boundary issue that it highlights. Imagine that we have an actual infinite number of people. Color all the finite placed people red and the non-finite placed people blue. Everyone one t the right of a red person should be red and everyone one to the left of blue person should be blue. So what does the boundary look like? Sure we can't finitely transverse from the start to the infinite numbers, but that doesn't affect the intuition that the boundary should still be there somewhere. And this makes me question whether the notion of an actual infinity is coherent (I really don't know).

**chris_leong**on An Extensive Categorisation of Infinite Paradoxes · 2018-12-13T23:26:20.111Z · score: 1 (3 votes) · LW · GW

"Put differently, just because we count up to n doesn't mean we pass through n/3" - The first possible objection I'll deal with is not what I think you are complaining about, but I think it's worth handling anyway. n/3 mightn't be an Omnific Number, but in this case we just take the integer part of n/3.

I think the issue you are highlighting is that all finite numbers are less than n/3. And if you define a sequence as consisting of finite numbers then it'll never include n/3. However, if you define a sequence as all numbers between 1 and x where x is a surreal number then you don't encounter this issue. Is this valid? I would argue that this question boils down to whether there are an actually infinite number of days on which Trump experiences or only a potential infinity. If it's an actual infinity, then the surreal solution seems fine and we can say that Trump should stop saying yes on day n/3. If it's only a potential infinity, then this solution doesn't work, but I don't endorse surreals in this case (still reading about this)

## An Extensive Categorisation of Infinite Paradoxes

2018-12-13T18:36:53.972Z · score: 39 (14 votes)**chris_leong**on What is "Social Reality?" · 2018-12-08T21:42:58.822Z · score: 0 (6 votes) · LW · GW

I don't see this question as very well defined. Words mean whatever we want them to mean. What do you want it to mean in this context?

**chris_leong**on Coherence arguments do not imply goal-directed behavior · 2018-12-08T10:32:13.354Z · score: 2 (1 votes) · LW · GW

Presumably twitching requires sending a signal to a motor control and the connection here can be broken

**chris_leong**on Coherence arguments do not imply goal-directed behavior · 2018-12-05T12:47:21.830Z · score: 2 (1 votes) · LW · GW

POMDP is an abstraction. Real agents can be interfered with.

**chris_leong**on The housekeeper · 2018-12-04T19:24:00.975Z · score: 3 (2 votes) · LW · GW

Why do you think it's fallen down to 10?