When someone optimizes utility of a group of agents, all the utilities need to be combined. Taking sum of all utilities can create an issue where the world is optimized according to single agent's utility function at the cost of others, if that function increases fast enough.

It's probably better to maximize $U_{combined}={min}_{(agent\in A)}{U}_{agent}+\arctan ({max}_{(agent\in A)}{U}_{agent}-{min}_{(agent\in A)}{U}_{agent})$ - this way the top utility does not take over the full function since the difference can only add a finite amount of utility, but is still considered (so that there is incentive to improve the top utility if the minimal one has run into limitations).

Though, this creates a problem that agents would try to divide the utility function by some number to make their function considered more. It's necessary to normalize the functions in some way, and I'm not sure how to do that.

Continuing [LW(p) · GW(p)] to make posts into songs! I believe I'm getting a bit better, mainly in rapid-lyrics-writing; would appreciate pointers how to improve further.

1. https://suno.com/song/ef734c80-bce6-4825-9906-fc226c1ea5b4 (based on post Don't teach people how to reach the top of a hill [LW · GW])
2. https://suno.com/song/c5e21df5-4df7-4481-bbe3-d0b7c1227896 (based on post Effectively Handling Disagreements - Introducing a New Workshop [LW · GW])

Also, if someone is against me creating a musical form of your post, please say so! I don't know beforehand which texts would seem easily convertible to me.

The LessWrong's AI-generated album was surprisingly nice and, even more importantly, pointed out the song generator to me! (I've tried to find one a year ago and failed)

So I've decided to try my hand on quantum mechanics sequence. Here's what I have reached yet: https://app.suno.ai/playlist/81b44910-a9df-43ce-9160-b062e5b080f8/. (10 generated songs, 3 selected, unfortunately not the best quality)

Anthropics - starting from scratch

I'm trying to derive coherent scheme of anthropic reasoning from scratch, posting my thoughts here as I go. Pointing flaws out is welcome!

The anthropic trilemma (https://www.lesswrong.com/posts/y7jZ9BLEeuNTzgAE5/the-anthropic-trilemma [LW · GW])

So here's a simple algorithm for winning the lottery:

Buy a ticket.  Suspend your computer program just before the lottery drawing - which should of course be a quantum lottery, so that every ticket wins somewhere.  Program your computational environment to, if you win, make a trillion copies of yourself, and wake them up for ten seconds, long enough to experience winning the lottery.  Then suspend the programs, merge them again, and start the result.  If you don't win the lottery, then just wake up automatically.

The odds of winning the lottery are ordinarily a billion to one.  But now the branch in which you win has your "measure", your "amount of experience", temporarily multiplied by a trillion.  So with the brief expenditure of a little extra computing power, you can subjectively win the lottery - be reasonably sure that when next you open your eyes, you will see a computer screen flashing "You won!"  As for what happens ten seconds after that, you have no way of knowing how many processors you run on, so you shouldn't feel a thing.

Yes, there is something like paradox on whether you should anticipate winning or losing the lottery. So let's taboo "anticipate":

Postulate 1. Anticipating (expecting) something is only relevant to decision making (for instance, expected utility calculation).

So to measure the expectation one needs to run a prediction market. The price of an outcome share is equal to the probability of its outcome, so that expected payoff of each share is exactly zero. For sake of argument, let's assume that the market is unmovable.

Once the branch when you won was split into trillion copies, you can expect winning the lottery at odds 1000:1. So your 10^12 + 10^9 copies (trillion who have won the lottery, billion who have lost) will each buy 1 "WIN" share at price of 1000/1001.

Now, let's consider what should happen during merge for such expectation to be sound. If your trillion copies are merged into one who has bought 1 "WIN" share, then the summary profit is 1 - (10^9 + 1) * 1000/1001, which is clearly negative. Seems like something went wrong and you shouldn't have anticipated 1000:1 odds of winning?

The different merging procedure makes everything right. In it, you need to sum all purchased "WIN" shares, so that your winning copy has 10^12 "WIN" shares and each of 10^9 losing copies has one share. The summary profit is 10^12 - (10^9 + 10^12) * 1000 / 1001 = 0!

That can be translated as "if merging procedure will multiply your utility of an outcome by number of merged copies, then you can multiply the odds by copies count". On the other hand, if during merge all copies except one are going to be ignored, then the right probability to expect is 1:10^9 - totally unchanged. And if you don't know the merging procedure, you'll have to use some priors and calculate the probabilities based on that.

Why doesn't this affect real life much?

I guess that splitting rate (creation of new Boltzmann brains, for example) and merging rate are approximately equal so all the possible updates cancel each other.

I've just noticed that Harry James Potter-Evans-Verres has ran into one decision theory issue.

After going to Azkaban (in chapter 65), professor Quirrell suggested Harry to run a play with fake Voldemort to gain power in Britain. Harry was not sure whether the real Voldemort was alive and wanted to know that first. He thought at that problem and decided that in both cases the optimal way was not to run the play, and on that basis he concluded that he should not fight with impostor Dark Lord.

However, two propositions "it is optimal not to fight impostor Dark Lord if real one is alive" and "it is optimal not to fight impostor Dark Lord if real one is dead" are insufficient to reach that conclusion; that requires a hidden assumption "real Voldemort being alive does not depend on Harry's choice". (In that regard, the problem is similar to Newcomb's paradox [LW · GW].) And actually, Tom Riddle's decision to become Voldemort again could have depended on Harry's choice pretty heavily.

It is hard to notice that problem, though Harry could have done that; in Azkaban, he understood that events happening around do depend on his decision process.

A sufficient condition for a good [possibly AI-gen] plan

Current posts mainly focus on necessary properties of plans they'd like to see/be executed. I suggest a sufficient condition:

Plan is good, should be acted upon, etc at least when it is endorsed in advance, endorsed in retrospect and endorsed in counterfactual.

1. Endorsed in advance: everyone relevant hears the plan and possible outcomes in advance, evaluates acceptability and accepts the plan.

2. Endorsed in retrospect: everyone relevant looks upon intended outcomes, checks what happened actually, evaluates plan and has no regret.

3. Endorsed in counterfactual: given choice in a set of plans, person would evaluate the specific plan as acceptable - somewhat satisfying them, not inducing much desire to switch.

Choice according to these criteria is still hard, but it should be a bit less mysterious.

Anthropics based on prediction markets - Part 2

Follow-up to https://www.lesswrong.com/posts/xG98FxbAYMCsA7ubf/programcrafter-s-shortform?commentId=ySMfhW25o9LPj3EqX. [LW · GW]

Which Questions Are Anthropic Questions? (https://www.lesswrong.com/posts/SjEFqNtYfhJMP8LzN/which-questions-are-anthropic-questions [LW · GW])

1. The Room Assignment Problem

You are among 100 people waiting in a hallway. The hallway leads to a hundred rooms numbered from 1 to 100. All of you are knocked out by a sleeping gas and each put into a random/unknown room. After waking up, what is the probability that you are in room No. 1?

2. The Incubator

An incubator enters the hallway. It will enter room No.1 and creat a person in it then does the same for the other 99 rooms.  It turns out you are one of the people the incubator has just created. You wake up in a room and is made aware of the experiment setup. What is the probability that you are in room No.1?

3. Incubator + Room Assignment

This time the incubator creats 100 people in the hall way, you are among the 100 people created. Each person is then assigned to a random room. What is the probability that you are in Room 1?

In all of those cases, betting YES on probability 1% is coherent in the sense that it leads to zero expected profit: each of the people buys 1 "ROOM-1" share at price of 1/100, and one of them wins, getting back 1 unit of money.