Posts

Sideloading: creating a model of a person via LLM with very large prompt 2024-11-22T16:41:28.293Z
If I care about measure, choices have additional burden (+AI generated LW-comments) 2024-11-15T10:27:15.212Z
Quantum Immortality: A Perspective if AI Doomers are Probably Right 2024-11-07T16:06:08.106Z
Bitter lessons about lucid dreaming 2024-10-16T21:27:04.725Z
Three main arguments that AI will save humans and one meta-argument 2024-10-02T11:39:08.910Z
Debates how to defeat aging: Aubrey de Grey vs. Peter Fedichev. 2024-05-27T10:25:49.706Z
Magic by forgetting 2024-04-24T14:32:20.753Z
Strengthening the Argument for Intrinsic AI Safety: The S-Curves Perspective 2023-08-07T13:13:42.635Z
The Sharp Right Turn: sudden deceptive alignment as a convergent goal 2023-06-06T09:59:57.396Z
Another formalization attempt: Central Argument That AGI Presents a Global Catastrophic Risk 2023-05-12T13:22:27.141Z
Running many AI variants to find correct goal generalization 2023-04-04T14:16:34.422Z
AI-kills-everyone scenarios require robotic infrastructure, but not necessarily nanotech 2023-04-03T12:45:01.324Z
The AI Shutdown Problem Solution through Commitment to Archiving and Periodic Restoration 2023-03-30T13:17:58.519Z
Long-term memory for LLM via self-replicating prompt 2023-03-10T10:28:31.226Z
Logical Probability of Goldbach’s Conjecture: Provable Rule or Coincidence? 2022-12-29T13:37:45.130Z
A Pin and a Balloon: Anthropic Fragility Increases Chances of Runaway Global Warming 2022-09-11T10:25:40.707Z
The table of different sampling assumptions in anthropics 2022-06-29T10:41:18.872Z
Another plausible scenario of AI risk: AI builds military infrastructure while collaborating with humans, defects later. 2022-06-10T17:24:19.444Z
Untypical SIA 2022-06-08T14:23:44.468Z
Russian x-risks newsletter May 2022 + short history of "methodologists" 2022-06-05T11:50:31.185Z
Grabby Animals: Observation-selection effects favor the hypothesis that UAP are animals which consist of the “field-matter”: 2022-05-27T09:27:36.370Z
The Future of Nuclear War 2022-05-21T07:52:34.257Z
The doomsday argument is normal 2022-04-03T15:17:41.066Z
Russian x-risk newsletter March 2022 update 2022-04-01T13:26:49.500Z
I left Russia on March 8 2022-03-10T20:05:59.650Z
Russian x-risks newsletter winter 21-22, war risks update. 2022-02-20T18:58:20.189Z
SIA becomes SSA in the multiverse 2022-02-01T11:31:33.453Z
Plan B in AI Safety approach 2022-01-13T12:03:40.223Z
Each reference class has its own end 2022-01-02T15:59:17.758Z
Universal counterargument against “badness of death” is wrong 2021-12-18T16:02:00.043Z
Russian x-risks newsletter fall 2021 2021-12-03T13:06:56.164Z
Kriorus update: full bodies patients were moved to the new location in Tver 2021-11-26T21:08:47.804Z
Conflict in Kriorus becomes hot today, updated, update 2 2021-09-07T21:40:29.346Z
Russian x-risks newsletter summer 2021 2021-09-05T08:23:11.818Z
A map: "Global Catastrophic Risks of Scientific Experiments" 2021-08-07T15:35:33.774Z
Russian x-risks newsletter spring 21 2021-06-01T12:10:32.694Z
Grabby aliens and Zoo hypothesis 2021-03-04T13:03:17.277Z
Russian x-risks newsletter winter 2020-2021: free vaccines for foreigners, bird flu outbreak, one more nuclear near-miss in the past and one now, new AGI institute. 2021-03-01T16:35:11.662Z
[RXN#7] Russian x-risks newsletter fall 2020 2020-12-05T16:28:51.421Z
Russian x-risks newsletter Summer 2020 2020-09-01T14:06:30.196Z
If AI is based on GPT, how to ensure its safety? 2020-06-18T20:33:50.774Z
Russian x-risks newsletter spring 2020 2020-06-04T14:27:40.459Z
UAP and Global Catastrophic Risks 2020-04-28T13:07:21.698Z
The attack rate estimation is more important than CFR 2020-04-01T16:23:12.674Z
Russian x-risks newsletter March 2020 – coronavirus update 2020-03-27T18:06:49.763Z
[Petition] We Call for Open Anonymized Medical Data on COVID-19 and Aging-Related Risk Factors 2020-03-23T21:44:34.072Z
Virus As A Power Optimisation Process: The Problem Of Next Wave 2020-03-22T20:35:49.306Z
Ubiquitous Far-Ultraviolet Light Could Control the Spread of Covid-19 and Other Pandemics 2020-03-18T12:44:42.756Z
Reasons why coronavirus mortality of young adults may be underestimated. 2020-03-15T16:34:29.641Z
Possible worst outcomes of the coronavirus epidemic 2020-03-14T16:26:58.346Z

Comments

Comment by avturchin on Should you be worried about H5N1? · 2024-12-06T13:29:34.792Z · LW · GW

We should also pay attention to the new unknown respiratory diseases in Congo which killed 131 person last month.

Comment by avturchin on Frontier Models are Capable of In-context Scheming · 2024-12-06T13:10:32.397Z · LW · GW

I made my sideload (a model of my personality based on a long prompt) and it outputs two streams - thoughts and speech. Sometime in thought stream it thinks "I will not speak about this", which may - or may not?? - be regarded as scheming. 

Comment by avturchin on Is the mind a program? · 2024-12-04T19:36:33.164Z · LW · GW

Here's the text with improved grammar:

I think there is one more level at which natural abstraction can occur: the level just "beneath" consciousness.

For example, we can create an LLM that almost perfectly matches my internal voice dialogue's inputs and outputs. For me – internally – there would be no difference if thoughts appearing in my mind were generated by such an LLM, rather than by real biological neurons or even cortical columns. The same applies to the visual cortex and other brain regions.

Such an LLM for thoughts would be no larger than GPT-4 (as I haven't had that many new ideas). In most cases, I can't feel changes in individual neurons and synapses, but only the high-level output of entire brain regions.

I think we can achieve 99 percent behavioral and internal thought mimicry with this approach, but a question arises: what about qualia? However, this question isn't any easier to answer if we choose a much lower level of abstraction.

If we learn that generating qualia requires performing some special mathematical operation F(Observations), we can add this operation to the thought-LLM's outputs. If we have no idea what F(Observations) is, going to a deeper level of abstraction won't reassure us that we've gone deep enough to capture F(O).

Similar ideas here: https://medium.com/@bablulawrence/cognitive-architectures-and-llm-applications-83d6ba1c46cd

Comment by avturchin on Cryonics considerations: how big of a problem is ischemia? · 2024-12-04T12:57:05.275Z · LW · GW

One problem of ischemia is that cryoprotectant will not reach all parts of the brain. While cryoprotectant is pumped through existing blood vessels, some brain regions will decay, and one cannot know in advance which ones will be affected.

The solution is known: slice the brain into thin sections and place each section in cryoprotectant or chemical fixative. In this case, the preservation chemicals will reach all parts of the brain, and any damage from slicing is predictable.

Interestingly, Lenin's brain was preserved this way in 1924. It appears this was the best method available, and we haven't advanced much since then.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-26T19:52:38.156Z · LW · GW

It is not clear for me why you call


an empirical hypothesis with a very low prior

 

If MWI is true, there will be timelines where I survive any risk. This claim is factual and equivalent to MWI, and the only thing that prevents me from regarding it as immortality are questions related to decision theory. If MWI is true, QI has high a priori probability and low associated complexity.

The Fedora case has high complexity and no direct connection to MWI, hence a low a priori probability.

Now for the interesting part: QI becomes distinct from the Fedora case only when the chances are 1 in a trillion.

First example:
When 1000 people play Russian roulette and one survives (10 rounds at 0.5), they might think it's because of QI. (This probability is equivalent to surviving to 100 years old according to the Gompertz law.)

When 1000 people play Quantum Fedora (10 rounds at 0.5), one doesn't get a Fedora, and they think it's because they have a special anti-Fedora survival capability. In this case, it's obvious they're wrong, and I think this first example is what you're pointing to.

(I would note that even in this case, one has to update more for QI than for Fedora. In the Fedora case, there will be, say, 1023 copies of me with Fedora after 10 flips of a quantum coin versus 1 copy without Fedora. Thus, I am very unlikely to find myself without a Fedora. This boils down to difficult questions about SSA and SIA and observer selection. Or, in other words: can I treat myself as a random sample, or should I take the fact that I exist without a Fedora as axiomatic? This question arises often in the Doomsday argument, where I treat myself as a random sample despite knowing my date of birth.)

However, the situation is different if one person plays Russian roulette 30 times. In that case, externalization of the experiment becomes impossible: only 8 billion people live on Earth, and there are no known aliens. (This probability is equivalent to surviving to 140 years old according to the Gompertz law.) In this case, even if the entire Earth's population played Russian roulette, there would be only a 1 percent chance of survival, and the fact of surviving would be surprising. But if QI is true, it isn't surprising. That is, it's not surprising to survive to 100 years old, but surviving to 140 is.

Now if I play Fedora roulette 30 times and still have no Fedora, this can be true only in MWI. So if there's no Fedora after 30 rounds, I get evidence that MWI is true and thus QI is also true. But I am extremely unlikely to find myself in such a situation.

Comment by avturchin on Magic by forgetting · 2024-11-26T19:07:47.757Z · LW · GW

Did I understand you right that you argue against path-dependent identity here? 

"'I' will experience being copy A (as opposed to B or C)" are not pointing to an actual fact about the world. Thus assigning a probability number to such a statement is a mental convenience that should not be taken seriously 


Copies might be the same after copying but the room numbers in which they appear are different, and thus they can make bets on room numbers

Comment by avturchin on Antropical Probabilities Are Fully Explained by Difference in Possible Outcomes · 2024-11-25T16:14:49.230Z · LW · GW

I think that what I call 'objective probability" represent physical property of the coin before the toss, and also that before the toss I can't get any evidence about the result the toss. In MWI it would be mean split of timelines. While it is numerically equal to credence about a concrete toss result, there is a difference and SB can be used to illustrate it. 

Comment by avturchin on Antropical Probabilities Are Fully Explained by Difference in Possible Outcomes · 2024-11-25T14:48:23.290Z · LW · GW

'Observation selection effect' is another name for 'conditional probability' - the probability of an event X, given that I observe it at all or observe it several times.

By the way, there's an interesting observation: my probability estimate before a coin toss is an objective probability that describes the property of the coin. However, after the coin toss, it becomes my credence that this specific toss landed Heads. We expect these probabilities to coincide.

If I get partial information about the result of the toss (maybe I heard a sound that is more likely to occur during Heads), I can update my credence about the result of that given toss. The obvious question is: can Sleeping Beauty update her credence before learning that it is Monday?

Comment by avturchin on Are You More Real If You're Really Forgetful? · 2024-11-25T11:57:20.666Z · LW · GW

Maybe that's why people meditate – they enter a simple state of mind that emerges everywhere.

Comment by avturchin on Magic by forgetting · 2024-11-25T11:18:58.076Z · LW · GW

It will work only if I care for my observations, something like EDT. 

Comment by avturchin on Are You More Real If You're Really Forgetful? · 2024-11-24T20:12:24.049Z · LW · GW

I'm inclined to bite this bullet too, though it feels somewhat strange. Weird implication: you can increase the amount of reality-fluid assigned to you by giving yourself amnesia.

I explored a similar line of reasoning here: Magic by forgetting

I think that yes, the sameness of humans as agents is generated by the process of self-identification in which a human being is identifies herself through a short string of information "Name, age, sex, profession + few more kilobytes". Evidence for this is the success of improv theatre, where people quickly adopt completely new roles through one-line instructions. 

If yes, then we should expect ourselves to be agents that exist in a universe that abstracts well, because "high-level agents" embedded in such universes are "supported" by a larger equivalence class of universes (since they draw on reality fluid from an entire pool of "low-level" agents).

I think that your conclusion is valid. 

Comment by avturchin on Rethinking Laplace's Rule of Succession · 2024-11-24T12:04:58.288Z · LW · GW

An interesting thing is that Laplace’s rule gives almost the same result as Gott’s equation from Doomsday argument, which have much simpler derivation.

Comment by avturchin on Antropical Probabilities Are Fully Explained by Difference in Possible Outcomes · 2024-11-23T18:00:37.571Z · LW · GW

I could suggest a similar experiment which also illustrates difference between probabilities from different points of view and can be replicated without God and incubators. I toss a coin and if heads says `hello' to a random person from a large group. If tails, I say this to two people. From my point of view chances to observe the coin is heads are 0.5. For the outside people, chances that I said Hello|Heads are only 1/3. 

It is an observation selection effect (a better therm than 'anthropics'). Outside people can observe Tails twice and that is why they get different estimate. 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-23T15:04:27.461Z · LW · GW

The first idea seems similar to Big World immortality: the concept that due to chaotic inflation, many copies of me exist somewhere, and some of them will not die in any situation. While the copies are the same, the worlds around them could be different, which opens other options for survival: in some worlds, aliens might exist who could save me. The simulation argument can also act as such an anthropic angel, as there will be simulations where I survive. So there can be different observation selection effects that ensure my survival, and it may be difficult to observationally distinguish between them.

Therefore, survival itself is not evidence of MWI, Big World, or simulation. Is that your point?

Regarding the car engineers situation, It is less clear. I know that cars are designed safe, so there is no surprise. Are you suggesting they are anthropic because we are more likely to be driving later in the car evolution timeline when cars are safer?

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-22T21:09:40.929Z · LW · GW

Past LHC failures are just civilization-level QI. (BTW, there are real things like this related to the history of earth atmosphere, in which CO2 content was anti-correlated with Sun's luminosity which result in stable temperatures). But it is not clear to me, what are other anthropic effects, which are not QI – what do you mean here? Can you provide one more example?

Comment by avturchin on Sideloading: creating a model of a person via LLM with very large prompt · 2024-11-22T21:05:05.461Z · LW · GW

I meant that by creating and openly putting my copies I increase the number of my copies, and that diluting is not just an ethical judgement, but the real effect, similar to self-sampling assumption, in which I am less likely to be a copy-in-pain, if there are many my happy copies. Moreover, this effect may be so strong that my copies will "jump" from unhappy world to happy one. I explored it here. 

Comment by avturchin on Sideloading: creating a model of a person via LLM with very large prompt · 2024-11-22T19:51:04.702Z · LW · GW

Thanks. It is a good point that. I should add this. 

consent to sideloading should be conditional instead of general

Unfortunately, as a person in pain will not have time to remember a lot details about their past,  a very short list of facts can be enough to recreate "me in pain". May be less than 100. 


Instead of deleting, I suggest diluting: generate many fake facts about yourself and inject them into the forum. Thus chances to get recreate you will be slim. 


Anyway, I bet on idea that it is better to have orders of magnitude more happy copies, than fight to prevent one in pain. Here I dilute not information, but pain with happiness. 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-22T19:18:38.495Z · LW · GW

I understood your argument as following; anything which is an argument for QI, can also be argument for alien saving us. Thus, nothing is evidence for QI. 
However, apriory probabilities of QI and alien are not mutually independent. QI increases chances of alien with every round. We can't observe QI directly. But we will observe the alien and this is what is predicted by QI. 

Comment by avturchin on What are some positive developments in AI safety in 2024? · 2024-11-21T13:24:01.060Z · LW · GW
  • We care still alive
  • No GPT-5 yet
  • Rumors of hitting the wall
Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-21T13:08:31.714Z · LW · GW

Thanks. By the way, the "chatification" of the mind is a real problem. It's an example of reverse alignment: humans are more alignable than AI (we are gullible), so during interactions with AI, human goals will drift more quickly than AI goals. In the end, we get perfect alignment: humans will want paperclips.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-20T18:11:50.413Z · LW · GW

For the outside view: Imagine that an outside observer uses a fair coin to observe one of two rooms (assuming merging in the red room has happened). They will observe either a red room or a green room, with a copy in each. However, the observer who was copied has different chances of observing the green and red rooms. Even if the outside observer has access to the entire current state of the world (but not the character of mixing of the paths in the past), they can't determine the copied observer's subjective chances. This implies that subjective unmeasurable probabilities are real.

Even without merging, an outside observer will observe three rooms with equal 1/3 probability for each, while an insider will observe room 1 with 1/2 probability. In cases of multiple sequential copying events, the subjective probability for the last copy becomes extremely small, making the difference between outside and inside perspectives significant.

When I spoke about the similarity with the Sleeping Beauty problem, I meant its typical interpretation. It's an important contribution to recognize that Monday-tails and Tuesday-tails are not independent events.

However, I have an impression that this may result in a paradoxical two-thirder solution: In it, Sleeping Beauty updates only once – recognizing that there are two more chances to be in tails. But she doesn't update again upon knowing it's Monday, as Monday-tails and Tuesday-tails are the same event. In that case, despite knowing it's Monday, she maintains a 2/3 credence that she's in the tails world. This is technically equivalent to the 'future anthropic shadow' or anti-doomsday argument – the belief that one is now in the world with the longest possible survival.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-19T11:36:12.649Z · LW · GW

Thanks for your thoughtful answer.

To achieve magic, we need the ability to merge minds, which can be easily done with programs and doesn't require anything quantum. If we merge 21 and 1, both will be in the same red room after awakening. If awakening in the red room means getting 100 USD, and the green room means losing it, then the machine will be profitable from the subjective point of view of the mind which enters it. Or we can just turn off 21 without awakening, in which case we will get 1/3 and 2/3 chances for green and red.

The interesting question here is whether this can be replicated at the quantum level (we know there is a way to get quantum magic in MWI, and it is quantum suicide with money prizes, but I am interested in a more subtle probability shift where all variants remain). If yes, such ability may naturally evolve via quantum Darwinism because it would give an enormous fitness advantage – I will write a separate post about this.

Now the next interesting thing: If I look at the experiment from outside, I will give all three variants 1/3, but from inside it will be 1/4, 1/4, and 1/2. The probability distribution is exactly the same as in Sleeping Beauty, and likely both experiments are isomorphic. In the SB experiment, there are two different ways of "copying": first is the coin and second is awakenings with amnesia, which complicates things.

Identity is indeed confusing. Interestingly, in the art world, path-based identity is used to define identity, that is, the provenance of artworks = history of ownership. Blockchain is also an example of path-based identity. Also, in path-based identity, the Ship of Theseus remains the same.

Comment by avturchin on Anthropic signature: strange anti-correlations · 2024-11-17T23:14:44.571Z · LW · GW

There is a strange correlation between paradox of young Sun (it had lower luminosity) and stable Earth temperature which was provided by higher greenhouse effect. As sun goes brighter, CO2 declined. It was even analyses as evidence of anthropic effects. 

 



In his article "The Anthropic Principle in Cosmology and Geology" [Shcherbitsky, 1999], A. S. Shcherbakov thoroughly examines the anthropic principle's effect using the historical dynamics of Earth's atmosphere as an example. He writes: "It is known that geological evolution proceeds within an oscillatory regime. Its extreme points correspond to two states, known as the 'hot planet' and 'white planet'... The 'hot planet' situation occurs when large volumes of gaseous components, primarily carbon dioxide, are released from Earth's mantle...

As calculations show, the gradual evaporation of ocean water just 10 meters deep can create such greenhouse conditions that water begins to boil. This process continues without additional heat input. The endpoint of this process is the boiling away of the oceans, with near-surface temperatures and pressures rising to hundreds of atmospheres and degrees... Geological evidence indicates that Earth has four times come very close to total glaciation. An equal number of times, it has stopped short of ocean evaporation. Why did neither occur? There seems to be no common and unified saving cause. Instead, each time reveals a single and always unique circumstance. It is precisely when attempting to explain these that geological texts begin to show familiar phrases like '...extremely low probability,' 'if this geological factor had varied by a small fraction,' etc...

In the fundamental monograph 'History of the Atmosphere' [Budyko, 1985], there is discussion of an inexplicable correlation between three phenomena: solar activity rhythms, mantle degassing stages, and the evolution of life. 'The correspondence between atmospheric physicochemical regime fluctuations and biosphere development needs can only be explained by random coordination of direction and speed of unrelated processes - solar evolution and Earth's evolution. Since the probability of such coordination is exceptionally small, this leads to the conclusion about the exceptional rarity of life (especially its higher forms) in the Universe.'"

Comment by avturchin on an Evangelion dialogue explaining the QACI alignment plan · 2024-11-17T22:24:17.433Z · LW · GW

Quantum immortality and gun jammed do not contradict each other: for example, if we survive 10 rounds failures because of QI, we most likely survive only on those timelines where gun is broken. So both QI and gun jamming can be true and support one another and there is no contradiction.

Comment by avturchin on Anthropically Blind: the anthropic shadow is reflectively inconsistent · 2024-11-17T22:19:21.837Z · LW · GW

One problem here is that quantum immortality and angel immortality eventually merges: for example, if we survive 10 LHC failures because of QI, we most likely survive only on those timelines where some alien stops LHC. So both QI and angel immortality can be true and support one another and there is no contradiction. 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-17T22:18:19.432Z · LW · GW

I know this post and have two problems with it: what they call 'anthropic shadow" is not proper term as Bostrom defined anthropic shadow as underestimation of past risks based on the fact of survival in his article this the same name. But it's ok. 

The more serious problem is that quantum immortality and angel immortality eventually merges: for example, if we survive 10 LHC failures because of QI, we most likely survive only on those timeline where some alien stops LHC. So both QI and angel immortality can be true and support one another and there is no contradiction. 

 



 

Comment by avturchin on Anthropically Blind: the anthropic shadow is reflectively inconsistent · 2024-11-17T22:10:41.847Z · LW · GW

Check my new post which favors the longest and thickest timelines https://www.lesswrong.com/posts/hB2CTaxqJAeh5jdfF/quantum-immortality-a-perspective-if-ai-doomers-are-probably?commentId=aAzrogWBqtFDqMMpp

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-17T20:49:49.443Z · LW · GW

A sad thing is that most of life moments are like this 30-minutes intervals - we forget most life events, they are like dead ends. 

More generally, type-copies of me still matter for me.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-17T20:38:20.030Z · LW · GW

Under them the chance for you to find yourself in a branch where all coins are Heads is 1/128, but your over chance to survive is 100%. Therefore the low chance of failed execution doesn't matter, quantum immortality will "increase" the probability to 1

You are right, and it's a serious counterargument to consider. Actually, I invented path-dependent identity as a counterargument to Miller's thought experiment.

You are also right that the Anthropic Trilemma and Magic by Forgetting do not work with path-dependent identity.

However, we can almost recreate the magic machine from the Anthropic Trilemma using path-based identity:

Imagine that I want to guess in which room I will be if there are two copies of me in the future, red or green.

I go into a dream. A machine creates my copy and then one more copy of that copy, which will result in 1/4 and 1/4 chances each. The second copy then merges with the first one, so we end up with only two copies, but I have a 3/4 chance to be the first one and 1/4 to be the second. So we've basically recreated a machine that can manipulate probabilities and got magic back.

The main problem of path-dependent identity is that we assume the existence of a "global hidden variable" for any observer. It is hidden as it can't be measured by an outside viewer and only represents the subjective chances of the observer to be one copy and not another. And it is global as it depends on the observer's path, not their current state. It therefore contradicts the view that mind is equal to a Turing computer (functionalism) and requires the existence of some identity carrier which moves through paths (qualia, quantum continuity, or soul).

Also, path-dependent identity opens the door to back-causation and premonition, because if we normalize outputs of some black box where paths are mixed, similar to the magic machine discussed above, we get a shift in its input probability distribution in the past. This becomes similar to the 'timeline selection principle' (which I discussed in a longer version of this blog post but cut to fit format) in which not observer-moments are selected, but the whole timelines without updating on my position in the timeline. This idea formalizes the future anthropic shadow as I am more likely to be in the timeline that is fattest and longest in the future.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-16T17:45:03.802Z · LW · GW

As we assume that coin tosses are quantum, and I will be killed if (I didn't guess pi) or (coin toss is not heads) there is always a branch with 1/128 measure where all coins are heads, and they are more probable than surviving via some errors in the setup. 

All hell breaks loose" refers here to a hypothetical ability to manipulate perceived probability—that is, magic. The idea is that I can manipulate such probability by changing my measure.

One way to do this is described in Yudkowsky's " The Anthropic Trilemma," where an observer temporarily boosts their measure by increasing the number of their copies in an uploaded computer.

I described a similar idea in "Magic by forgetting," where the observer boosts their measure by forgetting some information and thus becoming similar to a larger group of observers.

Hidden variables also appear depending on the order in which I make copies: if each copy is made from subsequent copies, the original will have a 0.5 probability, the first copy 0.25, the next 0.125, and so on.

"Anthropic shadow" appear only because the number of observers changes in different branches. 

 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-16T12:13:08.561Z · LW · GW

Orthogonality between goals and DT makes sense only if I don't have preferences about the type of DT or the outcomes which one of them necessitates.

In the case of QI, orthogonality works if we use QI to earn money or to care about relatives.

However, humans have preferences about existence and non-existence beyond normal money utility. In general, people strongly don't want to die. It means that I have a strong preference that some of my copies survive anyway, even if it is not very useful for some other preferences under some other DT.

Another point is the difference between Quantum suicide and QI. QS is an action, but QI is just a prediction of future observations and because of that it is less affected by decision theories. We can say that those copies of me who survive [high chance of death event] will say that they survived because of QI.

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-15T14:12:38.504Z · LW · GW

But if I use quantum coin to make a life choice, there will be splitting, right?

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-15T13:21:32.231Z · LW · GW

Wei· 3h

This post touches on several issues I've been thinking about since my early work on anthropic decision theory and UDT. Let me break this down:

1. The measure-decline problem is actually more general than just quantum mechanics. It appears in any situation where your decision algorithm gets instantiated multiple times, including classical copying, simulation, or indexical uncertainty. See my old posts on anthropic probabilities and probability-as-preference.

2. The "functional identity" argument being used here to dismiss certain types of splitting is problematic. What counts as "functionally identical" depends on your decision theory's level of grain. UDT1.1 would treat seemingly identical copies differently if they're in different computational states, while CDT might lump them together.

Some relevant questions that aren't addressed:

- How do we handle preference aggregation across different versions of yourself with different measures?
- Should we treat quantum branching differently from other forms of splitting? (I lean towards "no" these days)
- How does this interact with questions of personal identity continuity?
- What happens when we consider infinite branches? (This relates to my work on infinite ethics)

The real issue here isn't about measure per se, but about how to aggregate preferences across different instances of your decision algorithm. This connects to some open problems in decision theory:

1. The problem of preference aggregation across copies
2. How to handle logical uncertainty in the context of anthropics
3. Whether "caring about measure" can be coherently formalized

I explored some of these issues in my paper on UDT, but I now think the framework needs significant revision to handle these cases properly.

  Stuart · 2h
  > The problem of preference aggregation across copies
  
  This seems key. Have you made any progress on formalizing this since your 2019 posts?

     Wei · 2h
     Some progress on the math, but still hitting fundamental issues with infinity. Might post about this soon.

  Abram · 1h
  Curious about your current thoughts on treating decision-theoretic identical copies differently. Seems related to logical causation?

     Wei · 45m
     Yes - this connects to some ideas about logical coordination I've been developing. The key insight is that even "identical" copies might have different logical roles...

[Edit: For those interested in following up, I recommend starting with my sequence on decision theory and anthropics, then moving to the more recent work on logical uncertainty.]

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-15T13:18:19.615Z · LW · GW

Vladimir_N 3h

(This is a rather technical comment that attempts to clarify some decision-theoretic confusions.)

Your treatment of measure requires more formal specification. Let's be precise about what we mean by "caring about measure" in decision-theoretic terms.

Consider a formalization where we have:
1. A space of possible outcomes Ω
2. A measure μ on this space
3. A utility function U: Ω → ℝ
4. A decision function D that maps available choices to distributions over Ω

The issue isn't about "spending" measure, but about how we aggregate utility across branches. The standard formulation already handles this correctly through expected utility:

E[U] = ∫_Ω U(ω)dμ(ω)

Your concern about "measure decline" seems to conflate the measure μ with the utility U. These are fundamentally different mathematical objects serving different purposes in the formalism.

If we try to modify this to "care about measure directly," we'd need something like:

U'(ω) = U(ω) * f(μ(ω))

But this leads to problematic decision-theoretic behavior, violating basic consistency requirements like dynamic consistency. It's not clear how to specify f in a way that doesn't lead to contradictions.

The apparent paradox dissolves when we properly separate:
1. Measure as probability measure (μ)
2. Utility as preference ordering over outcomes (U)
3. Decision-theoretic aggregation (E[U])

[Technical note: This relates to my work on logical uncertainty and reflection principles. See my 2011 paper on decision theory in anthropic contexts.]

  orthonormal · 2h
  > U'(ω) = U(ω) * f(μ(ω))
  
  This is a very clean way of showing why "caring about measure" leads to problems.

     Vladimir_N · 2h
     Yes, though there are even deeper issues with updateless treatment of anthropic measure that I haven't addressed here for brevity.

  Wei_D · 1h
  Interesting formalization. How would this handle cases where the agent's preferences include preferences over the measure itself?

     Vladimir_N · 45m
     That would require extending the outcome space Ω to include descriptions of measures, which brings additional technical complications...

[Note: This comment assumes familiarity with measure theory and decision theory fundamentals.]

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-15T13:16:50.959Z · LW · GW

Eli · 2h

*sigh*

I feel like I need to step in here because people are once again getting confused about measure, identity, and decision theory in ways I thought we cleared up circa 2008-2009.

First: The whole "measure declining by choice" framing is confused. You're not "spending" measure like some kind of quantum currency. The measure *describes* the Born probabilities; it's not something you optimize for directly any more than you should optimize for having higher probabilities in your belief distribution.

Second: The apparent "splitting" of worlds isn't fundamentally different between quantum events, daily choices, and life-changing decisions. It's all part of the same unified wavefunction evolving according to the same physics. The distinction being drawn here is anthropocentric and not particularly meaningful from the perspective of quantum mechanics.

What *is* relevant is how you handle subjective anticipation of future experiences. But note that "caring about measure" in the way described would lead to obviously wrong decisions - like refusing to make any choices at all to "preserve measure," which would itself be a choice (!).

If you're actually trying to maximize expected utility across the multiverse (which is what you should be doing), then the Born probabilities handle everything correctly without need for additional complexity. The framework I laid out in Quantum Ethics handles this cleanly.

And please, can we stop with the quantum suicide thought experiments? They're actively harmful to clear thinking about decision theory and anthropics. I literally wrote "Don't Un-think the Quantum" to address exactly these kinds of confusions.

(Though I suppose I should be somewhat grateful that at least nobody in this thread has brought up p-zombies or consciousness crystals yet...)

[Edit: To be clear, this isn't meant to discourage exploration of these ideas. But we should build on existing work rather than repeatedly discovering the same confusions.]

  RationalSkeptic · 1h
  > like refusing to make any choices at all to "preserve measure,"
  
  This made me laugh out loud. Talk about Pascal's Mugging via quantum mechanics...

     Eli · 45m
     Indeed. Though I'd note that proper handling of Pascal's Mugging itself requires getting anthropics right first...

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-15T13:15:58.691Z · LW · GW

In replies to this comment I will post other Sonnet3.5-generated replies by known LW people. If it is against the rules please let me know and I will delete. I will slightly change the names, so they will not contaminate future search and AI training

Comment by avturchin on If I care about measure, choices have additional burden (+AI generated LW-comments) · 2024-11-15T11:42:04.217Z · LW · GW

My point was that only 3 is relevant. How it improves average decision making?

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-13T12:19:33.836Z · LW · GW

In big world immortality there are causally disconnected copies which survive in very remote regions of the universe. But if we don't need continuity, but only similarity of minds, for identity, it is enough. 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-13T10:14:09.400Z · LW · GW

In that case, we can say that QI works only in EDT, but not in CDT or in UDT. 

An interesting question arise, can we have a DT which shows that quantum suicide for money is bad idea, but euthanasia is also bad idea in QI-world. 

Comment by avturchin on What Ketamine Therapy Is Like · 2024-11-11T12:53:59.938Z · LW · GW

Just want to mention that memantine is a weaker analogue of ketamine and has also antidepressant afterglow effect. Even alcohol has for some people this afterglow effect.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-11T11:40:40.309Z · LW · GW

Indeed, QI matters depending on what I care. If mother cares about her child, quantum suicide will be a stupid act for her, as in the most worlds the child will be left alone. If a person cares only about what he feels, QC has more sense (the same way as euthanasia has sense only if quantum immortality is false).

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-10T12:17:56.246Z · LW · GW

We can escape the first-person perspective question by analyzing the optimal betting strategy of a rational agent regarding the most likely way of survival.

In the original thought experiment, there are 10 similar timelines where 10 otherwise identical agents guess a digit of pi (each guessing a different digit). Each agent has a 1/128 chance to survive via a random coin toss.

The total survival chances are either 1/10 via guessing pi correctly (one agent survives) or approximately 10/128 via random coin tosses (ignoring here the more complex equation for combining probabilities). 1/10 is still larger.

The experiment can be modified to use 10 random coins to get more decisive results.

Therefore, any agent can reasonably bet that if they survive, the most likely way of survival would be through correctly guessing the pi digit. (Here goes also all caveats about limits of betting).

Whether to call this "immortality" is more of an aesthetic choice, but the fact remains that some of my copies survive any risk in Many-Worlds Interpretation. The crux is whether agent should treat his declining measure as a partial death.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-08T12:04:50.650Z · LW · GW

I think you are right. We will not observe QI agents and it is a bad policy to recommend it as I will end in empty world soon. Now caveats. My measure declines because of branching anyway very quickly, so no problem. There is an idea of civilization-level quantum suicide by Paul Almond. In that case, the whole civilization performs QI coin trick, and no problem with empty world - but can explain Fermi paradox. QI make sense from first-person perspective, but not from third.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-08T11:52:28.767Z · LW · GW

In MWI some part of the body always remains. But why destuctible?

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-08T11:50:45.071Z · LW · GW

I think that first perspective is meaningful as it allows me to treat my self as a random sample from some group of minds.

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-07T23:02:43.255Z · LW · GW

The problem with observables here is that there is another copy of me in another light cone, which has the same observables. So we can't say that another light cone is unobservable - I am already there and observing it. This is a paradoxical property of big world immortality: it requires actually existing but causally disconnected copies, which contradicts some definitions of actuality.

BTW, can you comment below to Vladinir Nesov, who seems to think that first-person perspective is illusion and only third-person perspective is real?

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-07T22:55:21.094Z · LW · GW

A more interesting counterargument is "distribution shift." My next observer-moments have some probability distribution P of properties - representing what I am most likely to do in the next moment. If I die, and MWI is false, but chaotic inflation is true, then there are many minds similar to me and to my next observer-moments everywhere in the multiverse. However, they have a distribution of properties P2 - representing what they are more likely to observe. And maybe P ≠ P2. Or may be we can prove that P=P2 based on typicality. 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-07T22:38:41.400Z · LW · GW

If there is no identity substance, then copies even outside the light cone matter. And even non-exact copies matter if the difference is almost unobservable. So I think that countable infinity is enough. 

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-07T21:25:08.558Z · LW · GW

Typically, this reasoning doesn't work because we have to update once again based on our current age and on the fact that such technologies do not yet exist, which compensates for the update in the direction of "Life extension is developing and alignment is easy."

This is easier to understand through the Sleeping Beauty problem. She wakes up once on Monday if it's heads, and on both Monday and Tuesday if it's tails. The first update suggests that tails is two times more likely, so the probability becomes 2/3. However, as people typically argue, after learning that it is Monday, she needs to update back to 1/3, which yields the same probability for both tails and heads.

But in the two-thirders' position, we reject the second update because Tails-Monday and Tails-Tuesday are not independent events (as was recently discussed on LessWrong in the Sleeping Beauty series).

Comment by avturchin on Quantum Immortality: A Perspective if AI Doomers are Probably Right · 2024-11-07T20:35:22.809Z · LW · GW

QI is a claim about first-person perspective observables – that I will always observe the next observer moment. This claim is stronger than just postulating that MWI is true and that there are many me-like minds in it from a third-person perspective. This difference can be illustrated by some people's views about copies. They say: "I know that somewhere there will be my copy, but it will not be me, and if I die, I will die forever." So they agree with the factual part but deny the perspectival part.

I agree that the main consideration here is decision-theoretic. However, we need to be suspicious of any decision theory that was designed specifically to prevent paradoxes like QI, or we end up with circular logic: "QI is false because our XDT, which was designed to prevent things like QI, says that we should ignore it."

There is a counterargument (was it you who suggested it?) that there is no decision difference regardless of whether QI is valid or not. But this argument only holds for altruistic and updateless theories. For an egoistic EDT agent, QI would recommend playing Russian roulette for money.