Would I think for ten thousand years?

Why do you think this problem needs to be solved now? Couldn't the idealized version of yourself spend the first few years to figure out how best to protect again value drift during the rest of the available time? It seems to me that a more urgent problem is, given that a person thinking alone for even a few years would likely go crazy, how do we set up the initial social dynamics for a group of virtual humans?

In most cases my thought is "well, what's the alternative?"

I'm either doing what I would have done after thinking for N years, or I'm committing to a course of action after thinking less than N years. The former risks value drift, the latter risks... well, not having had as much time to think, which isn't obviously better than value drift.

I do think there's a few variations that seem like improvements, like:

• run X copies of myself, slightly randomizing their starting conditions, running them for a range of times (maybe as wide as "1 week" to "10,000 years". Before revealing their results to me, reveal how convergent they were. If there's high convergence I'm probably less worried about the answer. )
• make sure simulated me can only think about certain classes of things ("solve this problem with these constraints"). I'm more worried about value drift from "10,000 year me who lived life generally" than "10,000 year me who just thought about this one problem. Unless the problem was meta-ethics, in which case I probably want some kind of value drift."

What more worry me is not a value drift, but the hardening of values in the wrong position.

We could see examples of people whose values have formed during their youth, and these values didn't evolve in the older age but instead become a rigid self-supporting system, not connected with reality. These old-schoolers don't have any new wisdom to tell.

Obviously, brain aging plays a role here, but it is not only a cellular aging, but also an "informational aging", that is, in particular, hardening of pavlovian reflexes between thoughts. Personally, I found that I have the same thought in my mind every time I am start eating, which is annoying ( it is not related to food: it is basically a number).

People will have to do a lot of maths and philosophy to get an AI system that works at all.

Suppose you have a lead of 1 week over any ufai projects, and you have your AI system to the point where it can predict what you would do in a box. (Actually, we can say the AI has developed mind uploading tech + lotsa compute) The human team needs say 5 years of thinking to come up with better metaethics, defense against value drift or whatever. You want to simulate the humans in some reasonably human friendly environment for a few years to work this thing out. You pick a nice town, and ask the AI to create a virtual copy of the town. (More specifically, you randomly sample from the AI's probability distribution, after conditioning on enough data that the town will be townlike.) The virtual town is created with no people except the research team in it. All the services are set to work without any maintenance. (Water in virtual pipes, food in virtual shops, virtual internet works.). The team of people uploaded into this town is at least 30, ideally a few hundred, including plenty of friends and family.

This "virtual me in a box" seems likely to be useful and unlikely to be dangerous. I agree that any virtual box trick that involves people thinking for a long time compared to current lifespans is dangerous. A single person trapped in low res polygon land would likely go crazy from the sensory deprivation.

You need an environment with a realistic level of socializing and leisure activities to support psycologically healthy humans. Any well done "virtual me in a box" is going to look more like a virtual AI safety camp or research department than 1 person in a blank white room containing only a keyboard.

Unfortunately, all those details would be hard to manually hard code in. You seem to need an AI that can be trusted to follow reasonably clear and specific goals without adversarial optimization. You want a virtual park, manually creating it would be a lot of hard work, see current video games. You need an AI that can fill in thousands of little details in a manor not optimized to mess with humans. This is not an especially high bar.

Also, the thinking about own meaning of life is an important part of human activity, and if we delegate this to an AI, we will destroy significant part of human values. In other words, if AI will make philosophers unemployed, they will not be happy. Or else, we will not accept the AI's answer, and will continue to search for the ultimate goal.

One more thought: the more one argues for the creation for her own copies for some task, the more one should think that she is already is such task-solving simulation. Welcome to our value-solving matrix!