Truth vs Utility

obtaining the truth is the rationalists' ultimate goal

Nope. It's an instrumental goal. We just believe it to be very useful, because in nontrivial situations it is difficult to find a strategy to achieve X without having true beliefs about X.

Are there scenarios in which a rationalist should actively try to avoid the truth to maximize their possible utility?

Omega tells you: "Unless you start believing in horoscopes, I will torture all humans to death." (Or, if making oneself believe something false is too difficult, then something like: "There is one false statement in your math textbook, and if you even find out which one it is, I will torture all humans to death." In which case I would avoid looking at the textbook ever again.)

Option 2 is Omega will answer one question on absolutely any subject truthfully pertaining to our universe with no strings attached. You can ask about the laws governing the universe, the meaning of life, the origin of time and space, whatever and Omega will give you a absolutely truthful, knowledgeable answer.

I guess it would depend on how much would I trust myself to ask a question what could bring me even more benefit than option 1. For example: "What is the most likely way that I could become Omega-powerful without losing my values? (Most likely = relative to my current situation and abilities.)" Because a lucky answer on this one could be even better than the first option. -- So it comes to an estimate about whether such lucky answer exists, what is my probability to follow the strategy successfully if I get the answer, and what is my probability to ask the question correctly. Which I admit I don't know.

As remarked many times on this site and elsewhere, if you are given evidence that Omega is capable of simulating an environment as rich as our observed Universe, you should apply the Copernican principle and assign high probability that our world is not special and is already a simulation. The Matrix-like dualism (real/simulated) is a very low-probability alternative, which only seems likely because we are used to anthropocentrically thinking of our world as "real".

Once you realize that, Option 1 becomes "pick a different simulation" and Option 2 "improve current simulation".

As you can't know is current reality simulation or not, you could propose 50-50 probability as you have no anchor to operate on.

a)Current reality is a simulation ran by superintellegent computers. In this case you shouldn't hesitate even for a second - you are obviously going for option 1 where you will live in a better world

b)You are living in reality. If given reality is not a simulation your utility function in simulation will differ from your real needs and wants, there you should ask a question that will maximize your real utility which can be how to bring peace to earth, bring down inequality and lift happiness level and maybe even kill death.

But actually we have some evidence of the world we live in, because supercomputer came to you and gave you two choices, it doesn't somehow fit for what we expect to see in our daily routine. It means that supercomputers exists and probability that you live in simulation is close to 100% But in fact it is likely that Omega doesnt run simulation because you can speak to it and see it, it is more likely that there is another more powerful supercomputer out there who rans simulation on this simulation. You should choose option 1

This cries fr a poll. To make this into a more balanced question I changed the "simulation" variant into something more 'real':

Suppose Omega, a supercomputer, comes down to Earth to offer you a choice:

[pollid:750]

Making the assumption that since #2 comes with 'No strings attached' it is implying safety measures such as 'The answer does not involve the delivery of a star sized super computer that kills you with it's gravity well' since that feels like a string, and #1 does not have such safety measures (implying you have infinite utility because you have been turned into a paperclipper in simulated paperclippium is an interpretation), I find myself trying to ponder ways of getting the idealized results of #1 with the safety measures of #2, such as

"If you were willing to answer an unlimited number of questions, and I asked you all the questions I could think of, What are all question answer pairs where I would consider any set of those question answer pairs a net gain in utility, answered in order from highest net gain of utility to smallest net gain of utility?"

Keeping in mind that the questions such as the below would be part of the hilariously meta above question:

"Exactly, in full detail without compression and to the full extent of time, what would all of my current and potentially new senses experience like if I took the simulation in Option 1?"

It was simply an idea that I found interesting that I wanted to put into writing. Thank you for reading.

This was an interesting idea to read! (Even if I don't think my interpretation was what you had in mind.) Thank you for writing!

Option 1 is to live in a stimulated world where you have infinite utility

No. First, if you want to learn about us, you don't get to define our utility functions. Beings with utility functions you make up are made up beings.

Second, there is a distinction that I believe EY makes which seems to me a good one, which I think you are mistaking. Utility functions may be functions on the world, not only on the state of our feelings or experience. Telling me how wonderful I will feel in a simulation does not ping my preferences over reality beyond my feelings.

This is a fundamental error Sam Harris makes, thinking we only care about conscious experience. Conscious experience may be how we experience caring, but it is not necessarily the only object of our caring.

I accept Option 1. Anything less would be a failure of instrumental rationality.

Let me propose a scenario to illustrate the conundrum. Suppose Omega

I don't think I shall. But for what it's worth, I would reject Option 1 even if the alternative was just the status quo.