LessWrong 2.0 Reader

It sure would.

Would you give 3:1 odds against a major player showing the untrusted model the old CoT, conditional on them implementing a "replace suspicious outputs" flow on a model with hidden CoT and publicizing enough implementation details to determine whether or not they did that?

It might be the case that AISC was extra late-skewed because the MATS rejection letters went out on the 14th (guess how I know) so I think a lot of people got those and then rushed to finish their AISC applications (guess why I think this) before the 17th. This would predict that the ratio of technical:less-technical applications would increase in the final few days.

Many people have no context in their life where they can get feedback on socially undesirable ideas from thoughtful people so that they can potentially update them. E.g. you hear socially undesirable thing online that you suspect has some truth to it, you can't have any reasonable discussion about which aspects might be true, which might be false, and even amongst the more true parts how to navigate having that belief or what would be a wholesome framework to use to work with it, bc no feedback.

I'll give an egregious example. At one time, iodizing salt in developing countries was opposed by some NGOs on the grounds that the argument that it raised IQ was some sort of fake racist thing. A person in that environment might have wanted to be able to discuss things in a safer space than whatever environment produced that insanity.

Yep! You might have to show it CoT from previous, non-resampled steps though

If OP were advocating banning normal parties, in favor of only having cancellable parties, I would agree with this comment.

What makes a discussion heavy? What requires that a conversation be conducted in a way that makes it heavy?

I feel like for a lot of people it just never has to be, but I'm pretty sure most people have triggers even if they're not aware of it and it would help if we knew what sets this off so that we can root them out.

Sadly for your friend, the hottest objects in the known universe are still astronomical rather than manmade. The LHC runs on the scale of 10 TeV (10^13 eV). The Auger observatory studies particles that start at 10^18 eV and go up from there.

Identifying and solving bootstrap problems for others could be a good way to locally perform effective altruism

You acknowledge the bug, but don't fully explain how to avoid it by putting EVs before Ps, so I'll elaborate slightly on that:

This way, they [the simulators] can influence the predictions of entities like me in base Universes

This is the part where we can escape the problem as long as our oracle's goal is to give accurate answers to its makers in the base universe, rather than to give accurate probabilities wherever it is. Design it correctly, and it will be indifferent to its performance in simulations and wont regard them.

Don't make pure oracles, though. They're wildly misaligned. Their prophecies will be cynical and self-fulfilling. [LW · GW] (can we please just solve the alignment problem instead)

This means that my probabilities about the fundamental nature of reality around me change minute by minute, depending on what I'm doing at the moment. As I said, probabilities are cursed.

My fav moments for having absolute certainty that I'm not being simulated is when I'm taking a poo. I'm usually not even thinking about anything else while I'm doing it, and I don't usually think about having taken the poo later on. Totally inconsequential, should be optimized out. But of course, I have no proof that I have ever actually been given the experience of taking a poo or whether false memories of having experienced that^[1] are just being generated on the fly right now to support this conversation.

Please send a DM to me first before you do anything unusual based on arguments like this, so I can try to explain the reasoning in more detail and try to talk you out of bad decisions.

You can also DM me about that kind of thing [LW · GW].
 

1. ^{^}

   Note, there is no information in the memory that tells you whether it was really ever experienced, or whether the memories were just created post-hoc. Once you accept this, you can start to realise that you don't have that kind of information about your present moment of existence either. There is no scalar in the human brain that the universe sets to tell you how much observer-measure you have. I do not know how to process this and I especially don't know how to explain/confess it to qualia enjoyers.

It would be crazy to show the untrusted model the old CoT when resampling.