LessWrong 2.0 Reader

QI is a claim about first-person perspective observables – that I will always observe the next observer moment. This claim is stronger than just postulating that MWI is true and that there are many me-like minds in it from a third-person perspective. This difference can be illustrated by some people's views about copies. They say: "I know that somewhere there will be my copy, but it will not be me, and if I die, I will die forever." So they agree with the factual part but deny the perspectival part.

I agree that the main consideration here is decision-theoretic. However, we need to be suspicious of any decision theory that was designed specifically to prevent paradoxes like QI, or we end up with circular logic: "QI is false because our XDT, which was designed to prevent things like QI, says that we should ignore it."

There is a counterargument (was it you who suggested it?) that there is no decision difference regardless of whether QI is valid or not. But this argument only holds for altruistic and updateless theories. For an egoistic EDT agent, QI would recommend playing Russian roulette for money.

Thanks for writing this up! Sad to have missed this sprint. This comment mainly has pushback against things you've said, but I agreed with a lot of the things I'm not responding to here.

Second, there is evidence that CoT does not help the largest LLMs much.

I think this is clearly wrong, or at least way too strong. The most intuitively obvious way I've seen this is reading the cipher example in the o1 blog post, in the section titled "Chain of Thought." If you click on where it says "Thought for 5 seconds" for o1, it reveals the whole chain of thought. It's pretty long, maybe takes 5 mins to skim, but it's well worth the time for building intuition about how the most cutting edge model thinks imo. The model uses CoT to figure out a cipher and decode it. I think it's intuitively obvious that the model could not have solved this problem without CoT.

Additionally, when trying to search for this paper, I found this paper on arxiv which finds situations where the CoT is just rationalizing the decision made by the LLM. If you look at papers which cite this paper, you will find other research in this vain.

True. I trust post-hoc explanations much less than pre-answer reasoning for problems that seem to require a lot of serial reasoning, like the o1 cipher problem. This post [LW · GW] and this comment [LW(p) · GW(p)] on it discuss different types of CoT unfaithfulness in a way similar to how I'm thinking about it, highly recommend.

But why are the Aether team organising these mini-sprints? The short summary is that deception is a big risk in future AI systems, and they believe that nailing down what it means for LLMs and LLM agents to believe something is an important step to detecting and intervening on deceptive systems.

Fwiw only that one sprint was specifically on beliefs. I think I'm more interested in what the agents believe, and less in figuring out exactly what it means to believe things (although the latter might be necessary in some confusing cases). I'd say the sprints are more generally aimed at analyzing classic AI risk concepts in the context of foundation model agents, and getting people outside the core team to contribute to that effort.

Another caveat:

• I am believable and have expertise in very few major life skills, and possibly don't have expertise in the thing you're asking advice for.

Related note: I think developing the skill of identifying believability and expertise is very powerful (though I have only been applying said skill for a couple of years explicitly; caveat emptor, lol.)

Here's Cedric Chin outlining believability defined by Ray Dalio: 

Technique summary:
Believable people are people who have 1) a record of at least three relevant successes and 2) have great explanations of their approach when probed.

You may evaluate a person's believability on the subject matter at hand by applying this heuristic. When interacting with them:

1. If you’re talking to a more believable person, suppress your instinct to debate and instead ask questions to understand their approach. This is far more effective in getting to the truth than wasting time debating.
2. You’re only allowed to debate someone who has roughly equal believability compared to you.
3. If you’re dealing with someone with lower believability, spend the minimum amount of time to see if they have objections that you’d not considered before. Otherwise, don’t spend that much time on them.

Here's Gary Klein, founder of Naturalistic Decision Making, outlining seven dimensions of expertise:

We want pragmatic guidelines for deciding which if any purported experts to listen to when making a difficult and important decision. How can we know who is really credible?

Bottom line: We cannot know for sure. There are no iron-clad criteria.

However, there are soft criteria, indicators we can pay attention to. I have identified seven so far, drawing on papers such as Crispen & Hoffman, 2016, and Shanteau, 2015, and on suggestions by Danny Kahneman and Robert Hoffman. Even though none of these criteria are fool-proof, all of them seem useful and relevant:

(a) Successful performance—measurable track record of making good decisions in the past. (But with a large sample, some we do very well just by luck, such as stock-pickers who have called the market direction accurately in the past 10 years.)

(b) Peer respect. (But peer ratings can be contaminated by a person’s confident bearing or fluent articulation of reasons for choices.)

(c) Career—number of years performing the task. (But some 10-year veterans have one year of experience repeated 10 times and, even worse, some vocations do not provide any opportunity for meaningful feedback.)

(d) Quality of tacit knowledge such as mental models. (But some experts may be less articulate because tacit knowledge is by definition hard to articulate.)

(e) Reliability. (Reliability is necessary but not sufficient. A watch that is consistently one hour slow will be highly reliable but completely inaccurate).

(f) Credentials—licensing or certification of achieving professional standards. (But credentials just signify a minimal level of competence, not the achievement of expertise.)

(g) Reflection. When I ask "What was the last mistake you made?" most credible experts immediately describe a recent blunder that has been eating at them. In contrast, journeymen posing as experts typically say they can't think of any; they seem sincere but, of course, they may be faking. And some actual experts, upon being asked about recent mistakes, may for all kinds of reasons choose not to share any of these, even ones they have been ruminating about. So this criterion of reflection and candor is not any more foolproof than the others.

Yes, lots of socioeconomic problems have been solved on a 5 to 10 year timescale.

I also disagree that problems will become moot after the singularity unless it kills everyone-- the US has a good chance of continuing to exist, and improving democracy will probably make AI go slightly better.

But red exists only because we have the experience of seeing red. That is, red exists because we have found it useful to tell red apart from other colors. We can "objectively" define red to be light with a wavelength between 620 and 750 nanometers, but we define it thus because those wavelengths correspond to what many people subjectively identify as red. Thus, whether or not an apple is red is neither a properly objective nor subjective fact, but intersubjective knowledge that depends on both the world and how we experience it. So it goes for all truth that can be known.

At first glance, red seems like such a special color to me. It's the color of blood and many fruit, the advanced color that primates and flying animals see and other animals can't distinguish, it's the most primitive step towards heat vision, and it's the color at the lowest end of the range of common electron bandgap. Obviously the word itself is kind of arbitrary but the color seems as non-arbitrary as could be.

Thanks. No I didn't. (And probably don't have time to look into it but still nice to know.)

If quantum immortality is true...

To discuss truth of a claim, it's first crucial to clarify what it means. What does it mean for quantum immortality to be true or not? The only relevant thing that comes to mind is whether MWI is correct. Large quantum computers might give evidence to that claim (though ASI very likely will be here first, unless there is a very robust AI Pause).

Once we know there are physical branching worlds, there is no further fact of "quantum immortality" to figure out. There are various instances of yourself in various world branches, a situation that doesn't seem that different from multiple instances that can occur within a single world. Decision theory then ought to say how to weigh the consequences of possible influences and behaviors spread across those instances.

If QI is false, it must have theoretical cost:

Either:

• Universe is finite and no MWI
• Personal identity is based on some kind of destructible soul and my copy is not me
• We use some variant of updateless decision theory, especially designed to prevent this type of problems (which is rather circular counterargument)

As the prediction markets on Trump winning went from ~50% to ~100% over 6 hours, S&P 500 futures moved less than the rest of the time. Why?

Really interesting post! Out of curiosity, have you ever read Star Maker? This is basically one of the  civilizations he came up with, a symbiotic setup where the land-based crab-like animals provide the tool using intelligence and the aquatic whale-like animals provide the theoretical intelligence.