Across all websites I've been on in my life, I have posted more than 100000 comments (resulting in many interactions), so while things like psychoanalyzing people, assuming intentions, and making stereotypes is "bad", I simply have too much training data, and too few incorrect guesses not to do this.

On the contrary, your guess did not take context into account and was bad. They were downvoted for answering in a way which didn't answer the question, had many typos, and otherwise took more effort to read than the information it contained was worth.

Your comment "added more heat than light" for no reason, with no prompting. I explicitly am only making this comment because you posted a long paragraph explaining why you are so sure that you did a good job analyzing when it looks like you did a very poor one. Perhaps people in the past have not given useful feedback, so I will give a short piece: do not do this psycho-analysis until you have generated at least 2 alternate theories for what happened.

I've learned a weird amount about whales from here this week.

If unsupervised translation is possible for creatures with a language as different from ours as whales, that would be amazing. Especially if it could be done without monitoring their behaviors (although that might be asking for too much)

Typically, this reasoning doesn't work because we have to update once again based on our current age and on the fact that such technologies do not yet exist, which compensates for the update in the direction of "Life extension is developing and alignment is easy."

This is easier to understand through the Sleeping Beauty problem. She wakes up once on Monday if it's heads, and on both Monday and Tuesday if it's tails. The first update suggests that tails is two times more likely, so the probability becomes 2/3. However, as people typically argue, after learning that it is Monday, she needs to update back to 1/3, which yields the same probability for both tails and heads.

But in the two-thirders' position, we reject the second update because Tails-Monday and Tails-Tuesday are not independent events (as was recently discussed on LessWrong in the Sleeping Beauty series).

The note that it isn't what the username is for is kind of interesting. There are two places to put information on social media: your account and the media attached to your account. Here, instead of attaching some GWWC information to every piece of media (Tweet/post/picture/essay etc), the idea is to attach it to the account, but that really does feel quite weird to me.

I'm glad you said that.

I think it can be both rational to doubt his edge and not trade on it.

Useful clarification and thanks for writing this up!

Inspired by and building on this, I decided to clean up some thoughts of my own in a similar direction. Here they are on my short forum: What are the actual use cases of memory systems like Anki? [LW(p) · GW(p)]

A person is a complicated machine, we can observe how this machine develops or could develop through processes that we could set up in the world or hypothetically. This is already quite clear, and things like "first person perspective" or "I will observe" don't make this clearer.

So I don't see a decision theory proclaiming "QI is false!", it's just not a consideration it needs to deal with at any point, even if somehow there was a way of saying more clearly what that consideration means. Like a chip designer doesn't need to appreciate the taste of good cheese to make better AI accelerators.

What are the actual use cases of memory systems like Anki?

Epistemic status: spent 30min cleaning up some notes from my Obsidian I jotted down yesterday. This ontology is rough and a bit illegible but potentially useful for narrowing down the actual use cases of memory systems.

Inspired by @Saul Munn [LW · GW]'s recent short form: Active Recall and Spaced Repetition are Different Things [LW(p) · GW(p)]. The concepts of active recall and spaced repetition apply pretty well here, but I saw Saul's post after writing most of the text below.

Roughly, there are types of knowledge in domains (recalling from Scott Young's Ultralearning, I might be slightly off):

• Proceduralized knowledge
• Conceptual knowledge
• Factual knowledge

I think spaced repetition systems are useful for three types of domains based on the nature of the cues in the domains:

1. Domains where you are already cued on the learnings often (e.g., key business procedures in your full-time job). You naturally get spaced repetition in these domains, so memory systems like Anki are less useful.
2. Domains where you are not cued on the learnings often or at properly spaced intervals. This may include domains where you were an expert at one point and then stopped being an expert.
3. Domains where you need to understand cues before being effective. (e.g. understanding a new language or technical domain like quantum mechanics; see Quantum Country.)

Combining the two frameworks above, Anki is useful for:

• Proceduralized knowledge where you can appropriately cue yourself and you're not cued often on.
  • Cueing procuduralized knowledge is hard, though. For example, if I wanted to review my knowledge of 'how-to-play-beach-volleyball' on a regular basis, I have to spend a time coordinating or attending an event.
  • Though, some proceduralized knowledge is possible to cue: for example, I have been using Anki to review LeetCode problems.
• Conceptual and factual knowledge that you're (2 above) not cued often on or (3 above) need to meet a criteria of knowing.
  • Conceptual and factual knowledge is much easier than procedural knowledge because these types of knowledge neatly fit in flashcards. I'm still not sure about the strict boundary between conceptual and factual knowledge, though.
  • And I think the general heuristic that conceptual knowledge is harder to cue than factual knowledge is true: medical students are obsessed with Anki while students in other vocational schools (e.g. law) don't seem to be

Accordingly, memory systems may not be useful for:

• Domains where you are cued on information often enough to get natural spaced repetition.
• Are highly procedural in a way that can't be Ankified and therefore active recall is hard.

QI is a claim about first-person perspective observables – that I will always observe the next observer moment. This claim is stronger than just postulating that MWI is true and that there are many me-like minds in it from a third-person perspective. This difference can be illustrated by some people's views about copies. They say: "I know that somewhere there will be my copy, but it will not be me, and if I die, I will die forever." So they agree with the factual part but deny the perspectival part.

I agree that the main consideration here is decision-theoretic. However, we need to be suspicious of any decision theory that was designed specifically to prevent paradoxes like QI, or we end up with circular logic: "QI is false because our XDT, which was designed to prevent things like QI, says that we should ignore it."

There is a counterargument (was it you who suggested it?) that there is no decision difference regardless of whether QI is valid or not. But this argument only holds for altruistic and updateless theories. For an egoistic EDT agent, QI would recommend playing Russian roulette for money.

Thanks for writing this up! Sad to have missed this sprint. This comment mainly has pushback against things you've said, but I agreed with a lot of the things I'm not responding to here.

Second, there is evidence that CoT does not help the largest LLMs much.

I think this is clearly wrong, or at least way too strong. The most intuitively obvious way I've seen this is reading the cipher example in the o1 blog post, in the section titled "Chain of Thought." If you click on where it says "Thought for 5 seconds" for o1, it reveals the whole chain of thought. It's pretty long, maybe takes 5 mins to skim, but it's well worth the time for building intuition about how the most cutting edge model thinks imo. The model uses CoT to figure out a cipher and decode it. I think it's intuitively obvious that the model could not have solved this problem without CoT.

Additionally, when trying to search for this paper, I found this paper on arxiv which finds situations where the CoT is just rationalizing the decision made by the LLM. If you look at papers which cite this paper, you will find other research in this vain.

True. I trust post-hoc explanations much less than pre-answer reasoning for problems that seem to require a lot of serial reasoning, like the o1 cipher problem. This post [LW · GW] and this comment [LW(p) · GW(p)] on it discuss different types of CoT unfaithfulness in a way similar to how I'm thinking about it, highly recommend.

But why are the Aether team organising these mini-sprints? The short summary is that deception is a big risk in future AI systems, and they believe that nailing down what it means for LLMs and LLM agents to believe something is an important step to detecting and intervening on deceptive systems.

Fwiw only that one sprint was specifically on beliefs. I think I'm more interested in what the agents believe, and less in figuring out exactly what it means to believe things (although the latter might be necessary in some confusing cases). I'd say the sprints are more generally aimed at analyzing classic AI risk concepts in the context of foundation model agents, and getting people outside the core team to contribute to that effort.