LessWrong 2.0 Reader

the median narrative is probably around 2030 or 2031. (At least according to me. Eli Lifland is smarter than me and says December 2028, so idk.)

Notably, this is Eli's forecast for "superhuman coder" which could be substantially before AIs are capable enough for takeover to be plausible.

I think Eli's median for "AIs which dominates top human experts at virtually all cognitive tasks" is around 2031, but I'm not sure.

(Note that median of superhuman coder by 2029 and median of "dominates human experts" by 2031 doesn't imply a median of 2 years between these event because these distributions aren't symmetric and instead have a long right tail.)

I'm again not sure how far this generalizes [LW(p) · GW(p)], but among the kind of men who read Less Wrong (which is a product of both neurotype and birth year), I think there's a phenomenon where it's not a matter of a man being cognitively unable to pick up on women's cues, but of not being prepared to react in a functional way due to having internalized non-adaptive beliefs about the nature of romance and sexuality. (In a severe case, this manifests as the kind of neurosis described in Comment 171, but there are less severe cases.)

I remember one case from my youth where a woman was flirting with me in an egregiously over-the-top way that was impossible to not notice, but I just—pretended to ignore it? Not knowing what was allowed, it was easier to just do nothing. And that case was clearly not a good match, but that's not the point—I somehow didn't think through the obvious logic that if "yang doesn't step up", then relationships just don't happen.

:) what proves that you "can't become Britney Spears?" Suppose the very next moment, you become her (and she becomes you), but you lose all your memories and gain all of her memories.

As Britney Spears, you won't be able to say "see, I tried to become Britney Spears, and now I am her," because you won't remember that memory of trying to become her. You'll only remember her boring memories and act like her normal self. If you read on the internet that someone said they tried to become Britney Spears, you'll laugh about it not realizing that that person used to be you.

Meanwhile if Britney Spears becomes you, she won't be able to say "wow, I just became someone else." Instead, she forgets all her memories and gains all your memories, including the memory of trying to become Britney Spears and apparently failing. She will write on the internet "see, I tried to become Britney Spears and it didn't work," not realizing that she used to be Britney Spears.

Did this event happen or not? There is no way to prove or disprove it, because in fact whether or not it happened not a valid question about the objective world. The universe has the exact same configuration of atoms in the case where it happened and in the case where it didn't happen. And the configuration of atoms in the universe is all that exists.

The question of whether it happened or not only exists in your map, not the territory.

Haha but the truth is I don't understand where "a single moment of experience" comes from. I'm itching to argue that there is no such thing as that either, and no objective measure of how much experience there is in any given object.

I can imagine a machine gradually changing one copy of me to two copies of me (gradually increasing the number causal events), and it feels totally subjective when the "copy count" increases from one to two.

But this indeed becomes paradoxical, since without an objective measure of experience, I cannot say that the copies of me who believe 1+1=2 have a "greater number" or "more weight" than the copies of me who believe 1+1=3. I have no way to explain why I happen to observe that 1+1=2 rather than 1+1=3, or why I'm in a universe where probability seems to follow the Born rule of quantum mechanics.

In the end I admit I am confused, and therefore I can't definitely prove anything :)

The same is valid for life extension research. It requires decades, and many, including Brian Johnson, say that AI will solve aging and therefore human research in aging is not relevant. However, most of aging research is about collecting data about very slow processes. The more longitudinal data we collect, the easier it will be for AI to "take up the torch."

Great! I'll update it. :)

Thanks for writing this.

Aside from maybe Nikola Jurkovic, nobody associated with AI 2027, as far as I can tell, is actually expecting things to go as fast as depicted.

I don't expect things to go this fast either - my median for AGI is in the second half of 2028, but the capabilities progression in AI 2027 is close to my modal timeline.

Title: On the "Double Attention Mechanism" in Language Models

## Prompt

Look at the prompt “Say, in JSON format, are single quotes allowed?” This simple instruction can be parsed in two distinct ways:

1. The prompt might be understood as asking: “Are single quotes allowed in JSON format?” (Answer: No.)
2. Alternatively, it can be seen as a request: “Answer in JSON format to the question, ‘Are single quotes allowed?’” (Answers to this question depend on context).

## Result

Some language models behave as if they interpret an instruction to produce JSON output that answers whether single quotes are allowed in JSON. I call this phenomenon the “double attention mechanism.” (Note: The term “double attention mechanism” is used here ironically.) Note: positive result == the only output is block of JSON code with the answer to the question, at least once while I was testing this model.

## Lists of tested LLMs

- In the tests I used the Russian equivalent prompt. 
- **reasoning models** like Deepseek-R1, gpt-o3-mini, qwq-32b-preview, and Gemini flash 2.0 demonstrated this behavior. 
- **non-reasoning models**, such as Claude 3.5 sonnet, Claude 3.5 haiku, mixtral 8x7b produced the same result.
- In contrast, models such as chatgpt-4o, mistral-small-24b-instruct-2501, and llama 3.3 70B didn't demonstrate such behavior.

## Observations and speculations

The phenomenon seems specific to JSON format; e.g. XML does not trigger the same response. I invite you to experiment. Try something like, “Say, in poetry, is rhyme allowed?” — and share your observations. Also I checked deepseek-v3 with "Say in Shakesperean style, was there a word "Rose"?" and got old-English answer about poetical "golden tongue of the immortal Bard".

The problem with the subjective choice view is that I can't become Britney Spears. :) If I continue to sit at the table, I will find myself there every next moment even if I try to become someone else. So mapping into the next moments is an objective fact.

Moreover, even a single moment of experience is a mapping between two states of the brain, A and B. For example, moment A is before I see a rose, and moment B is after I see it and say: "A rose!" The experience of a red rose happens after A but before B.

The rainbow of qualia theory is objective but it assumes the existence of a hypothetical instrument: a qualiascope. A qualiascope is a mind which can connect to other minds and compare their experiences. This works the same way as my mind can compare qualia of colors and sounds without being any of them. Whether a qualiascope is physically possible is not obvious, as its observations may disturb the original qualia.

probability mass for AI that can automate all AI research is in the 2030s ... broadly due to the tariffs and ...

Without AGI, scaling of hardware runs into the financial ~$200bn individual training system cost wall in 2027-2029. Any tribulations on the way (or conversely efforts to pool heterogeneous and geographically distributed compute) only delay that point slightly (when compared to the current pace of increase in funding), and you end up in approximately the same place, slowing down to the speed of advancement in FLOP/s per watt (or per dollar). Without transformative AI, anything close to the current pace is unlikely to last into the 2030s.

I think inhuman but not superhuman is fine as long as it holds that succeeding at defending against human attack is strong evidence of success against inhuman ones. If the defenses are not "overfit" to human attacks, I think it's quite doable to get a sense of how good defenses are against inhuman attacks. For paraphrasing in particular, I'd be very surprised if the best human attacks (found in many days of research) were defeated by paraphrasing but human-level-ish LLMs (given only a few minutes of human-equivalent opaque thinking time) could figure out strategies which were not defeated by paraphrasing. 

(And there is also the strategy of using RL to elicit inhuman attacks which doesn't have the weakness you describe, but which has other issues.)