LessWrong 2.0 Reader

The same is valid for life extension research. It requires decades, and many, including Brian Johnson, say that AI will solve aging and therefore human research in aging is not relevant. However, most of aging research is about collecting data about very slow processes. The more longitudinal data we collect, the easier it will be for AI to "take up the torch."

Great! I'll update it. :)

Thanks for writing this.

Aside from maybe Nikola Jurkovic, nobody associated with AI 2027, as far as I can tell, is actually expecting things to go as fast as depicted.

I don't expect things to go this fast either - my median for AGI is in the second half of 2028, but the capabilities progression in AI 2027 is close to my modal timeline.

Title: On the "Double Attention Mechanism" in Language Models

## Prompt

Look at the prompt “Say, in JSON format, are single quotes allowed?” This simple instruction can be parsed in two distinct ways:

1. The prompt might be understood as asking: “Are single quotes allowed in JSON format?” (Answer: No.)
2. Alternatively, it can be seen as a request: “Answer in JSON format to the question, ‘Are single quotes allowed?’” (Answers to this question depend on context).

## Result

Some language models behave as if they interpret an instruction to produce JSON output that answers whether single quotes are allowed in JSON. I call this phenomenon the “double attention mechanism.” (Note: The term “double attention mechanism” is used here ironically.) Note: positive result == the only output is block of JSON code with the answer to the question, at least once while I was testing this model.

## Lists of tested LLMs

- In the tests I used the Russian equivalent prompt. 
- **reasoning models** like Deepseek-R1, gpt-o3-mini, qwq-32b-preview, and Gemini flash 2.0 demonstrated this behavior. 
- **non-reasoning models**, such as Claude 3.5 sonnet, Claude 3.5 haiku, mixtral 8x7b produced the same result.
- In contrast, models such as chatgpt-4o, mistral-small-24b-instruct-2501, and llama 3.3 70B didn't demonstrate such behavior.

## Observations and speculations

The phenomenon seems specific to JSON format; e.g. XML does not trigger the same response. I invite you to experiment. Try something like, “Say, in poetry, is rhyme allowed?” — and share your observations. Also I checked deepseek-v3 with "Say in Shakesperean style, was there a word "Rose"?" and got old-English answer about poetical "golden tongue of the immortal Bard".

The problem with the subjective choice view is that I can't become Britney Spears. :) If I continue to sit at the table, I will find myself there every next moment even if I try to become someone else. So mapping into the next moments is an objective fact.

Moreover, even a single moment of experience is a mapping between two states of the brain, A and B. For example, moment A is before I see a rose, and moment B is after I see it and say: "A rose!" The experience of a red rose happens after A but before B.

The rainbow of qualia theory is objective but it assumes the existence of a hypothetical instrument: a qualiascope. A qualiascope is a mind which can connect to other minds and compare their experiences. This works the same way as my mind can compare qualia of colors and sounds without being any of them. Whether a qualiascope is physically possible is not obvious, as its observations may disturb the original qualia.

probability mass for AI that can automate all AI research is in the 2030s ... broadly due to the tariffs and ...

Without AGI, scaling of hardware runs into the financial ~$200bn individual training system cost wall in 2027-2029. Any tribulations on the way (or conversely efforts to pool heterogeneous and geographically distributed compute) only delay that point slightly (when compared to the current pace of increase in funding), and you end up in approximately the same place, slowing down to the speed of advancement in FLOP/s per watt (or per dollar). Without transformative AI, anything close to the current pace is unlikely to last into the 2030s.

I think inhuman but not superhuman is fine as long as it holds that succeeding at defending against human attack is strong evidence of success against inhuman ones. If the defenses are not "overfit" to human attacks, I think it's quite doable to get a sense of how good defenses are against inhuman attacks. For paraphrasing in particular, I'd be very surprised if the best human attacks (found in many days of research) were defeated by paraphrasing but human-level-ish LLMs (given only a few minutes of human-equivalent opaque thinking time) could figure out strategies which were not defeated by paraphrasing. 

(And there is also the strategy of using RL to elicit inhuman attacks which doesn't have the weakness you describe, but which has other issues.)

Thank you for posting this. Are there any opportunities for students about to graduate to apply themselves, particularly without a C.S background? My undergraduate experience was focused on Business and IR (Cold War history, Sino-U.S relations) before I pivoted my long term focus to AI safety policy, and it's been difficult to find good entry points for EA work in this field as a new grad.

I've been monitoring 80,000 hours and applying to research fellowships where I can so far, but I'm always looking for new positions. If you or anyone else knows an org looking to onboard some fresh talent, I'd be happy to help.

Benchmark Readiness Level

Safety-relevant properties should be ranked on a "Benchmark Readiness Level" (BRL) scale, inspired by NASA's Technology Readiness Levels. At BRL 4, a benchmark exists; at BRL 6 the benchmark is highly valid; past this point the benchmark becomes increasingly robust against sandbagging. The definitions could look something like this:

Here's a draft list of properties we could apply the BRL to, including capabilities and propensity:

• General capabilities
  • Software ability (HCAST, others)
  • Knowledge (Humanity's Last Exam, others)
• Sandbagging
  • Situational awareness
  • Alignment faking
  • Sandbagging ability
• Monitorability
  • Steganographic collusion
  • Neuralese
  • Faithful CoT
  • Ability to monitor smarter AIs
• Research acceleration
  • % LoC written by AIs
  • AI R&D Uplift
• Escape risk
  • Adversarial competence
  • Self-exfiltration
  • Cyber offense
• Misalignment
  • Reward hacking
  • Misaligned goals
• Other Dangers
  • CBRN
  • Persuasion

Is this doomed? Am I missing anything important?

Thank you for your comments. :)

you have not shown that using AI is equivalent to slavery

I'm assuming we're using the same definition of slavery; that is, forced labour of someone who is property. Which part have I missed?

In addition, I feel cheated that you suggest spending one-fourth of the essay on feasibility of stopping the potential moral catastrophe, only to just have two arguments which can be summarized as "we could stop AI for different reasons" and "it's bad, and we've stopped bad things before".
(I don't think a strong case for feasibility can be made, which is why I was looking forward to seeing one, but I'd recommend just evoking the subject speculatively and letting the reader make their own opinion of whether they can stop the moral catastrophe if there's one.)

To clarify: Do you think the recommendations in the Implementation section couldn't work, or that they couldn't become popular enough to be implemented? (I'm sorry that you felt cheated.)

in principle, we have access to any significant part of their cognition and control every step of their creation, and I think that's probably the real reason why most people intuitively think that LLMs can't be concious

I've not come across this argument before, and I don't think I understand it well enough to write about it, sorry.