LessWrong 2.0 Reader

There's a kind of paradox in all of these "straight line" extrapolation arguments for AI progress as your timelines assume (e.g., the argument for superhuman coding agents based on the rate of progress in the METR report).

One could extrapolate many different straight lines on graphs in the world right now (GDP, scientific progress, energy consumption, etc.).   If we do create transformative AI within the next few years, then all of those straight lines will suddenly hit an inflection point.  So, to believe in the straight line extrapolation of the AI line, you must also believe that almost no other straight lines will stay that way.

This seems to be the gut-level disagreement between those who feel the AGI and those who don't; the disbelievers don't buy that the AI line is straight and thus all the others aren't.  

I don't know who's right and who's wrong in this debate, but the method of reasoning here reminds me of the viral tweet: "My 3-month-old son is now TWICE as big as when he was born. He's on track to weigh 7.5 trillion pounds by age 10."  It could be true, but I have a fairly strong prior from nearly every other context that growth/progress tends to bend down into an S-curve at one point or another, and so these forecasts seems deeply suspect to me unless there's some kind of better reason to suspect that trends will continue along the same path.

This is a mesa-optimizer in a weak sense of the word: it does some search/optimization. I think the model in the paper here is weakly mesa-optimizing, maybe more than base models generating random pieces of sports news, and maybe roughly as much as a model trying to follow weird and detailed instructions - except that here it follows memorized "instructions" as opposed to in-context ones.

Off-topic, but what the heck is "The Tyranny of the Marginal Spice Jar"?

Just because the average person disapproves of a protest tactic doesn't mean that the tactic doesn’t work. Roger Hallam's "Designing the Revolution" series outlines the thought process underlying disruptive actions like the infamous soup-throwing protests.  Reasonable people may disagree (I disagree with quite a few things he says), but if you don't know the arguments, any objection is going to miss the point.  To be clear, PauseAI does not endorse or engage in disruptive civil disobedience, but I discuss it here to illustrate some broader points.  Anyways, “Designing the Revolution” is very long, so here's a tl/dr:

• If the public response is: "I'm all for the cause those protestors are advocating, but I can't stand their methods"...notice that the first half of this statement was approval of the only thing that matters—approval of the cause itself.

But that approval may have predated the protest, and might have been reduced by it.  Have you encountered this research?  "Extreme Protest Tactics Reduce Popular Support for Social Movements".  Abstract:

Social movements are critical agents of change that vary greatly in both tactics and popular support. Prior work shows that extreme protest tactics – actions that are highly counter-normative, disruptive, or harmful to others, including inflammatory rhetoric, blocking traffic, and damaging property – are effective for gaining publicity. However, we find across three experiments that extreme protest tactics decreased popular support for a given cause because they reduced feelings of identification with the movement. Though this effect obtained in tests of popular responses to extreme tactics used by animal rights, Black Lives Matter, and anti-Trump protests (Studies 1-3), we found that self-identified political activists were willing to use extreme tactics because they believed them to be effective for recruiting popular support (Studies 4a & 4b). The activist’s dilemma – wherein tactics that raise awareness also tend to reduce popular support – highlights a key challenge faced by social movements struggling to affect progressive change.

Excerpt:

Study 1. Participants read about a fictional animal rights activist group called Free the Vulnerable (FTV). The extremity of the movement’s protest behavior was manipulated at three levels: Moderate Protest, Extreme Protest, or Highly Extreme Protest conditions. The protesters in the two extreme protest conditions engaged in unlawful activities (e.g., breaking into an animal testing facility) modeled after protest activities of real-life activists, while the activists in the Moderate Protest condition peacefully marched in protest. [...] After participants read their assigned article, they completed measures of perceived extremity, social identification with the movement and support for it.

Important results: Social identification with the movement was 2.70, 2.48, 2.28 for moderate, extreme, and highly extreme protest conditions respectively.  Support for the movement: 3.07, 2.60, 2.62.

Read more of the paper to see more similar results.  It is nice to see them.  For example, in the study with fictional BLM protests:

These results suggest that both African Americans and non-African Americans perceived protesters as more extreme and felt less support for them in the Extreme Protest condition.

[...] Thus, as with participant race, these results suggest that participants, regardless of their political ideology, reacted negatively to extreme protests.

And the study with fictional anti-Trump protests:

Together, these results suggest that regardless of pre-existing attitudes regarding Trump’s candidacy, participants in the Extreme Protest condition viewed the protesters as more extreme and reported less support for the movement.

I did the exercise of distinguishing this from bucket errors and fallacies of compression, and really appreciated the well-chosen name for this: rounding as lowering precision in a controlled, predictable way; simplification in the sense of moving up in some hierarchy of refinements. Like rounding, it is sometimes desirable (to simplify or to compare distant objects), and sometimes necessary (e.g. when you have a detailed best guess with a lot of uncertainty; 12.527g ±1g might actually be your scale's best guess but the information is still mostly misleading).

My solution to the exercise:

That excerpt says “compute-efficient” but the rest of your comment switches to “sample efficient”, which is not synonymous, right? Am I missing some context?

News of the new models percolates slowly through the US government and beyond.

Well fleshed out scenario, but this kind of assumption is always a dealbreaker for me.

Why would the government not be aware of the development of the mightiest technology and weapon ever created if "we" aware of it?

Could you please elaborate why you choose to go for the "stupid and uninformed government", instead of the more plausible scenario where the government actually knows exactly what is going on in every step of the process and is the driving force behind it? 

Pretty sure “DeepCent” is a blend of DeepSeek & Tencent—they have a footnote: “We consider DeepSeek, Tencent, Alibaba, and others to have strong AGI projects in China. To avoid singling out a specific one, our scenario will follow a fictional “DeepCent.””. And I think the “brain” in OpenBrain is supposed to be reminiscent of the “mind” in DeepMind.

Inspired by critical remarks from @Laura-2 [LW · GW] about "bio/acc", my question is, when and how does something like this give rise to causal explanation and actual cures? Maybe GWAS is a precedent. You end up with evidence that a particular gene or allele is correlated with a particular trait, but you have no idea why. That lets you (and/or society) know some risks, but it doesn't actually eliminate disease, unless you think you can get there by editing out risky alleles, or just screening embryos. Otherwise this just seems to lead (optimistically) to better risk management, and (pessimistically) to a "Gattaca" society in which DNA is destiny, even more than it is now. 

I'm no biologist. I'm hoping someone who is, can give me an idea of how far this GWAS-like study of genotype-phenotype correlations, actually gets us towards new explanations and new cures. What's the methodology for closing that gap? What extra steps are needed? How much have we benefited from GWAS so far? 

Just had the idea to share my experience when I tried this in February. Previous Experience: Had played world 1 before Goal: Actually get it right on the first try with ~90% confidence

Approach:

• I wanted high legibility on my thoughts, to increase surface area, that I can reflect on

• thinking out loud with screen + audio recording

• thinking through writing (e.g. "assumption: ...", "plan A", "realization", "insight")

• durations (e.g. 4min) to allow me to see my speed at a glance

• Fatebook predictions for my solution working

• Feb08 at 20:00 I played world 2 level 1: took 45min + 20min of reflection

• Feb23 at 05:40 I played world 2 level 2: took 40min + 20min of reflection

• Feb23 at 07:00 I played world 2 level 3: was quite easy, so I found 5 solution candidates. My confidence ranging from 65%-90%; all ended up working

My initial idea for both level 1 and level 2 would have failed though I knew to continue and then found a correct solution. I like that this exercise gave me a glimpse of what it feels like to solve a problem that you felt stuck on. I think it is useful for me continue this exercise

If it's useful feedback to you, I can send you my notes (3 levels) and the recording (only first level)