LessWrong 2.0 Reader

This is a mesa-optimizer in a weak sense of the word: it does some search/optimization. I think the model in the paper here is weakly mesa-optimizing, maybe more than base models generating random pieces of sports news, and maybe roughly as much as a model trying to follow weird and detailed instructions - except that here it follows memorized "instructions" as opposed to in-context ones.

Off-topic, but what the heck is "The Tyranny of the Marginal Spice Jar"?

Just because the average person disapproves of a protest tactic doesn't mean that the tactic doesn’t work. Roger Hallam's "Designing the Revolution" series outlines the thought process underlying disruptive actions like the infamous soup-throwing protests.  Reasonable people may disagree (I disagree with quite a few things he says), but if you don't know the arguments, any objection is going to miss the point.  To be clear, PauseAI does not endorse or engage in disruptive civil disobedience, but I discuss it here to illustrate some broader points.  Anyways, “Designing the Revolution” is very long, so here's a tl/dr:

• If the public response is: "I'm all for the cause those protestors are advocating, but I can't stand their methods"...notice that the first half of this statement was approval of the only thing that matters—approval of the cause itself.

But that approval may have predated the protest, and might have been reduced by it.  Have you encountered this research?  "Extreme Protest Tactics Reduce Popular Support for Social Movements".  Abstract:

Social movements are critical agents of change that vary greatly in both tactics and popular support. Prior work shows that extreme protest tactics – actions that are highly counter-normative, disruptive, or harmful to others, including inflammatory rhetoric, blocking traffic, and damaging property – are effective for gaining publicity. However, we find across three experiments that extreme protest tactics decreased popular support for a given cause because they reduced feelings of identification with the movement. Though this effect obtained in tests of popular responses to extreme tactics used by animal rights, Black Lives Matter, and anti-Trump protests (Studies 1-3), we found that self-identified political activists were willing to use extreme tactics because they believed them to be effective for recruiting popular support (Studies 4a & 4b). The activist’s dilemma – wherein tactics that raise awareness also tend to reduce popular support – highlights a key challenge faced by social movements struggling to affect progressive change.

Excerpt:

Study 1. Participants read about a fictional animal rights activist group called Free the Vulnerable (FTV). The extremity of the movement’s protest behavior was manipulated at three levels: Moderate Protest, Extreme Protest, or Highly Extreme Protest conditions. The protesters in the two extreme protest conditions engaged in unlawful activities (e.g., breaking into an animal testing facility) modeled after protest activities of real-life activists, while the activists in the Moderate Protest condition peacefully marched in protest. [...] After participants read their assigned article, they completed measures of perceived extremity, social identification with the movement and support for it.

Important results: Social identification with the movement was 2.70, 2.48, 2.28 for moderate, extreme, and highly extreme protest conditions respectively.  Support for the movement: 3.07, 2.60, 2.62.

Read more of the paper to see more similar results.  It is nice to see them.  For example, in the study with fictional BLM protests:

These results suggest that both African Americans and non-African Americans perceived protesters as more extreme and felt less support for them in the Extreme Protest condition.

[...] Thus, as with participant race, these results suggest that participants, regardless of their political ideology, reacted negatively to extreme protests.

And the study with fictional anti-Trump protests:

Together, these results suggest that regardless of pre-existing attitudes regarding Trump’s candidacy, participants in the Extreme Protest condition viewed the protesters as more extreme and reported less support for the movement.

I did the exercise of distinguishing this from bucket errors and fallacies of compression, and really appreciated the well-chosen name for this: rounding as lowering precision in a controlled, predictable way; simplification in the sense of moving up in some hierarchy of refinements. Like rounding, it is sometimes desirable (to simplify or to compare distant objects), and sometimes necessary (e.g. when you have a detailed best guess with a lot of uncertainty; 12.527g ±1g might actually be your scale's best guess but the information is still mostly misleading).

My solution to the exercise:

That excerpt says “compute-efficient” but the rest of your comment switches to “sample efficient”, which is not synonymous, right? Am I missing some context?

News of the new models percolates slowly through the US government and beyond.

Well fleshed out scenario, but this kind of assumption is always a dealbreaker for me.

Why would the government not be aware of the development of the mightiest technology and weapon ever created if "we" aware of it?

Could you please elaborate why you choose to go for the "stupid and uninformed government", instead of the more plausible scenario where the government actually knows exactly what is going on in every step of the process and is the driving force behind it? 

Pretty sure “DeepCent” is a blend of DeepSeek & Tencent—they have a footnote: “We consider DeepSeek, Tencent, Alibaba, and others to have strong AGI projects in China. To avoid singling out a specific one, our scenario will follow a fictional “DeepCent.””. And I think the “brain” in OpenBrain is supposed to be reminiscent of the “mind” in DeepMind.

Inspired by critical remarks from @Laura-2 [LW · GW] about "bio/acc", my question is, when and how does something like this give rise to causal explanation and actual cures? Maybe GWAS is a precedent. You end up with evidence that a particular gene or allele is correlated with a particular trait, but you have no idea why. That lets you (and/or society) know some risks, but it doesn't actually eliminate disease, unless you think you can get there by editing out risky alleles, or just screening embryos. Otherwise this just seems to lead (optimistically) to better risk management, and (pessimistically) to a "Gattaca" society in which DNA is destiny, even more than it is now. 

I'm no biologist. I'm hoping someone who is, can give me an idea of how far this GWAS-like study of genotype-phenotype correlations, actually gets us towards new explanations and new cures. What's the methodology for closing that gap? What extra steps are needed? How much have we benefited from GWAS so far? 

Just had the idea to share my experience when I tried this in February. Previous Experience: Had played world 1 before Goal: Actually get it right on the first try with ~90% confidence

Approach:

• I wanted high legibility on my thoughts, to increase surface area, that I can reflect on

• thinking out loud with screen + audio recording

• thinking through writing (e.g. "assumption: ...", "plan A", "realization", "insight")

• durations (e.g. 4min) to allow me to see my speed at a glance

• Fatebook predictions for my solution working

• Feb08 at 20:00 I played world 2 level 1: took 45min + 20min of reflection

• Feb23 at 05:40 I played world 2 level 2: took 40min + 20min of reflection

• Feb23 at 07:00 I played world 2 level 3: was quite easy, so I found 5 solution candidates. My confidence ranging from 65%-90%; all ended up working

My initial idea for both level 1 and level 2 would have failed though I knew to continue and then found a correct solution. I like that this exercise gave me a glimpse of what it feels like to solve a problem that you felt stuck on. I think it is useful for me continue this exercise

If it's useful feedback to you, I can send you my notes (3 levels) and the recording (only first level)

My previous criticism was aimed at another post of yours, it likely wasn't your main thesis. Some nitpicks I have with it are:

"Developing AGI responsibly requires massive safeguards that reduce performance, making AI less competitive" you could use the same argument for AIs which are "politically correct", but we still choose to take this step, censorsing AIs and harming their performance, thus, it's not impossible for us to make such choices as long as the social pressure is sufficiently high.

"The most reckless companies will outperform the most responsible ones" True in some ways, but most large companies are not all that reckless at all, which is why we are seeing many sequels, remakes, and clones in the entertainment sector. It's also important to note that these incentives have been true for all of human nature, but that they've never mainfested very strongly until recent times. This suggests that that the antidote to Moloch is humanity itself, good faith, good taste and morality, and that these can beat game theoritical problem which are impossible when human beings are purely rational (i.e. inhuman). 

We're also assuming that AI becomes useful enough for us to disregard safety, i.e. that AI provides a lot of potential power. So far, this has not been true. AIs do not beat humans, companies are forcing LLMs into products but users did not ask for them. LLMs seem impressive at first, but after you get past the surface you realize that they're somewhat incompetent. Governments won't be playing around with human lives before these AIs provide large enough advantages.

"The moment an AGI can self-improve, it will begin optimizing its own intelligence."
This assumption is interesting, what does "intelligence" mean here? Many seems to just give these LLMS more knowledge and then call them more intelligent, but intelligence and knowledge are different things. Most "improvements" seem to lead to higher efficiency, but that's just them being dumb faster or for cheaper. That said, self-improving intelligence is a dangerous concept.

I have many small objections like this to different parts of the essay, and they do add up, or at least add additional paths to how this could unfold.

I don't think AIs will destroy humanity anytime soon (say, within 40 years). I do think that human extinction is possible, but I think it will be due to other things (like the low birthrate and its economic consequences. Also tech. Tech destroys the world for the same reasons that AIs do, it's just slower).

I think it's best to enjoy the years we have left instead of becoming depressed. I see a lot of people like you torturing themselves with x-risk problems (some people have killed themselves over Roko's basilisk as well). Why not spend time with friends and loved ones?

Extra note: There's no need to tie your identity together with your thesis. I'm the same kind of autistic as you. The futures I envision aren't much better than yours, they're just slightly different, so this is not some psychological cope. People misunderstand me as well, and 70% of the comments I leave across the internet get no engagement at all, not even negative feedback. But it's alright. We can just see problems approaching many years before they're visible to others.