LessWrong 2.0 Reader

Ironically, arguably the most important/useful point of the essay is arguing for a rebranded version of the "precisely timed short slow/pause/pivot resources to safety" proposal. Dario's rebranded it as spending down a "security buffer".

(I don't have a strong view on whether this is a good rebrand, seems reasonable to me I guess and the terminology seems roughly as good for communicating about this type of action.)

Population is highly relevant to war between neighbors, particularly conscript war. But is it relevant to colonization? Over the past half century the relative population of Congo to Belgium increased by a factor of 5. A factor of 4 makes a difference in a single battle, but on the scale of a colonization project where a handful of people take over a large territory, it is nothing. At most it would require scaling down ambition by a factor of 5. Wikipedia claims that the colonial population increased by 2 orders of magnitude from 1900 to 1960, while native population did not change. The factor of 100 with no obvious cause is the big story and the factor of 5 historically irrelevant. Belgium did not lose control because the colony became too large to control, for the population grew only after independence. Perhaps it gave up because the colonial population was 1% of the metropolitan population, a real expense, but that was the result of how it governed the colony, not a necessary consequence of relative populations.

The past really was different. The ability of a thousand Belgians to control the whole of the Congo is shocking. Social technologies change much faster than populations.

Nod, this feels a bit at the intersection of what I had in mind with "Cyborgism", and the "Schleppy work in narrow domains" section.

Some thoughts: for this sort of thing, there's a hypothesis ("making it easier to change representations will enable useful thinking in hmath", and a bunch of annoying implementation details you need to test the hypothesis (i.e. actually getting an LLM to do all that work reliably).

So my next question here is "can we test out a version of this sort of thing powered by some humans-in-a-trenchcoat", or otherwise somehow test the ultimate hypothesis without having to build the thing." I'm curious for your intuitions on that

Did anyone predict that we’d see major AI companies not infrequently releasing blatantly misaligned AIs (like Sidney, Claude 3.7, o3)?

Just four days later, X blew up with talk of how GPT-4o has become sickeningly sycophantic in recent days, followed by an admission from Sam Altman that something went wrong (with lots of hilarious examples in replies):

the last couple of GPT-4o updates have made the personality too sycophant-y and annoying (even though there are some very good parts of it), and we are working on fixes asap, some today and some this week.

at some point will share our learnings from this, it's been interesting.

I feel for the position you’re in - I wish I had more that is useful to say. I also worry about future career prospects, what a world looks like where people can’t find work, etc. I think it’s really understandable to be feeling concerned

If I were in your position, I’d try to separate out “Should I go into something like a trade?” from “And if so, should I leave college now?” If you think you’d enjoy a trade, that does strike me as a reasonable career to choose (which might or might not mean leaving college). At least in the US there’s pretty good money to be made by being a small business owner of a reliable trade service, or so is my impression. Note that this is different of course than being new to the trade and reasonably might take a while to transition over (not sure exactly how long), but many trades are undersupplied even at a worker level (again in the US)

I think there’s a broader question to consider here, which is “what are your values/goals for life”, both professionally and personally. If your preferred lens were social impact, that might look different than if you’re eg just trying to live a happy enough, stable enough life with the people you love. I don’t have great advice here, but I wonder if you’ve looked over resources like 80,000 Hours in terms of thinking about career choice?

I'm glad you mentioned the orthogonality hypothesis (that goals are orthogonal to intelligence). Part of our argument can be seen as reconceptualizing orthogonality, and rejecting a strong version of it. 

So it's interesting to me that you see orthogonality as leading folks to conflict theory over mistake theory. I am not sure it has to lead either way.

In our language, we say that it's better to have an "endogenous preferences" theory than an "exogenous preferences" theory, meaning that it's better to be able to construct models where preferences can change as a function of the mechanisms and processes being modeled. This is not true for most rational actor and RL models; they typically assume preferences at the start (i.e. "exogenously").

Here's a paragraph making this more concrete, which I pulled from a shorter summary of https://arxiv.org/abs/2412.19010 which I've been working on.

"Consider an individual's preference for certain types of music, say, rock versus classical. We need not assume they begin with a scalar utility or ``taste'' for one over the other. Instead, in our theory, their preferences are shaped by their experiences: perhaps they grow up hearing rock music frequently at home, attend rock concerts with friends, and rarely encounter classical music. These experiences are encoded as memories. When prompted by the question "What kind of person am I?" and summarizing these memories, the individual's pattern completion network p might generate the assembly "I am the kind of person who likes rock music". This self-attribution, influenced by past behavior and exposure, becomes part of the global workspace context. Subsequently, when faced with a choice like "What music should I listen to?", p, conditioned on the self-description "I like rock music", is more likely to complete the pattern with "Listen to rock music". Through this process, the individual's behavior (listening to rock) is influenced by their inferred identity, which itself arose from past behavior and exposure. This example illustrates how preferences are constructed endogenously through the aggregation and consolidation of experience, rather than being static, external inputs to the model."

The AI safety gravy train has hit the buffers.

It's okay! I share your aversion to people just flatly declaring things to be impossible due to their own blinders, and I can see how my post was interpreted that way. Next time, I'll pay more attention to my instinct to add stuff like what I put in my reply to you directly in the LW version of one of my blog posts.

Wait I think the "indirect evolutionary game theory" mentioned in footnote 1, where agents optimize for subjective preferences (inner objectives) that are themselves subject to selection based on the fitness they induce cleared it up for me

There's a temporal aspect here - spite might start as instrumental but become intrinsic through training dynamics.

I expected your comment to be hyperbolic, but no. I mean sheesh:

In the decade that I have been working on AI, I’ve watched it grow from a tiny academic field to arguably the most important economic and geopolitical issue in the world.  In all that time, perhaps the most important lesson I’ve learned is this: the progress of the underlying technology is inexorable, driven by forces too powerful to stop, but the way in which it happens—the order in which things are built, the applications we choose, and the details of how it is rolled out to society—are eminently possible to change, and it’s possible to have great positive impact by doing so.  We can’t stop the bus, but we can steer it.  In the past I’ve written about the importance of deploying AI in a way that is positive for the world and of ensuring that democracies build and wield the technology before autocracies do. 

(Emphasis mine.) What rhetorical cleverness. This translates as: "I have expertise and foresightedness; here's your Overton window." Then he goes gears-level (ish) for a whole essay, reinscribing in the minds of Serious People the lethal assumptions laid out here: "We can't slow down; if you knew what I knew you'd see the 'forces' that make this obvious, and besides do you want the commies to win?"

I'm not just doing polemic. I think the rhetorical strategy "dismissing pause and cooperation out of hand instead of arguing against them" tells us something. I'm not sure what, alas. I do think that labs' arguments to the governments work best if they've already set the terms of the debate. It helps Dario's efforts if "pause/cooperate" is something "all the serious people know" is not worth paying attention to. 

I 80% think he also believes that pausing and cooperation are bad ideas (despite his obvious cognizance of the time-crunch). But I doubt he dismisses it so out-of-hand privately.