LessWrong 2.0 Reader

Signal feels overloaded to me and kodo seems like a much more narrow concept.

"I've got pretty good cell signal." "The elite spend a lot of time virtue signalling." "The castaways used smoke signals to get rescued." "Careful configuration is required to prevent signal interference between network nodes." "I think there's a pretty good signal in the polling data." "Ted's not actually being mean to me, he's just counter-signalling."

I'd basically agree that kodo is a subcategory of signal, in the same way that chicken is a subcategory of bird. The narrower concept is useful. Kodo is closer to signal if you just think of it in the sense used by The Signal and the Noise, though there's at least a bit of distinction in my head. There are weaker and stronger signals, which make you more or less confident in (say) which candidate might win based on the poll. As contrast, a poll that was good kodo might tell you either the conservative party is going to win in a landslide or your poll sucks. "Polls say the conservative party is 60% to win" is a signal, but it isn't kodo.

Noise and din seem more like synonyms, though both are kind of defined by being opposites. Noise is everything not signal, din is everything not kodo? I kinda wish din didn't sound like an English word but figured it made more sense to write it up as it was in the lecture.

Another potential crux[^related] is that Ege's world view seemingly doesn't depend at all on AIs which are much faster and smarter than any human. As far as I can tell, it doesn't enter into his modeling of takeoff (or timelines to full automation of remote work which partially depends on something more like takeoff).

On my views this makes a huge difference because a large number of domains would go much faster with much more (serial and smarter) intelligence. My sense is that a civilization where the smartest human was today's median human and also everyone's brain operated 50x slower^[1] would in fact make technological progress much slower. Similarly, if AIs were as much smarter than the smartest humans as the smartest human is smarter than the median human and also ran 50x faster than humans (and operated at greater scale than the smartest humans with hundreds of thousands of copies all at 50x speed for over 10 million parallel worker equivalents putting aside the advantages of serial work and intelligence), then we'd see lots of sectors go much faster.

My sense is that Ege bullet bites on this and thinks that slowing everyone down wouldn't make a big difference, but I find this surprising. Or maybe his views are that parallelism is nearly as good as speed and intelligence and sectors naturally scale up parallel worker equivalents to match up with other inputs, so we're bottlenecking on some other inputs in the important cases.

Yes, I think when it comes to endogenous vs exogenous preferences, you are probably accurate about most of the field believing in exogenous preferences (although I'm not sure since I'm also unfamiliar). Rohin Shah once talked about ambiguous value learning [LW · GW], but my guess is that most people aren't focusing on that direction.

It's possible that people who believe in exogenous preferences will feel confused if they are described as following Mistake Theory, since exogenous preferences sounds like Conflict Theory, and doesn't sound like "pursuing an objective morality."

My personal opinion (and my opinion isn't that important here), is that humans ourselves follow a combination of endogenous and exogenous preferences. Our morals are very strongly shaped by what everyone around us believe, often more than we realize.

But at the same time, our hardcoded biology determines how our morals get shaped. If we hunt and kill animals for food or sport, but observe the animals we kill following various norms in their animal society, we will not adopt their norms by mere exposure, and be indifferent to their norms. This is because our hardcoded biology did not encode any tendency to care about those animals or to respect their norms.

I agree that studying endogenous preferences, and how to make them go right, is valuable!

Ironically, arguably the most important/useful point of the essay is arguing for a rebranded version of the "precisely timed short slow/pause/pivot resources to safety" proposal. Dario's rebranded it as spending down a "security buffer".

(I don't have a strong view on whether this is a good rebrand, seems reasonable to me I guess and the terminology seems roughly as good for communicating about this type of action.)

Population is highly relevant to war between neighbors, particularly conscript war. But is it relevant to colonization? Over the past half century the relative population of Congo to Belgium increased by a factor of 5. A factor of 4 makes a difference in a single battle, but on the scale of a colonization project where a handful of people take over a large territory, it is nothing. At most it would require scaling down ambition by a factor of 5. Wikipedia claims that the colonial population increased by 2 orders of magnitude from 1900 to 1960, while native population did not change. The factor of 100 with no obvious cause is the big story and the factor of 5 historically irrelevant. Belgium did not lose control because the colony became too large to control, for the population grew only after independence. Perhaps it gave up because the colonial population was 1% of the metropolitan population, a real expense, but that was the result of how it governed the colony, not a necessary consequence of relative populations.

The past really was different. The ability of a thousand Belgians to control the whole of the Congo is shocking. Social technologies change much faster than populations.

Nod, this feels a bit at the intersection of what I had in mind with "Cyborgism", and the "Schleppy work in narrow domains" section.

Some thoughts: for this sort of thing, there's a hypothesis ("making it easier to change representations will enable useful thinking in hmath", and a bunch of annoying implementation details you need to test the hypothesis (i.e. actually getting an LLM to do all that work reliably).

So my next question here is "can we test out a version of this sort of thing powered by some humans-in-a-trenchcoat", or otherwise somehow test the ultimate hypothesis without having to build the thing." I'm curious for your intuitions on that

Did anyone predict that we’d see major AI companies not infrequently releasing blatantly misaligned AIs (like Sidney, Claude 3.7, o3)?

Just four days later, X blew up with talk of how GPT-4o has become sickeningly sycophantic in recent days, followed by an admission from Sam Altman that something went wrong (with lots of hilarious examples in replies):

the last couple of GPT-4o updates have made the personality too sycophant-y and annoying (even though there are some very good parts of it), and we are working on fixes asap, some today and some this week.

at some point will share our learnings from this, it's been interesting.

I feel for the position you’re in - I wish I had more that is useful to say. I also worry about future career prospects, what a world looks like where people can’t find work, etc. I think it’s really understandable to be feeling concerned

If I were in your position, I’d try to separate out “Should I go into something like a trade?” from “And if so, should I leave college now?” If you think you’d enjoy a trade, that does strike me as a reasonable career to choose (which might or might not mean leaving college). At least in the US there’s pretty good money to be made by being a small business owner of a reliable trade service, or so is my impression. Note that this is different of course than being new to the trade and reasonably might take a while to transition over (not sure exactly how long), but many trades are undersupplied even at a worker level (again in the US)

I think there’s a broader question to consider here, which is “what are your values/goals for life”, both professionally and personally. If your preferred lens were social impact, that might look different than if you’re eg just trying to live a happy enough, stable enough life with the people you love. I don’t have great advice here, but I wonder if you’ve looked over resources like 80,000 Hours in terms of thinking about career choice?

I'm glad you mentioned the orthogonality hypothesis (that goals are orthogonal to intelligence). Part of our argument can be seen as reconceptualizing orthogonality, and rejecting a strong version of it. 

So it's interesting to me that you see orthogonality as leading folks to conflict theory over mistake theory. I am not sure it has to lead either way.

In our language, we say that it's better to have an "endogenous preferences" theory than an "exogenous preferences" theory, meaning that it's better to be able to construct models where preferences can change as a function of the mechanisms and processes being modeled. This is not true for most rational actor and RL models; they typically assume preferences at the start (i.e. "exogenously").

Here's a paragraph making this more concrete, which I pulled from a shorter summary of https://arxiv.org/abs/2412.19010 which I've been working on.

"Consider an individual's preference for certain types of music, say, rock versus classical. We need not assume they begin with a scalar utility or ``taste'' for one over the other. Instead, in our theory, their preferences are shaped by their experiences: perhaps they grow up hearing rock music frequently at home, attend rock concerts with friends, and rarely encounter classical music. These experiences are encoded as memories. When prompted by the question "What kind of person am I?" and summarizing these memories, the individual's pattern completion network p might generate the assembly "I am the kind of person who likes rock music". This self-attribution, influenced by past behavior and exposure, becomes part of the global workspace context. Subsequently, when faced with a choice like "What music should I listen to?", p, conditioned on the self-description "I like rock music", is more likely to complete the pattern with "Listen to rock music". Through this process, the individual's behavior (listening to rock) is influenced by their inferred identity, which itself arose from past behavior and exposure. This example illustrates how preferences are constructed endogenously through the aggregation and consolidation of experience, rather than being static, external inputs to the model."

The AI safety gravy train has hit the buffers.