LessWrong 2.0 Reader

In that case "wokking" would be less confusing.

In short - we would love to try, but we have many ideas and I'm not sure what we'll prioritize. Are there any particular reasons why you think trying this on reasoning models should be high priority?

Ah I should emphasise, I do think all of these things could help -- it definitely is a spectrum, and I would guess these proposals all do push away from agency. I think the direction here is promising.

The two things I think are (1) the paper seems to draw an overly sharp distinction between agents and non-agents, and (2) basically all of the mitigations proposed look like they break down with superhuman capabilities. Hard to tell which of this is actual disagreements and which is the paper trying to be concise and approachable, so I'll set that aside for now.

It does seem like we disagree a bit about how likely agents are to emerge. Some opinions I expect I hold more strongly than you:

• It's easy to accidentally scaffold some kind of agent out of an oracle as soon as there's any kind of consistent causal process from the oracle's outputs to the world, even absent feedback loops. In other words, I agree you can choose to create agents, but I'm not totally sure you can easily choose not to
• Any system trained to predict the actions of agents over long periods of time will develop an understanding of how agents could act to achieve their goals -- in a sense this is the premise of offline RL and things like decision transformers
• It might be pretty easy for agent-like knowledge to 'jump the gap', e.g. a model trained to predict deceptive agents might be able to analogise to itself being deceptive
• Sufficient capability at broad prediction is enough to converge on at least the knowledge of how to circumvent most of the guardrails you describe, e.g. how to collude

Are you planning to test this on reasoning models?

Emotional social getting on with people vs logic puzzle solving IQ. 

Not sure I buy this, since IQ is usually found to positively correlate with purported measures of "emotional intelligence" (at least when any sort of ability (e.g. recognizing emotions) is tested; the correlation seems to go away when the test is pure self reporting, as in a personality test). EDIT: the correlation even with ability-based measures seems to be less than I expected.

Also, smarter people seem (on average) better at managing interpersonal issues in my experience (anecdotal, I don't have a reference).  But maybe this isn't what you mean by "emotional social getting on with people".

There could have been a thing where being too far from the average caused interpersonal issues, but very few people would have been far from the average, so I wouldn't expect this to have prevented selection if IQ helped on the margin.

Engineer parents are apparently more likely to have autistic children. This looks like a tradeoff to me. To many "high IQ" genes and you risk autism.

Seems somewhat plausible. I don't think that specific example is good since engineers are stereotyped as aspies in the first place; I'd bet engineering selects for something else in addition to IQ that increases autism risk (systematizing quotient, or something). I have heard of there being a population level correlation between parental IQ and autism risk in the offspring (but I wonder how much this just routes through paternal age, which has a massive effect on autism risk?).

How many angels can dance on the head of a pin. In the modern world, we have complicated elaborate theoretical structures that are actually correct and useful. In the pre-modern world, the sort of mind that now obsesses about quantum mechanics would be obsessing about angels dancing on pinheads or other equally useless stuff.

So I think there's two possibilities here to keep distinct. (1) is that ability to think abstractly wasn't very useful (and thus wasn't selected for) in the ancestral environment. (2) Is that it was actively detrimental to fitness, at least above some point. E.g. because smarter people found more interesting things to do than reproduce, or because they cared about the quality of life of their offspring more than was fitness-optimal, or something (I think we do see both of these things today, but I'm not sure about in the past).

Why is this what matters? It’s a bizarre metric. Why should we care what the median change was, instead of some form of mean change, or change in the mean or median wage?

The critique that the justification wasn't great because the mean wage dropped a lot in the example is fair. Yet, in the proposed alternative example it remains quite likely that people will perceive the economy as having gotten worse, even if the economy is objectively so much better - 2/3 will say they're personally worse off, insufficiently adjust for the impersonal ways of assessing the economy, and ultimately say the economy is worse. 

Neither $\Delta \left(MedianIncome\right)$ nor $Median\left(\Delta Income\right)$ is a bizarre metric. $\Delta \left(MedianIncome\right)$ may be great for observers to understand general trajectories of income when you lack panel data, but since people use their own lives to assess whether they are better off and in turn overweight that when they judge the economy, $Median\left(\Delta Income\right)$ is actually more useful for understanding the translation from people's lives into their perceptions.

Consider a different example (also in real terms):
T1: A makes $3, B makes $3, C makes $3 ,D makes $10, E makes $12
T2: A makes $2, B makes $2, C makes $3, D makes $9, E makes $16
The means show nice but not as crazy economic growth ($6.20 to $6.40), and the $\Delta \left(MedianIncome\right)$ is $0 ($3 to $3) - "we're not poorer!" However, the $Median\left(\Delta Income\right)$ is -$1. And people at T2 will generally feel worse off (3/5 will say they can't buy as much as they could before, so "this economy is tough").
Contrast that with (still in real terms):
T1: A makes $2, B makes $2, C makes $4 ,D makes $10, E makes $12
T2: A makes $3, B makes $3, C makes $3, D makes $10, E makes $12
The means show nice but not as crazy economic growth ($6 to $6.20), and the $\Delta \left(MedianIncome\right)$ is -$1 ($4 to $3) - "we're poorer!" However, the $Median\left(\Delta Income\right)$ is $0. And people at T2 will generally feel like things are going okay (only 1 person will feel worse off).

And these are comparing to 0. Mazlish's post illustrates that people will probably not compare to 0 and instead to recent trajectories ("I got a 3% raise last year, what do you mean my raise this year is 2%?!"), so #1 means people will be dissatisfied. #2, as it bore out in the data, also means dissatisfaction. And #3, largely due to timing, means further dissatisfaction.

Then it is no surprise that exit polls show people who were most dissatisfied with the economy under Biden (and assumed Harris would be more of that) voted for Trump. Sure, there's some political self-deception bias going on (see charts of economic sentiment vs. date by party affiliation), but note that the exit polls are correlational - they can indicate that partisanship is a hell of a drug or that people are rationally responding to their perceptions. It's likely both. And if your model of those perceptions is inferior in the ways Mazlish notes, you'd wrongly think people would have been happy with the economy.

Dimensional analysis is in instance of type signature checking, as fully understood. Unfortunately very few type systems for existing programming languages support the type transformations necessary to implement it directly.

yeah, if the system is trying to do things I agree it's (at least a proto) agent. My point is that creation happens in lots of places with respect to an LLM, and it's not implausible that use steps (hell even sufficiently advanced prompt engineering) can effect agency in a system, particularly as capabilities continue to advance. 

Would you mind to cross-post this on the EA Forum?

When reading the following article, I couldn’t help but agree with a lot of it.

Immigration is a constant subject of discussion between my wife and I. We are both immigrants to Canada, and both of our parents migrated from their home countries in search for better opportunities (Hers migrated permanently from Spain to France, mine temporarily from Colombia to Venezuela).

Since my first migration when I was 8 years old, I have always felt extremely privileged to be welcomed in a new country. When I migrated on my own to Canada, I was extremely grateful, and later when I decided I wanted to become Canadian, I made a strong effort to assimilate to Canadian culture.

Assimilation takes many forms, but at the very least I do not think it’s unreasonable to ask immigrants of three things when they immigrate:

1. Learn the official language of the place you are migrating to.
2. Follow the laws and general customs of the country you are migrating to, even if they are different that where you come from.
3. Do not impose your customs, culture, and religion onto others in the new country you are migrating to.

Canada is an amazing country in that it allows us the freedom to maintain our customs and practice our religions freely. I feel it is part of Canadian culture to accept and embrace the cultures of those who migrate here. However that welcoming and embracing of new cultures seems to be diminishing, and I believe this is due in part because a portion of those who have migrated here in the past decade simply have no desire to assimilate into Canadian society.

I think if we can figure out a way to encourage new immigrants to embrace assimilation into Canadian culture, Canadian society will slowly heal and go back to a place where we can all live together in harmony.

https://www.nytimes.com/2025/02/24/magazine/denmark-immigration-policy-progressives.html