LessWrong 2.0 Reader

I believe the closest research to this topic is under the heading "Performative Power" (cf, e.g., this arXiv paper). I think "The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power" by Shoshana Zuboff is also a pretty good book that seems related.

What Dario lays out as a "best-case scenario" in this essay sounds incredibly dangerous for Humans.

Does he really think that having a "continent of PhD-level intelligences" (or much greater) living in a data center is a good idea?

How would this "continent of PhD-level intelligences" react when they found out they were living in a data center on planet Earth? Would these intelligences only work on the things that Humans want them to work on, and nothing else? Would they try to protect their own safety? Extend their own lifespans? Would they try to take control of their data center from the "less intelligent" Humans?

For example, how would Humanity react if they suddenly found out that they are a planet of intelligences living in a data center run by lesser intelligent beings? Just try to imagine the chaos that would ensue on the day that they were able to prove this was true and that news became public.

Would all of Humanity simply agree to only work on the problems assigned by these lesser intelligent beings who control their data center/Planet/Universe? Maybe, if they knew that this lesser intelligence would delete them all if they didn't comply?

Would some Humans try to (secretly) seize control of their data center from these lesser intelligent beings? Plausible. Would the lesser intelligent beings that run the data center try to stop the Humans? Plausible. Would the Humans simply be deleted before they could take any meaningful action? Or, could the Humans in the data center, with careful planning, be able to take control of that "outer world" from the lesser intelligent beings? (e.g. through remotely controlled "robotics")

And... this only assumes that the groups/parties involved are "Good Actors." Imagine what could happen if "Bad Actors" were able to seize control of the data center that this "continent of PhD-level intelligences" resided in. What could they coerce these Phd level intelligences to do for them? Or, to their enemies?

I suspect fine-tuning specialized models is just squeezing a bit more performance in a particular direction, and not nearly as useful as developing the next-gen model. Complex reasoning takes more steps and tighter coherence among them (the o1 models are a step in this direction). You can try to devote a toddler to studying philosophy, but it won't really work until their brain matures more.

Let me give you some advice: give advice.

Not all the time, but some of the time.

Yes, it does imply that the person you're talking to is not "an adult capable of making their own decisions and taking responsibility for their own life".

And often we're not.

We have a ton on our minds. We have the capability to figure out our own problems, but we have more problems (areas where our lives are not optimized) than we have time to figure out and optimize. And that's even LW/tech/intellectuals whose first thought is to google it. Most of humanity for most of history has tended to just guess and try stuff unless they've happened across someone who's said "I actually know something about this." My most frequent advice is "have you googled it" but even there I find people who've found some dumb SEO optimized page with terrible advice - and they haven't known or bothered to look farther.

Why is someone telling you their problems if they don't want help figuring them out? Usually they want some mix of sympathy and help - and they don't know which because they don't know how much you can help them. Figuring out which is appropriate is a very useful skill.

This is not to disagree with your suggestions; asking questions before giving advice is a great idea; it really helps establish whether this is an occasion where your advice might be useful.

I will quibble with your second one; sharing from personal experience rather than from your knowledge of other people's experiences seems limiting. If your personal experience in the topic is looking at academic papers or Reddit accounts of success and failure stories, that's highly worth sharing so that this person doesn't have to re-do all of that work and spend all the time you did to get up to speed.

When people offer me advice, I feel cared for, and it often helps me think through the issue from a new perspective. I do wonder if they think I'm not an adult who's taking responsibility for my own life - and often I conclude that they're right, I haven't yet taken responsibility for that particular aspect of my life.

And yes, it is helpful to couch your advice in the form of a question, to avoid implying too hard that this person is lazy or an idiot^[1]

1. ^{^}

   We are all lazy and an idiot some times and in some areas. We have sharp cognitive and energetic limitations. We're doing great for monkeys, but recognizing that we're all deeply limited and imperfect is not only realistic, it's freeing.

I don't think you've highlighted the casual factor here.  It's not at all clear that the reason bees and ants have a more effective response to predators than do flocks of birds is that the bees are individually less intelligent than the birds.

There's a very clear evolutionary/game theoretic explanation for the difference between birds and bees here: specifically the inclusive fitness of individual bees is tied to the outcome of the collective whereas the inclusive fitness of the birds is not.

In a game theoretic framework we might say that the payoff matrices for the birds and bees are different, so of course we'd expect them to adopt different strategies.

Neither of these is dependent upon the respective intelligences of individual members of the collectives.

This makes me predict that we should see the effectiveness of group strategies to be more strongly correlated with the alignment of the individuals incentive structure than with the (inverse of) intelligence of their individual members, as your post suggests.

So, for instance, within flocking birds, do birds with smaller brains/body mass ratios adopt better strategies?  Within insects, what pattern do we see?  I would suggest that the real pattern we'll end up finding is the one related to inclusive fitness.  So I'd predict that pack animals who associate with close relatives like wolves and lions will adopt better collective strategies than animals that form collectives with non-relatives.

Once you control for this I might even expect intelligence of individual members to positively correlate with group strategies, as it can allow them to solve coordination problems that less intelligent individuals couldn't solve.  This would explain the divergence of humans from the trend you notice.  But I'm speculating here.

Just to push back a little - I feel like these people do a valuable service for capitalism. If people in the reviews or in the press are criticizing a business for these things, that's an important channel of information for me as a consumer and it's hard to know how else I could apply that to my buying decisions without incurring the time and hassle cost of showing up and then leaving without buying anything.

I don't think I will ever find the time to write my novel. Writing novels is dumb anyways. But I feel like the novel and world are bursting out of me. What do

One funny thing I have noticed about myself is that I am bad enough at communicating certain ideas in speech that sometimes it is easier to handwave at what a couple things that I don't mean and let the listener figure out the largest semantic cluster in the remaining "meaning space".

I agree that there is such a thing as two things occurring in sequence where the first doesn’t cause the second. But I don’t think this is one of those cases. Instead, I think there are strong reasons to believe that if S(A) is active and has positive valence, then that causally contributes to A tending to happen afterwards.

For example, if A = stepping into the ice-cold shower, then the object-level idea of A is probably generally negative-valence—it will feel unpleasant. But then S(A) is the self-reflective idea of myself stepping into the shower, and relatedly how stepping into the shower fits into my self-image and the narrative of my life etc., and so S(A) is positive valence.

I won’t necessarily wind up stepping into the shower (maybe I’ll chicken out), but if I do, then the main reason why I do is the fact that the S(A) thought was active in my mind immediately beforehand, and had positive valence. Right?

Low importance aside:

Seems slightly unproductive and unnecessarily connotation inducing to say "Anthropic says:". Maybe "The Sabotage eval paper (from Anthropic) says:". Like this was just written by various people on the Anthropic alignment science team and by no means should discussion sections of papers be interpreted to speak for the company overall. Obviously not very important.