Searching for the Root of the Tree of Evil

You point out several problems in the world: people have unhealthy lifestyles, nuclear power isn't used to its full potential, ecosystems are not protected, our social lives are not in accordance with human flourishing. Then you say all these problems could be solved by a "cooperation machine". But you don't seem to explain why these problems could be solved by the same "machine". Maybe they're all separate problems that need to be solved separately.

Maybe one exercise to try is holding off on proposing solutions [LW · GW]. Can you discuss the problems in more detail, but without mentioning any solutions? Can you point out commonalities between the problems themselves? For example, "all these problems would be solved by a cooperation machine" wouldn't count, and "they happen because people are bad at cooperating" is too vague. I'm looking for something more like "these problems happen because people get angry", or "because people get deceived by politicians". Does that make sense? Can you give it a try?

Sooo, you need to build some super Shapley values calculator and additionally embed it into our preexisting coordination mechanisms such that people who use it on average do better that people who don't.

Magnificent! In particular, your pacing and composition was perfect - when I had a thought, you'd answer it in a sentence to a paragraph.

Unfortunately, improved coordination is going to have to be accomplished extremely quickly to help with superintelligence, which is going to happen very soon now on a historical timeframe. See my other comment here for thoughts on a reputation system that could be used to find better leadership to represent groups.

A similar algorithm balanceing priorities for each individual could work. But those with power would have to want to cooperate with those without. Some of them are nice people and do; lots of them aren't and don't. There are solutions for cooperation and we should search for them. There aren't solutions for people simply wanting different, conflicting things, and two of the problems you mention are in that category: arguably, for the most part, people simply don't want to be healthy or prevent climate change as much as they want to have fun now.

I guess you could say they're bad at cooperating with their future selves, so a better coordination mechanism could help that short-term bias, too.

To your point about improved understanding: you imply that AI could be part of that improvement, and I very much agree. Even current LLMs are probably able to do "translation" of cultural viewpoints, (e.g., explain to a liberal how someone could ever support anti-abortion laws without simply wanting to control women's bodies and punish sex). Getting people to use them or exposing them to those arguments would be the hard part. This is sort of back to the problem being that humans don't want to cooperate.

They'd want to cooperate if you could show them that the simple provable fact is that they'll probably get terrible results if they keep on competing. And that might very well be the case with a multipolar scenario with AGIs aligned to different humans' intentions.

It seems to me that the next level will be one of the following, or its combination:

• a nicer, more cooperative society
• prediction markets
• superhuman artificial intelligence
• eugenics
• genetically modified humans
• cyborgs
• ems

In some sense, these are all just different strategies for "becoming smarter", the main difference is between creating individuals that are smarter (AI, mutants, cyborgs), creating more of the smart individuals (eugenics, ems), or improving cooperation between existing smart individuals (niceness, prediction markets).

In current situation, I see the problem as the following:

• most people are not smart; most smart people are not sane
• most people are kinda nice, but the bad actors are somehow over-represented
• a lot of energy is wasted in zero-sum competitions, and we do not have good mechanisms for cooperation
• we do not even have good ways to aggregate information about people

The first point is obvious. Maybe not if you live in the Bay Area, but I assume that everywhere else the lack of smart and sane people is visible and painful. I have no idea how to build a nicer society with stupid and insane people. Democracy selects for ideas that appeal to many, i.e. to the stupid and insane majority. Dictatorship selects for people who are not nice. My best guess would be to bet on creating a smart and sane subculture, and hope that it will inspire other people. But outside of Bay Area we probably don't have enough people to start it, and within Bay Area most people are on drugs or otherwise crazy.

The second point... I wish there was a short explanation I could point at, but the concept is approximately in the direction of "you are not hiring the top 1% [LW · GW]" and "the asshole filter". It's a combination of "less desirable people circulate more (because they have to)" and "naive defenses against bad people are actually more likely to discourage good ones (because they respect your intent to be left alone) and less likely to discourage bad ones (because predators have a motive to get to you)". As a result, our interactions with random people are more likely to be unpleasant than the statistics of the population would suggest.

The fourth point... I often feel like there should be some mechanism for "people reviewing people", where we could get information about other people in ways more efficient and reliable than gossip. Otherwise, trust can't scale. But when I imagine such review system, there are many obvious problems with no obvious solutions. For starters, people lie. If I say "avoid him, he punched me for no reason", nothing prevents the other person to write exactly the same thing about me, even if that did not happen. A naive solution might be to expect that people will reciprocate both good and bad reviews, and thus treat all relations as directionless; we cannot know whether X or Y is the bad guy, but we can know that X and Y don't like each other. Then look at the rest of the graph, and if X has conflicts with dozen different people, and Y only has a conflict with X, then probably X is the bad guy. Except, nope, maybe Y just asked all his friends to falsely accuse X. Also, maybe Y has a lot of money and threatens to sue everyone who gives him a bad rating in the system.

Interesting read, though I find it not easy to see exactly what your main message is. Two points strike me as potentially relevant regarding

what do I think the real root of all evil is? As you might have guessed from the above, I believe it’s our inability to understand and cooperate with each other at scale. There are different words for the thing we need more of: trust. social fabric. asabiyyah. attunement. love.

The first more relevant, the second a term I'd simply find naturally core in a discussion on the topic:

1. Even increased "trust. social fabric." is not so clear to bring us forward. Let's assume people remain similarly self-interested, similarly egoistic, but they are able to cooperate better in limited groups: easy to imagine circumstances in which dominant effects could include: (i) easier for hierarchies in tyrannic dictatorships to cooperate to oppress their population and/or (ii) easier for firms to cooperate to create & exploit market power, replacing some reasonably well-working markets by, say, crazily exploitative oligopolies and oligarchies.
2. Altruism: simply the sheer limitation to our degree of altruism*^[1] with the wider population, might one call that one out as a single most dominant root of the tree of evil? Or, say, lack of altruism, combined with the necessary imperfection in self-interested positive collaboration given our world features at the time (i) our limited rationality and (ii) a hugely complex natural and economic environment? Increase our altruism, and most of today's billions of bad incentives we're exposed to become a bit less disastrous...

1. ^{^}

   Along with self-serving bias, i.e. our brain's sneaky way to reduce our actual behavioral/exhibited altruism to levels even below our (already limited) 'heartfelt' degree of altruistic interest, where we often think we try to act in other people's interests while in reality pursuing our own interests.

I like and agree with a lot in this essay. But I have to admit I'm confused by your conclusion. You dismiss social science research as probably not going anywhere, but then your positive proposal is basically more social science research. Doesn't building "a Cooperation Machine that takes in atomized people and raw intelligence and produces mutual understanding and harmonious collective action" require being "better at organizing our social life in accordance with human flourishing than the Victorians, the Qing, or the pre-conquest Lakota" in exactly the way you claim social science is trying and failing to produce?