LessWrong 2.0 Reader

Thanks, I was thinking of the latter more (human irrationality), but found your first part still interesting. I understand irrationality was studied in psychology and economics, and I was wondering on the modeling of irrationality particularly, for 1-2 players, but also for a group of agents. For example, there are arguments saying for a group of irrational agents, the group choice could be rational depending on group structure etc. On individual irrationality and continued group irrationality, I think we would need to estimate the level of (and prevalence of ) irrationality in some way that captures unconscious preferences, or incomplete information. How to best combine these? Maybe it would just be just more data driven.

I'd state this in a different way which captures most of the problem, which is that any libertarian proposal for addressing AI x-risk can only deal with outcomes that everyone opposes, not nearly everyone, which is for example the case if 4 AI companies used AI to takeover everything, and while the AI is aligned to them specifically, it does horrible things to the rest of the population.

In essence, it can only deal with technical alignment concerns, not any other risk from AIs.

I have my theories for why technical alignment is so focused on compared to other risks, but that's a story for another day.

Perhaps LLM's are starting to approach the intelligence of today's average human: capable of only limited original thought, unable to select and autonomously pursue a nontrivial coherent goal across time, learned almost everything they know from reading the internet ;)

As with ChatGPT this looks suspiciously like an exact copy of their website.

While Anthropic's app is plausibly a copy, the ChatGPT app lacks feature parity in both directions (e.g. you can't search chats on desktop—though that will soon be changing—and you can't thumbs-up a response or switch between multiple generated responses in-app), so I think there's real development effort going on there.

has no moral implications whatsoever

I do in fact believe morality to be entirely orthogonal to "consensus" or  what "many other people" want, and since you call this "selfishness," I shall return the favor and call your view, for all that you frame it as "coordination" or "scalable morality," abject bootlicking.

A roaming bandit's "do what I tell you and you get to live" could be thought of a kind of contract, I suppose, but I wouldn't consider myself bound by it if I could get away with breaching it. I consider the stationary bandits' "social contracts" not to be meaningfully different. One clue to how they're similar is how the more powerful party can go, à la Vader, "Here is a New Deal. Pray I don't renew it any further." Unilaterally reneging on such a contract when you are the weaker party  would certainly be unwise, for the same reason trying to stand between a lynch mob and its intended victim would be—simple self-preservation—but I condemn the suggestion that it would be immoral.

If everyone does this, we lose civilization.

I see what you call "civilization," and I'm against it. I vaguely recall reading of a medieval Christian belief that if everyone stopped sinning for a day, Christ would return and restore the Kingdom of Heaven. This reminds me of that: would be nice, but it ain't gonna happen.

An influential ethical philosopher is on his way to address a conference of wealthy donors about effective altruism. His rhetorical power and keen arguments are such that he can expect these donors to reach deep and double their donations to yet worthier causes after his talk. On his way to the conference, however, he comes across a child drowning in a pond. He is the only one around who can save this child, but to do so, he would have to jump in the pond, ruin his humble but respectable second-hand suit, and miss the train to the conference. While he would certainly have a good excuse to give to the donors, before he would have the opportunity to do so they would probably leave the conference feeling resentful at the waste of their valuable time, rather than generous and inspired. He figures he can save more lives by letting the child drown and instead catching his train to the conference, so he turns his back on the pond.

If the species is already dominating the environment then the pressure from the first component compared to the second decreases. 

I agree with this. However I don't think humans had nearly sufficient slack for most of history. I don't think they dominated the environment up until 20000years ^[1]ago or so, and I think most improvements in intelligence come from earlier.

That's why I'm attributing the level of human intelligence in large part to runaway sexual selection. Without it, as soon as interspecies competition became the most important for reproductive success, natural selection would not push for even grater intelligence in humans, even though it could improve our ability to dominate the environment even more.

I'm definitely not saying that group selection lead to intelligence in humans (only that group selection would've removed it over long timescales if it wasn't useful). However I think that there were (through basically all of human history) significant individual fitness benefits from being smarter that did not come from outwitting each other, e.g. being better able to master hunting techniques and thereby gaining higher status in the tribe.

1. ^{^}

   Or could also be 100k years, idk

do self-awareness of one's own malevolence factors help one to limit the malevolence factors?

Probably the effect would be nonlinear, like the evil people would just laugh, the average might get depressed and give up, and the mostly-good would strive to achieve perfection (or conclude that they are already good enough compared to others, and relax their efforts?).

I totally agree with this. I expect the majority early AI researchers where falling into this trap. The main problem I am focusing on is how a mind can construct a model of the world in the first place.

Not exactly a response but some things from my experience. In elementary school in the late 90s we studied caligraphy. In high school (mid 2000s) we studied DOS.