On Downvotes, Cultural Fit, and Why I Won’t Be Posting Again

alignment is structurally impossible under competitive pressur

Alignment contrasts with control, as a means to AI safety.

Alignment roughly means the AI has goals, or values similar to human ones (which are assumed, without much evidence to be similar across humans), so that it will do what we want , because it's what it wants.

Control means that it doesn't matter what the AI wants, if it wants anything.

In short, there is plenty of competitive pressure towards control , because no wants an AI they can't control. Control is part of capability.

alignment will be optimised away, because any system that isn’t optimising as hard as possible won’t survive the race

Off the top of my head, this post [LW · GW]. More generally, this is an obvious feature of AI arms races [? · GW] in the presence of alignment tax [? · GW]. Here's a 2011 writeup that lays it out:

Given abundant time and centralized careful efforts to ensure safety, it seems very probable that these risks could be avoided: development paths that seemed to pose a high risk of catastrophe could be relinquished in favor of safer ones. However, the context of an arms race might not permit such caution. A risk of accidental AI disaster would threaten all of humanity, while the benefits of being first to develop AI would be concentrated, creating a collective action problem insofar as tradeoffs between speed and safety existed.

I assure you the AI Safety/Alignment field has been widely aware of it since at least that long ago.

Also,

alignment will be optimised away, because any system that isn’t optimising as hard as possible won’t survive the race

Any (human) system that is optimizing as hard as possible also won't survive the race. Which hints at what the actual problem is: it's not even that we're in an AI arms race, it's that we're in an AI suicide race [LW · GW] which the people racing incorrectly believe to be an AI arms race. Convincing people of the true nature of what's happening is therefore a way to dissolve the race dynamic. Arms races are correct strategies to pursue under certain conditions; suicide races aren't.

Yeah, valid correction.

Exactly. That’s the point I’ve been making - this isn’t about capitalism as an ideology, it’s about competition. Capitalism is just the most efficient competitive structure we’ve developed, so it accelerates the outcome. But any decentralised system with multiple actors racing for advantage - whether nation-states or corporations - will ultimately produce the same incentives. That’s the core of the argument.

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

It's "mainstream" here, described well many times before.

If people downvoted because they thought the argument wasn’t useful, fine - but then why did no one say that? Why not critique the focus or offer a counter? What actually happened was silence, followed by downvotes. That’s not rational filtering. That’s emotional rejection.

Yeah, I do not endorse the reaction. The situation pattern-matches to other cases where someone new writes things that are so confusing and all over the place that making them ditch the community (which is often the result of excessive downvoting) is arguably a good thing. But I don't think this was the case here. Your essays look to me to be coherent (and also probably correct). I hadn't seen any of them before this post but I wouldn't have downvoted. My model is that most people are not super strategic about this kind of thing and just go "talking politics -> bad" without really thinking through whether demotivating the author is good in this case.

So if I understand you correctly: you didn’t read the essay, and you’re explaining that other people who also didn’t read the essay dismissed it as “political” because they didn’t read it.

Yes -- from looking at it, it seems like it's something I agree with (or if not, disagree for reasons that I'm almost certain won't be addressed in the text), so I didn't see a reason to read. I mean reading is a time investment, you have to give me a reason to invest that time, that's how it works. But I thought the (lack of) reaction was unjustified, so I wanted to give you a better model of what happened, which also doesn't take too much time.

Most people say capitalism makes alignment harder. I’m saying it makes alignment structurally impossible.

The point isn’t to attack capitalism. It’s to explain how a system optimised for competition inevitably builds the thing that kills us.

I mean that's all fine, but those are nuances which only become relevant after people read, so it doesn't really change the dynamic I've outlined. You have to give people a reason to read first, and then put more nuances into the text. Idk if this helps but I've learned this lesson the hard way by spending a ridiculous amount of time on a huge post that was almost entirely ignored (this was several years ago).

(It seems like you got some reactions now fwiw, hope this may make you reconsider leaving.)

I'm afraid "Good at presenting their ideas in a persuasive manner" is doing all the heavy lifting here.

If the community had a good impression of him, they'd value his research over that of a PhD. If the community had a bad impression of him, they'd not give a second of thought towards his "research" and they would refer to it with the same mocking quotation marks that I just used. However, in the latter case, they'd find it more difficult to dismiss his PhD.

In other words, the interpretation depends if the community likes you or not. I've been in other rationalist communities and I'm speaking from experience (if I'm less vague that this, I'd be recognizable, which I don't want to be). I saw all the negative social dynamics that you'd find on Reddit or in young female friend groups with a lot of "drama" going on, in case you're unfortunate enough to have an intuition for such a thing.

In any "normie" community there's the staff in charge, and a large number of regular users who are somewhat above the law, and who feel superior to new users (and can bully them all they want, as they're friends with the staff). The treatment of users users depend on how well they fit in culturally, and it requires that they act as if the regulars are special (otherwise their ego is hurt). Of course, some of these effects are borderline invisible on this website, so they're either well-hidden or kept in check.

Still, this is not a truth-maximizing website, the social dynamics and their false premises (e.g. the belief that popularity is a measure of quality) are just too strong. The sort of intllectuals who don't care about social norms, status or money are better at truth-seeking and generally received poorly by places like this.

In theory, I’d agree with you. That’s how lesswrong presents itself: truth-seeking above credentials. But in practice, that’s not what I’ve experienced. And that’s not just my experience, it’s also what LW has a reputation for. I don’t take reputations at face value, but lived experience tends to bring them into sharp focus.

If someone without status writes something long, unfamiliar, or culturally out-of-sync with LW norms - even if the logic is sound - it gets downvoted or dismissed as “political,” “entry-level,” or “not useful.” Meanwhile, posts by established names or well-known insiders get far more patience and engagement, even when the content overlaps.

You say a self-taught blogger would be trusted if they’re good at presenting ideas persuasively. But that’s exactly the issue - truth is supposed to matter more than form. Ideas stand on their own merit, not on an appeal to authority. And yet persuasion, style, tone, and in-group fluency still dominate the reception. That’s not rationalism. That’s social filtering.

So while I appreciate the ideal, I think it’s important to distinguish it from the reality. The gap between the two is part of what my post is addressing.

A lot of the words we use are mathematical and thus more precise and with less connotations that people can misunderstand. This forum has a lot of people with STEM degrees, so they use a lot of tech terms, but such vocab is very useful for talking about AI risk. The more precise language is used, the less misunderstandings can occur.

Moloch describes a game theory problem, and these problems generally seem impossible to solve. But even though they're not possible to solve mathematically doesn't mean that we're doomed (I've posted about this on here before but I don't think anyone understood me. In short, game theory problems only play out when certain conditions are met, and we can prevent these conditions from becoming true).

I haven't read all your posts from end to end but I do agree with your conclusions that alignment is impossible and that AGI will result in the death or replacement of humanity. I also think your conclusions are valid only for LLMs which happen to be trained on human data. Since humans are deceptive, it makes sense that AIs training on them are as well. Since humans don't want to die, it makes sense that AIs trained on them also don't want to die. I find it unlikely that the first AGI we get is a LLM since I expect it to be impossible LLMs to improve much further than this.

I will have to disagree that your post is rigorous. You've proven that human errors bad enough to end society *could* occur, not not that they *will* occur. Some of your examples have many years between them because these events are infrequent. I think "There will be a small risk of extinction every year, and eventually we will lose the dice throw" is more correct.
Your essay *feels* like it's outlining tendencies in the direction of extinction, showing transitions which look like the following:
A is like B
A has a tendency for B
For at least some A, B follows.
If A, then B occurs with nonzero probability.
If A, then we cannot prove (not B).
If A, then eventually B.

And that if you collect all of these things in to directed acyclic graph, that there's a *path* from our current position to an extinction event. I don't think you've proven that each step A->B will be taken, and that it's impossible with a probability of 1 to prevent it (even if it's impossible to prevent it with a possibility of 1, which is a different statement)
I admit that my summary was imperfect. Though, if you really believe that it *will* happen, why are you writing this post? There would be no point in warning other people if it was necessarily too late to do anything about it. If you think "It will happen, unless we do X", I'd be interested in hearing what this X is.

I've read some of your other replies on here and I think I've found a pattern, but it's actually more general than AI. 

Harmful tendencies outcompete those which aren't harmful

This is true (even outside of AI), but only at the limit. When you have just one person, you cannot tell if he will make the moral choice or not, but "people" will make the wrong choice. The harmful behaviour is emergent at scale. Discrete people don't follow these laws, but the continous person does.

Again, even without AGI, you can apply this idea to technology and determine that it will eventually destroy us, and this is what Ted Kaczynski did. Thinking about incentives in this manner is depressing, because it feels like everything is deterministic and that we can only watch as everything gets worse. Those who are corrupt outcompete those who are not, so all the elites are corrupt. Evil businessmen outcompete good businessmen, so all successful businessmen are evil. Immoral companies outcompete moral companies, so all large companies are immoral.

I think this is starting to be true, but it wasn't true 200 years ago. At least, it wasn't half as harmful as it is now, why? It's because the defense against this problem is human taste, human morals, and human religions. Dishonesty, fraud, selling out, doing what's most efficient with no regard for morality. We consider this behaviour to be in bad taste, we punished it and branded it low-status, so that it never succeeded in ruining everything.

But now, everything could kill us (if the incentives are taken as laws, at least), you don't even need to involve AI. For instance, does Google want to be shut down? No, so they will want to resist antitrust laws. Do they want to be replaced? No, so they will use cruel tricks to kill small emerging competitors. When fines for illegal behaviour are less than the gains Google can make by doing illegal things, they will engage in illegal behaviour, for that is the logical best choice available to Google if all which matters is money. If we let it, Google would take over the world, in fact, it couldn't do otherwise. You can replace "Google" with any powerful structure in which no human is directly in charge. When it starts being more profitable to kill people than it is to keep them alive, the global population will start dropping fast. When you optimize purely for Money, and you optimize strongly enough, everyone dies. An AI just kills us faster because it optimizes more strongly, we already have something which acts similarly to AI. If you optimize too hard for anything, no matter what it is (even love, well-being, or happiness), everyone eventually dies (hence the paperclip maximizer warning).

If this post gave you existential dread, I've been told that Elinor Ostrom's books make for a good antidote.