We're already in AI takeoff

I think your pushback is ignoring an important point. One major thing the big contributors have in common is that they tend to be unplugged from the stuff Valentine is naming!

So even if folks mostly don't become contributors by asking "how can I come more truthfully from myself and not what I'm plugged into", I think there is an important cluster of mysteries here. Examples of related phenomena:

• Why has it worked out that just about everyone who claims to take AGI seriously is also vehement about publishing every secret they discover?
• Why do we fear an AI arms race, rather than expect deescalation and joint ventures?
• Why does the industry fail to understand the idea of aligned AI, and instead claim that "real" alignment work is adversarial-examples/fairness/performance-fine-tuning?

I think Val's correct on the point that our people and organizations are plugged into some bad stuff, and that it's worth examining that.

But as far as I know, none of them have made it a focus of theirs to fight egregores, defeat hypercreatures

Egregore is an occult concept representing a distinct non-physical entity that arises from a collective group of people.

I do know one writer who talks a lot about demons and entities from beyond the void. It's you, and it happens in some of, IMHO, the most valuable pieces you've written.

I worry that Caplan is eliding the important summoner/demon distinction. This is an easy distinction to miss, since demons often kill their summoners and wear their skin.

That civilization is dead. It summoned an alien entity from beyond the void which devoured its summoner and is proceeding to eat the rest of the world.

https://slatestarcodex.com/2016/07/25/how-the-west-was-won/

And Ginsberg answers: “Moloch”. It’s powerful not because it’s correct – nobody literally thinks an ancient Carthaginian demon causes everything – but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.

But the current rulers of the universe – call them what you want, Moloch, Gnon, whatever – want us dead, and with us everything we value. Art, science, love, philosophy, consciousness itself, the entire bundle. And since I’m not down with that plan, I think defeating them and taking their place is a pretty high priority.

https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

It seems pretty obvious to me: 

1.) We humans aren't conscious of all the consequences of our actions, both because the subconscious has an important role in making our choices, and because our world is enormously complex so all consequences are practically unknowable 

2.) In a society of billions, these unforeseeable forces combine in something larger than humans can explicitly plan and guide: "the economy", "culture", "the market", "democracy", "memes"

3.) These larger-than-human-systems prefer some goals that are often antithetical to human preferences. You describe it perfectly in Seeing Like A State: the state has a desire for legibility and 'rationally planned' designs that are at odds with the human desire for organic design. And thus, the 'supersystem' isn't merely an aggregate of human desires, it has some qualities of being an actual separate agent with its own preferences. It could be called a hypercreature, an egregore, Moloch or the devil. 

4.) We keep hurting ourselves, again and again and again. We keep falling into multipolar traps, we keep choosing for Moloch, which you describe as "the god of child sacrifice, the fiery furnace into which you can toss your babies in exchange for victory in war". And thus, we have not accomplished for ourselves what we want to do with AI. Humanity is not aligned with human preferences. This is what failure looks like. [LW · GW]

5.) If we fail to align humanity, if we fail to align major governments and corporations, if we don't even recognize our own misalignment, how big is the chance that we will manage to align AGI with human preferences? Total nuclear war has not been avoided by nuclear technicians who kept perfect control over their inventions - it has been avoided by the fact that the US government in 1945 was reasonably aligned with human preferences. I dare not imagine the world where the Nazi government was the first to get its hands on nuclear weapons. 

And thus, I think it would be very, very valuable to put a lot more effort into 'aligning humanity'. How do we keep our institutions and our grassroots movement "free from Moloch"? How do we get and spread reliable, non-corrupt authorities and politicians? How do we stop falling into multipolar traps, how do we stop suffering unnecessarily? 

Best case scenario: this effort will turn out to be vital to AGI alignment 

Worst case scenario: this effort will turn out to be irrelevant to AGI alignment, but in the meanwhile, we made the world a much better place 

I sadly don't have time to really introspect what is going in me here, but something about this comment feels pretty off to me. I think in some sense it provides an important counterpoint to the OP, but also, I feel like it also stretches the truth quite a bit: 

• Toby Ord primarily works on influencing public opinion and governments, and very much seems to view the world through a "raising the sanity waterline" lense. Indeed, I just talked to him last morning where I tried to convince him that misuse risk from AI, and the risk from having the "wrong actor" get the AI is much less than he thinks it is, which feels like a very related topic.
• Eliezer has done most of his writing on the meta-level, on the art of rationality, on the art of being a good and moral person, and on how to think about your own identity. 
• Sam Bankman-Fried is also very active in political activism, and (my guess) is quite concerned about the information landscape. I expect he would hate the terms used in this post, but I expect there to be a bunch of similarities in his model of the world and the one outlined in this post, in terms of trying to raise the sanity waterline and improve the world's decision-making in a much broader sense (there is a reason why he was one of the biggest contributors to the Clinton and Biden campaigns).

I think it is true that the other three are mostly focusing on object-level questions. 

I... also dislike something about the meta-level of arguing from high-status individuals. I expect it to make the discussion worse, and also make it harder for people to respond with counter arguments, because counter arguments arguments could be read as attacking the high-status people, which is scary.

I dislike the language used in the OP, and sure feel like it actively steers attention in unproductive ways that make me not want to engage with it, but I do have a strong sense that it's going to be very hard to actually make progress on building a healthy field of AI Alignment, because the world will repeatedly try to derail the field into being about defeating the other monkeys, or being another story about why you should work at the big AI companies, or why you should give person X or movement Y all of your money, which feels to me related to what the OP is talking about.

I wasn't convinced of this ten years ago [LW · GW] and I'm still not convinced.

Given the link, I think you're objecting to something I don't care about. I don't mean to claim that x-rationality is great and has promise to Save the World. Maybe if more really is possible [LW · GW] and we do something pretty different to seriously develop it [LW · GW]. Maybe. But frankly I recognize stupefying egregores here too and I don't expect "more and better x-rationality" to do a damn thing to counter those for the foreseeable future.

So on this point I think I agree with you… and I don't feel whatsoever dissuaded from what I'm saying.

The rest of what you're saying feels like it's more targeting what I care about though:

When I look at people who have contributed most to alignment-related issues […] what all of these people have in common is focusing mostly on object-level questions.

Right. And as I said in the OP, stupefaction often entails alienation from object-level reality.

It's also worth noting that LW exists mostly because Eliezer did in fact notice his own stupidity and freaked the fuck out. He poured a huge amount of energy into taking his internal mental weeding seriously in order to never ever ever be that stupid again. He then wrote all these posts to articulate a mix of (a) what he came to realize and (b) the ethos behind how he came to realize it.

That's exactly the kind of thing I'm talking about.

A deep art of sanity worth honoring might involve some techniques, awareness of biases, Bayesian reasoning, etc. Maybe. But focusing on that invites Goodhart. I think LW suffers from this in particular.

I'm pointing at something I think is more central: casting off systematic stupidity and replacing it with systematic clarity.

And yeah, I'm pretty sure one effect of that would be grounding one's thinking in the physical world. Symbolic thinking in service to working with real things, instead of getting lost in symbols as some kind of weirdly independently real.

But as far as I know, none of them have made it a focus of theirs to fight egregores, defeat hypercreatures, awaken to their own mortality, refactor their identity, or cultivate their will.

I think you're objecting to my aesthetic, not my content.

I think it's well-established that my native aesthetic rubs many (most?) people in this space the wrong way. At this point I've completely given up on giving it a paint job to make it more palatable here.

But if you (or anyone else) cares to attempt a translation, I think you'll find what you're saying here to be patently false.

• Raising the sanity waterline is exactly about fighting and defeating egregores=hypercreatures. Basically every debiasing technique is like that. Singlethink [LW · GW] is a call to arms.
• The talk about cryonics and xrisk is exactly an orientation to mortality. (Though this clearly was never Eliezer's speciality and no one took up the mantle in any deep way, so this is still mostly possessed.)
• The whole arc about free will is absolutely about identity. Likewise the stuff on consciousness and p-zombies. Significant chunks of the "Mysterious Answers to Mysterious Questions" Sequence were about how people protect their identities with stupidity and how not to fall prey to that.

The "cultivate their will" part confuses me. I didn't mean to suggest doing that. I think that's anti-helpful for the most part. Although frankly I think all the stuff about "Tsuyoku Naritai! [LW · GW]" and "Shut up and do the impossible" totally fits the bill of what I imagine when reading your words there… but yeah, no, I think that's just dumb and don't advise it.

Although I very much do think that getting damn clear on what you can and cannot do is important, as is ceasing to don responsibility for what you can't choose and fully accepting responsibility for what you can. That strikes me as absurdly important and neglected. As far as I can tell, anyone who affects anything for real has to at least stumble onto an enacted solution for this in at least some domain.

And if we pit their plumbing or truck-manuevering skills against those of an average person, I see no reason to think they would do better (besides maybe high IQ and general ability).

You seem to be weirdly strawmanning me here.

The trucking thing was a reverence to Zvi's repeated rants about how politicians didn't seem to be able to think about the physical world enough to solve the Canadian Trucker Convoy clog in Toronto. I bet that Holden, Eliezer, Paul, etc. would all do massively better than average at sorting out a policy that would physically work. And if they couldn't, I would worry about their "contributions" to alignment being more made of social illusion than substance.

Things like plumbing are physical skills. So is, say, football. I don't expect most football players to magically be better at plumbing. Maybe some correlation, but I don't really care.

But I do expect that someone who's mastering a general art of sanity and clarity to be able to think about plumbing in practical physical terms. Instead of "The dishwasher doesn't work" and vaguely hoping the person with the right cosmic credentials will cast a magic spell [LW · GW], there'd be a kind of clarity about what Gears one does and doesn't understand, and turning to others because you see they see more relevant Gears.

If the wizards you've named were no better than average at that, then that would also make me worry about their "contributions" to alignment.

It's seemed to me that the more that people talk about "rationality training" more exotic than what you would get at a really top-tier economics department, the more those people tend to get kind of navel-gazey, start fighting among themselves, and not accomplish things of the same caliber as the six people I named earlier. I'm not just saying there's no correlation with success, I'm saying there's a negative correlation.

I totally agree. Watching this dynamic play out within CFAR was a major factor in my checking out from it.

That's part of what I mean by "this space is still possessed". Stupefaction still rules here. Just differently.

You've seen the same data I have, so I'm curious what makes you think this line of research/thought/effort will be productive.

I think you and I are imagining different things.

I don't think a LW or CFAR or MIRI flavored project that focuses on thinking about egregores and designing counters to stupefaction is promising. I think that'd just be a different flavor of the same stupid sauce.

(I had different hopes back in 2016, but I've been thoroughly persuaded otherwise by now.)

I don't mean to prescribe a collective action solution at all, honestly. I'm not proposing a research direction. I'm describing a problem.

The closest thing to a solution-shaped object I'm putting forward is: Look at the goddamned question.

Part of what inspired me to write this piece at all was seeing a kind of blindness to these memetic forces in how people talk about AI risk and alignment research. Making bizarre assertions about what things need to happen on the god scale of "AI researchers" or "governments" or whatever, roughly on par with people loudly asserting opinions about what POTUS should do.

It strikes me as immensely obvious that memetic forces precede AGI. If the memetic landscape slants down mercilessly toward existential oblivion here, then the thing to do isn't to prepare to swim upward against a future avalanche. It's to orient to the landscape.

If there's truly no hope, then just enjoy the ride. No point in worrying about any of it.

But if there is hope, it's going to come from orienting to the right question.

And it strikes me as quite obvious that the technical problem of AI alignment isn't that question. True, it's a question that if we could answer it might address the whole picture. But that's a pretty damn big "if", and that "might" is awfully concerning.

I do feel some hope about people translating what I'm saying into their own way of thinking, looking at reality, and pondering. I think a realistic solution might organically emerge from that. Or rather, what I'm doing here is an iteration of this solution method. The process of solving the Friendliness problem in human culture has the potential to go superexponential since (a) Moloch doesn't actually plan except through us and (b) the emergent hatching Friendly hypercreature(s) would probably get better at persuading people of its cause as more individuals allow it to speak through them.

But that's the wrong scale for individuals to try anything on.

I think all any of us can actually do is try to look at the right question, and hold the fact that we care about having an answer but don't actually have one.

Does that clarify?

Don't have the time to write a long comment just now, but I still wanted to point out that describing either Yudkowsky or Christiano as doing mostly object-level research seems incredibly wrong. So much of what they're doing and have done focused explicitly on which questions to ask, which question not to ask, which paradigm to work in, how to criticize that kind of work... They rarely published posts that are only about the meta-level (although Arbital does contain a bunch of pages along those lines and Prosaic AI Alignment is also meta) but it pervades their writing and thinking.

More generally, when you're creating a new field of science of research, you tend to do a lot of philosophy of science type stuff, even if you don't label it explicitly that way. Galileo, Carnot, Darwin, Boltzmann, Einstein, Turing all did it.

(To be clear, I'm pointing at meta-stuff in the sense of "philosophy of science for alignment" type things, not necessarily the more hardcore stuff discussed in the original post)

Eliezer, at least, now seems quite pessimistic about that object-level approach. And in the last few months he's been writing a ton of fiction about introducing a Friendly hypercreature to an unfriendly world.

When I look at people who have contributed most to alignment-related issues - whether directly... or indirectly, like Sam Bankman-Fried

Perhaps I have missed it, but I’m not aware that Sam has funded any AI alignment work thus far.

If so this sounds like giving him a large amount of credit in advance of doing the work, which is generous but not the order credit allocation should go.

Putting this in a separate comment, because Reign of Terror moderation scares me and I want to compartmentalize. I am still unclear about the following things:

• Why do we think memetic evolution will produce complex/powerful results? It seems like the mutation rate is much, much higher than biological evolution.
• Valentine describes these memes as superintelligences, as "noticing" things, and generally being agents. Are these superintelligences hosted per-instance-of-meme, with many stuffed into each human? Or is something like "QAnon" kind of a distributed intelligence, doing its "thinking" through social interactions? Both of these models seem to have some problems (power/speed), so maybe something else?
• Misaligned (digital) AGI doesn't seem like it'll be a manifestation of some existing meme and therefore misaligned, it seems more like it'll just be some new misaligned agent. There is no highly viral meme going around right now about producing tons of paperclips.

I like this, thank you.

I score this as "Good enough that I debated not bothering to correct anything."

I think some corrections might be helpful though:

The internet is causing rapid memetic evolution…

While I think that's true, that's not really central to what I'm saying. I think these forces have been the main players for way, way longer than we've had an internet. The internet — like every other advance in communication — just increased evolutionary pressure at the memetic level by bringing more of these hypercreatures into contact with one another and with resources they could compete for.

These memes push people who host them (all of us, to be clear) towards behaviors which are not in the best interests of humanity, because Orthogonality Thesis

Yes. I'd just want to add that not all of them do. It's just that the ones that tend to dominate tend to be unFriendly.

Two counterexamples:

• Science. Not as an establishment, but as a kind of clarifying intelligence. This strikes me as a Friendly hypercreature. (The ossified practices of science, like "RCTs are the gold standard" and "Here's the Scientific Method!", tend to pull toward stupidity via Goodhart. A lot of LW is an attempt to reclaim the clarifying influence of this hypercreature's intelligence.)
• Jokes. These are sort of like innocuous memetic insects. As long as they don't create problems for more powerful hypercreatures, they can undergo memetic evolution and spread. They aren't particularly Friendly or unFriendly for the most part. Some of them add a little value via humor, although that's not what they're optimizing for. (The evolutionary pressure on jokes is "How effectively does hearing this joke cause the listener to faithfully repeat it?"). But if a joke were to somehow evolve into a more coherent behavior-controlling egregore, by default it'll be an unFriendly one.

Before you can work effectively on AI stuff, you have to clear out the misaligned memes stuck in your head.

Almost. I think it's more important that you have installed a system for noticing and weeding out these influences.

Like how John Vervaeke argues that the Buddha's Eightfold Noble Path is a kind of virtual engine for creating relevant insight. The important part isn't the insight but is instead the engine. Because the same processes that create insight also create delusion, so you need a systematic way of course-correcting.

This can get you the clarity/agency necessary, and make sure that (if successful) you actually produce AGI aligned with "you", not some meme

No correction here. I just wanted to say, this is a delightfully clear way of saying what I meant.

…we shouldn't try and just solve x-risk…

While I agree (both with the claim and with the fact that this is what I said), when I read you saying it I worry about an important nuance getting lost.

The emphasis here should be on "solve", not "x-risk". Solving xrisk is superhuman. So is xrisk itself for that matter. "God scale."

However! Friendly hypercreatures need our minds in order to think. In order for a memetic strategy to result in solving AI risk, we need to understand the problem. We need to see its components clearly.

So I do think it helps to model xrisk. See its factors. See its Gears. See the landscape it's embedded in.

Sort of like, a healthy marriage is more likely to emerge if both people make an effort to understand themselves, each other, and their dynamic within a context of togetherness and mutual care. But neither person is actually responsible for creating a healthy marriage. It sort of emerges organically from mutual open willingness plus compatibility.

…we should focus on rationality, cultivating our internal meme garden, and favoring memes which will push the world in the direction we want it to go

FWIW, this part sounds redundant to me. A "rationality" that is something like a magical completion of the Art would, as far as I can tell, consist almost entirely of consciously cultivating one's internal memetic garden, which is nearly the same thing as favoring Friendly memes.

But after reading and replying to Scott's comment, I'd adjust a little bit in the OP. For basically artistic reasons I mentioned "rationality for its own sake, period." But I now think that's distracting. What I'm actually in favor of is memetic literacy by whatever name. I think there's an important art here whose absence causes people to focus on AI risk in unhelpful and often anti-helpful ways.

Also, on this part:

…which will push the world in the direction we want it to go

I want to emphasize that best as I can figure, we don't have control over that. That's more god-scale stuff. What each of us can do is notice what seems clarifying and kind to ourselves and to lean that way. I think there's some delightful game theory that suggests that doing this supports Friendly hypercreatures.

And if not, I think we're just fucked.

These memes push people who host them (all of us, to be clear) towards behaviors which are not in the best interests of humanity, because Orthogonality Thesis

Im not entirely convinced. Memes are parasites, and thus, aim for equilibrium with its host. Hence why memeplexes that are truly evil and omnicidal never stick, memeplexes that are relatively evil peter out, and what we are left with are memeplexes that "kinda suck I guess" at worst. Succesful memeplex is one that ensures the host's survival while forcing the host to spend maximum energy and resources spreading the memeplex without harming themselves too badly.

I think there's an important difference Valentine tries to make with respect to your fourth bullet (and if not, I will make). You perhaps describe the right idea, but the wrong shape. The problem is more like "China and the US both have incentives to bring about AGI and don't have incentives towards safety." Yes deflecting at the last second with some formula for safe AI will save you, but that's as stupid as jumping away from a train at the last second. Move off the track hours ahead of time, and just broker a peace between countries to not make AGI.

Not only that, the system that could stand in the way — the government — has been captured by the same forces.

Yep.

The USA Constitution was an attempt to human-align an egregore.

But it was done in third person, and it wasn't mathematically perfect, so of course egregoric evolution found loopholes.

Random aside: If you like Sci-Fi, you should take a look at “Lady Of Mazes”. It’s the only post-scarcity book that feels weird enough to be even somewhat probable.  And I don’t wanna spoil it, but there’s a large of the book that relates very closely to your post.  

Thank you! By Karl Schroeder?

That's a good point. I hope you're right.

Well, that's just terrible.

Here's the post:

I think the world makes more sense if you recognize humans aren't on the top of the food chain.

We don't see this clearly, kind of like ants don't clearly see anteaters. They know something is wrong, and they rush around trying to deal with it, but it's not like any ant recognizes the predator in much more detail than "threat".

There's a whole type of living being "above" us the way animals are "above" ants.

Esoteric traditions sometimes call these creatures "egregores".

Carl Jung called a special subset of them "archetypes".

I often refer to them as "memes" — although "memeplex" might be more accurate. Self-preserving clusters of memes.

We have a hard time orienting to them because they're not made of stuff we're used to thinking of as living — in basically the same way that anteaters are tricky for ants to orient to as ant-like. Wrong pheromones, wrong size, more like reality than like members of this or another colony, etc.

We don't see a fleshy body, or cells, or a molecular mechanism. So there's no organism, right?

But we have a clear intuition for life without molecular mechanisms. That's why we refer to "computer viruses" as such: the analogy is actually insanely good. But the medium is computer code, not RNA.

Likewise with a piece of news "going viral" — although that steps a little into egregore territory.

What's the medium? What's the analog of molecules or code?

Well, it's patterns of behavior, thought, and attention.

Some spiritual traditions point at "identity" here. Who you think you are. Which opinions are "yours".

When you look at what's going on with Joe Rogan, or with the Canadian truckers, or with Australia's lockdowns, the easiest way to orient is to identify and pick a side.

But just imagine in each case that you're an ant trying to understand two animals having a fight. They're sometimes kicking your path or hitting an entrance to your colony. Sometimes bits of what you recognize as food lands near you or one of your colony mates.

Maybe one of those giants is an anteater.

Maybe both of them are. Maybe they're fighting over which gets to eat you and your kin.

Maybe you never get to know what they are or why they're fighting.

The analogy breaks down very quickly because the thought experiment projects human intelligence into an ant. What actually happens for humans is more like, the act of trying to look at an egregore tends to result in it affecting you. Like it reaches into your nervous system simply because you notice it.

This is why it's easier to orient by picking a side. You're becoming an extension of the egregore. It's using your biological resources to manifest in the physical world.

Many of these hyperobject creatures seem to like humans. Parts of science and religion come to mind. Wisdom practices. In some ways these spread more naturally through us, like the relationship is a kind of symbiotic. Loving-kindness practices and attitudes come to mind for instance.

But for this very reason, many of the predatory egregores do not want us functioning naturally. They intentionally disrupt us. They lean heavily on systematized trauma and threat responses. They want to keep us scared, or angry, or exhausted, or hopeless.

And it works in part because we have a hard time noticing the true origin of these drives. We rehearse the reasons we hear and think of, and learn how to point fingers at an outgroup as bad or inferior or threatening…

…and then over the course of years or decades we find ourselves having been chess pieces in a game we weren't really playing. Something else was playing us. History flowing through us in arcs much, much larger than we're accustomed to seeing.

I mean, at this point you've probably noticed at least one person who used to be close to you who has "lost it". Clearly believes falsehoods while insisting that you're the one who's confused.

That's what predation at the egregore level looks like. Ideological capture. People's lifeforce poured into sustaining an idea cluster that has possessed them.

Flat Earth, "plandemic" conspiracies, blind trust in demonstrably untrustworthy authorities, "cancel culture", Q-Anon…

…no matter where you stand on these or other controversial issues, I hope you can see a common thread of madness.

That's the zombie virus. The anteaters. Our cordyceps.

This picture — the one coming through me in this post for instance — comes from an egregore that wants us to see this. One that wants to end the egregoric culture wars via symbiosis and transparency. It wants to spread and thrive by making us immune to this predation.

It just requires that you learn to see the spirit world, so to speak.

To stop pretending to know that your free will is what you think it is — since your thoughts are part of what get reprogrammed in the course of this global memetic war.

Start by slowing down and looking.

When you read these bits of news, what does your body do? Do you lose track of your feet? The feeling of your skin on the air? Awareness of the room around you and your screen?

Are you just now waking up to those sensations right now as I name them? Did you forget them while reading this?

There's a lot of work to do here. Of learning to let go, and ground in safety, and choosing to enter time and truly face death. Feeling the depths of scarring that the predator memes clawed into us from a very young age. Truly healing.

And that all takes time.

(…and education. I'll offer some links below.)

But even beyond this, as HUMANS and not ants, we have an opportunity:

We can come to see, understand, and touch this level "above" us.

We can slow down and honestly witness our confusion, and our reactions, and recognize the kind of emergent intelligence that seems to pass through us and others.

We can integrate more symbiotic egregores into our consciousness and orient, intentionally, to the REAL causes of our global challenges.

We can stop letting the predatory aspects of corporations and governments sway us.

We can actually participate in our own future history.

I think that's very, very worthwhile.

It seems to me that you can think about questions of alignment from a purely technical mindset, [..] and that this kind of technical thinking is much less affected by hypercreatures than other things are.[...] And because this allows you to reason about alignment one step removed

I agree that thinking about alignment from a purely technical mindset provides a dissociative barrier that helps to keep the hypercreatures at bay. However, I disagree with the implication that this is "all good". When you're "removed" like that, you don't just cut the flow of bad influences. You cut everything which comes from a tighter connection to what you're studying.

If you're a doctor treating someone close to you, this "tighter connection" might bring in emotions that overwhelm your rational thinking. Maybe you think you "have to do something" so you do the something that your rational brain knows to have been performing worse than "doing nothing" in all the scientific studies.

Or... maybe you have yourself under control, and your intuitive knowledge of the patient gives you a feel of how vigilant they would be with physical therapy, and maybe this leads to different and better decisions than going on science alone when it comes to "PT or surgery?". Maybe your caring leads you to look through the political biases because you care more about getting it right than you do about the social stigma of wearing masks "too early" into the pandemic.

So if you want to be a good doctor to those you really care about, what do you do?

In the short term, if you can't handle your emotions, clearly you pass the job off to someone else. Or if you must, you do it yourself while "dissociated". You "Follow accepted practice", and view your emotions as "false temptations". In the long term though, you want to get to the place where such inputs are assets rather than threats, and that requires working on your own inner alignment.

In the example I gave, the some of the success of being "more connected" came from being more connected to your patient than you are to the judgment of the world at large. Maybe cutting off twitter would be a good start, since that's where these hypercreatures live, breed, and prey on minds. I think "How active are the leading scientists on twitter?" probably correlates pretty well for how much I'd distrust the consensus within that field.

As a thing that I fairly strongly suspect but cannot prove, maybe "remove yourself from twitter" is the scaled up equivalent of "remove yourself from your personal relationship with your patient" -- something that is prudent up until the point where you are able to use the input as an asset rather than getting bowled over and controlled it. In this view, it might be better to think of the problem as *growing* alignment. You start by nurturing your own independence and setting personal boundaries so you don't get sucked into codepentent relationships which end up mutually abusive -- even in subtle ways like "doing something" counterproductive as an emotionally overwhelmed doctor. Then once external influence on this scale can't kick you out of the driver seat, and you have things organized well enough that  you can both influence and be influenced in a way that works better than independence, then you move to increasingly larger scales. In this view, it's not necessarily "hypercreatures bad", but "hypercreatures bigger than I currently am, and not yet tamed".

- instead of thinking about the object-level values, you are reasoning about the system [...] that extracts the object-level values -

I strongly recommend doing inner alignment work this way too. A huge part of what people "care" about is founded on incomplete and/or distorted perspectives, and if you're not examining the bedrock your values are built on, then you're missing the most important part.

You may even reason about the hypercreatures abstractly enough not to trigger them, and design the AI with a hypercreature-elimination system which is good enough to detect and neutralize any irrational hypercreatures. This would, of course, be a threat to the hypercreatures possessing you, but part of their nature involves telling you that they are perfectly rational and justified and there is nothing irrational about them.

At the same time, an "AI removing hypercreatures" isn't really the kind of a threat they would have evolved to recognize or try to attack, so they feel safe just telling you that of course there will be no danger and that the creation of the AI will just lead to the AI implementing the_objectively_best_ideology everywhere. So you believe them, feel unconcerned as you design your AI to detect and neutralize irrational hypercreatures, and then suddenly oh what happened, how could you ever have believed that crazy old thing before.

I'm pretty pessimistic on the chances of that one. You're banking on what Val is describing as "superintelligences" being dumber than you are, despite the fact that it has recruited your brain to work for its goals. You're clearly smart enough to make the connection between "If I design an AI this way, I might not get what my hypercreature wants", since you just stated it. That means you're smart enough to anticipate it happening, and that's going to activate any defenses you have. There's no magic barrier that allows "debate partner is going to take my belief if I give them the chance" to be thought while preventing "AI partner is going to steal my belief if I give it the chance".

The framing of some unfriendly hypercreature taking over your brain against your values is an evocative one and not without use, but I think it runs out of usefulness here.

From the initial quote on Gwern's post on cults: [LW · GW]

"“Brainwashing”, as popularly understood, does not exist or is of almost zero effectiveness. [...] Typically, a conversion sticks because an organization provides value to its members.

This "enslavement to hypercreatures" typically happens because the person it "taken over" perceives it to have value. Sorting out the perceptions to match reality is the hard part, and "yer brain got eaten by hypercreature!" presupposes it away. At first glance the whole "anti-epistemology" thing doesn't seem to fit with this interpretation but we don't actually need "taken over by hostile forces" to explain it; watch regular people debate flat earthers and you'll see the same motivated and bad reasoning that shows that they're actually just taking things on faith. Faith in the consensus that the world is round is actually a really good way of dealing with the problem for people who can't reliably solve this kind of problem for themselves on the object level. So much of what we need to know we need to take "on faith", and figuring out which hypercreatues we can trust how far is the hard problem. Follow any of them too far and too rigidly and problems start showing up; that's what the "twelfth virtue" is about.

Trying to hide from the hypercreatures in your mind and design an AI to rid you of them is doomed to failure, I predict. The only way that seems to me to have any chance is to not have unhealthy relationships with hypercreatures as you bootstrap your own intelligence into something smarter, so that you can propogate alignment instead of misalignment.

…I'm much less certain about this part…

You might like to know: I debated erasing that part and the one that followed, thinking of you replying to it! :-D

But I figured hey, let's have the chat. :-)

It seems to me that you can think about questions of alignment from a purely technical mindset…

Yep, I know it seems that way.

And I disagree. I think it maintains a confusion about what "alignment" is.

However, I'm less certain of this detail than I am about the overall picture. The part that has me say "We're already in AI takeoff." Which is why I debated erasing all the stuff about identity and first-person. It's a subtle point that probably deserves its own separate post, if I ever care to write it. The rest stands on its own I think.

But! Setting that aside for a second:

To think of "questions of alignment from a purely technical mindset", you need to call up an image of each of:

• the AI
• human values
• some process by which these connect

But when you do this, you're viewing them in third person. You have to call these images (visual or not) in your mind, and then you're looking at them.

What the hell is this "human values" thing that's separable from the "you" who's looking?

The illusion that this is possible creates a gap that summons Goodhart. The distance between your subjective first-person experience and whatever concept of "human values" you see in third person is precisely what summons horror.

That's the same gap that unFriendly egregores use to stupefy minds.

You can't get around this by taking "subjectivity" or "consciousness" or whatever as yet another object that "humans care about".

The only way I see to get around this is to recognize in immediate direct experience how your subjectivity — not as a concept, but as a direct experience — is in fact deeply inextricable from what you care about.

And that this is the foundation for all care.

When you tune a mind to correctly reflect this, you aren't asking how this external AI aligns with "human values". You're asking how it synchs up with your subjective experience.

(Here minds get super squirrely. It's way, way too easy to see "subjective experience" in third person.)

As you solve that, it becomes super transparent that sorting out that question is actually the same damn thing as asking how to be free of stupefaction, and how to be in clear and authentic connection with other human beings.

So, no. I don't think you can solve this with a purely technical mindset. I think that perpetuates exactly the problem that this mindset would be trying to solve.

…but I could be wrong. I'm quite open to that.

So you believe them, feel unconcerned as you design your AI to detect and neutralize irrational hypercreatures, and then suddenly oh what happened, how could you ever have believed that crazy old thing before.

This part made me chuckle. :-)

I do think this is roughly how it works. It's just that it happens via memetics first.

But overall, I agree in principle. If I'm wrong and it's possible to orient to AI alignment as a purely technical problem, then yes, it's possible to sidestep hypercreature hostility by kind of hitting them in an evolutionary blindspot.

What do you mean by "technical" here? 

I think solving the alignment problem for government, corporations, and other coallitions would probably help solving the alignment problem in AGI.

I guess you are saying that even if we could solve the above alignment problems it would still not go all the way to solving it for AGI? What particular gaps are you thinking of?

This sounds like you're saying, it's not "our" (any of our?) job to solve technical problems in (third person non-human-AGI) alignment.

Right, but I mean something precise by that.

I agree with you. There's a technical problem, and it takes intelligent effort over time to solve it. And that's worthwhile.

It's also not up to any one individual whether or how that happens, and choice only ever happens (for now) at the scale of individuals.

So "should"s applied at the egregore scale don't make any coherent sense. They're mostly stupefying forces, those "should"s.

If you want to work on technical AGI alignment, great! Go for it. I don't think that's a mistake.

I also don't think it's a mistake to look around and model the world and say "If we don't sort out AI alignment, we all die."

But something disastrous happens when people start using fear of death to try to pressure themselves and others to do collective action.

That's super bad. Demon food. Really, really awful. Stupefying and horror-summoning.

I think getting super clear about that distinction is upstream of anyone doing any useful work on AI alignment.

I could be wrong. Maybe we've made enough collective (i.e., egregoric) progress on this area that the steps remaining for the technical problem aren't superhuman. Maybe some smart graduate student could figure it out over this summer.

I really wouldn't bet on it though.

Thanks for quoting the bit about Elua at the end. It is helpful to remember that despite Moloch, et al, humanity has managed some pretty impressive feats, even in the present day.

It's easy to think that the counterexample of science in earlier posts is something accomplished "Once upon a time in a land far away."

As a concrete example, I'm quite glad that the highly effective mRNA vaccines (Moderna/Pfizer) exist for the common man. They exist despite things like the FDA, the world of academic publishing, the need to find funding to survive, and so on.

Have you read Scott Alexander's Meditations on Moloch?

Absolutely!

You might like David Deutsch's book "The Beginning of Infinity". I read an early draft of one of the chapters (something about "cultural evolution" I think) several years ago. That got me thinking seriously about all this even moreso than Scott's brilliant essay.

Note that superforecasters read more news than the average person, and so are online a significant amount of time, yet they seem unaffected (this could be for many reasons, but is weak evidence against your theory).

I like this example.

Superforecasters are doing something real. If you make a prediction and you can clearly tell whether it comes about or not, this makes the process of evaluating the prediction mostly immune to stupefaction.

Much like being online a lot doesn't screw with your ability to shoot hoops, other than maybe taking time away from practice. You can still tell whether the ball goes in the basket.

This is why focusing on real things is clarifying. Reality reflects truth. Is truth, really, although I imagine that use of the word "truth" borders on heresy here.

Contrast superforecasters with astrologers. They're both mastering a skillset, but astrologers are mastering one that has no obvious grounding. Their "predictions" slide all over the place. Absolutely subject to stupefaction. They're optimizing for something more like buy-in. Actually testing their art against reality would threaten what they're doing, so they throw up mental fog and invite you to do the same.

Whenever you have this kind of grounding in reality, what you're working on is much, much more immune to stupefaction, which is the primary weapon of unFriendly egregores.

This was close to the heart of the main original insight of science.

It’s also unlikely to me that CFAR can scale up enough to make a dent in this problem.

Oh hell no. CFAR's hopes of mattering in this hypercreature war died around 2013. Partly because of my own stupidity, but also because no one involved understood these forces. Anna had some fuzzy intuitions and kind of panicked about this and tried to yank things in a non-terrible direction, but she had no idea which way that was. And stupid people like me dragged our feet while she did this, instead of tuning into what she was actually panicking about and working as a clear unified team to understand and orient to the issue.

So the thing became a lumbering bureaucratic zombie optimizing for exciting rationality-flavored workshops. Pretty website, great ops (because those are real!), fun conversations… but no teeth. Fake.

Lovely fantasy. Good luck!

Gosh, um…

I think I see where you are, and by my judgment you're more right than wrong, but from where I stand it sure looks like pain is still steering the ship. That runs the risk of breaking your interface to places like this.

(I think you're intuiting that. Hence the "crazy alert".)

I mean, vividly apropos of what you're saying, it looks to me like you've rederived a lot of the essentials of how symbiotic egregores work, what it's like to ally with them, and why we have to do so in order to orient to the parasitic egregores.

But the details of what you mean by "religion" and "cult" matter a lot, and in most interpretations of "extremely missionary" I just flat-out disagree with you on that point.

…the core issue being that symbiotic memes basically never push themselves onto potential hosts.

You actually hint at this: 

And it must be able to do this while they know it perfectly well, and consent before joining to begin with to its doing so - as obviously one which does so without consent is not aligned to true human values, even though, ironically, it has to be so good at rhetoric that consent is almost guaranteed to be given.

But I claim the core strategy cannot be rhetoric. The convergent evolutionary strategy for symbiotic egregores is to make the truth easy to see. Rather than rhetoric, the emphasis is clarity.

Why? Well, these memetic structures are truth-tracking. They spread by offering real value to their potential hosts and making this true fact vividly clear to their potential hosts.

Whereas rhetoric is symmetric. And its use favors persuasive memetic strategies, which benefits parasitic memes over symbiotic ones.

So what you'll find is that symbiotic memetic environments tend to develop anti-rhetoric psychotechnology. To the extend memetic symbiosis defines the environment, you'll find that revealing replaces persuasion, and invitations to understand replace mechanisms like social pressure to conform.

Less Wrong is a highly mixed space in this regard, but it leans way more symbiotic than most. Which is why your approach is likely to get downvoted into oblivion: A lot of what you're saying is meant to evoke strong emotions, which bypasses conscious reflection in order to influence, and that's a major trigger for the LW memetic immune system in its current setup for defending against memetic parasites.

I think the allergic reaction goes too far. It's overly quick to object to the "egregore" language as something like "a metaphor taken too far", which frankly is just flat-out incoherent with respect to a worldview that (a) is reductionist-materialist and yet (b) talks about "agents". It has over-extended the immune response to object to theme ("woo") instead of structure. And that in turn nurtures parasitic memes.

But the response is there because LW really is trying to the best of its ability to aim for symbiosis. And on net, talk of things like "I'm a prophet" tends to go in directions that are sorely memetically infected in ways the LW toolkit cannot handle beyond a generic "Begone, foul demon!"

…which, again, I think you kind of know. Which, again, is why you give the "crazy alert".

You might find my dissection of symbiotic vs. parasitic memes in this podcast episode helpful if you resonate with what I've just written here.

Well…I’d definitely read the book.

For my own take on this, read this

https://www.lesswrong.com/posts/yenr6Zp83PHd6Beab/which-singularity-schools-plus-the-no-singularity-school-was [LW · GW]

Spoiler Alert: I cover the same theme that AI-PONR has already happened in my TL;DR

I suspect that there's a particular sort of spirituality-adjacent recreational philosophy whose practice may make it easier to examine the meta-organisms you're describing. Even with it, they seem to often resist being named in a way that's useful when speaking to mixed company.

Absolutely. I used to feel victimized by this. Now I just build immunity, speak freely, and maybe some folk will hear me.

Can you point out some of the existing ones that meet your definition of Friendly?

That's a beautiful question.

It'd be my pleasure.

I tend to focus more on the unFriendly ones since they actively blind people. So I've thought in less detail about this branch of the memetic zoo. But I'll share a few bits I think are good examples:

• The art of knowing. Greek "mathema", Latin "sciencia". This is a wisdom thread that keeps building and rebuilding clarity & sanity. It's the core seed of deeply and humiliatingly sincere curiosity that gave birth to mathematics & science.
  • Not to be confused with the content or methods of modern social institutions though. Anytime you ossify parts of Friendly memes (like with RCTs or "the scientific method"), they provide an attack surface for Goodhart and thus for unFriendly hypercreatures. As a rule, unFriendly egregores lose power and "food" when truth rings through clearly, so they keep numbing and fogging access to the art of knowing wherever they can. (Ergo why e.g. most math classes teach computation, not math, and usually they do so via dominance & threat and in ways that feel utterly pointless and body-denying to the students.)
  • I think this is the core breath of life in Less Wrong. A deep remembrance that this is real and matters. (Sometimes LW drifts toward ossification though. Focus on Bayes and biases and the like tends in this direction.)
• Mettā. Not the meditative practice of loving-kindness, but what which those meditations cultivate. The kindness with no opposite and no exceptions. The Buddha called this an "bramavihara", which you can translate as something like "endless abode". It's something like the inner remembrance of Friendliness that's felt, not (just) understood, such that it shapes your thinking and behavior "from below". Much like the art of knowing, this is something you embody and become, not just cognitively understand.
• Death's clarity. Super unpopular in transhumanist circles because of the "death" part. It's super rare to hear people point at this one without going full deathist. ("Death is what makes life worth living!") But the fact remains, losing something precious tends to cut through bullshit and highlight what really, truly matters. If you can trust that and breathe through the pain and survive, you come out the other side clearer. You don't have to wait for Death to seek this clarity. Lots of spiritual traditions try to lean this way (e.g. "Memento Mori") — but like with science and mettā, there's some tendency to ossify the practices. Part of death's clarity is in the willingness to trust the unknown.
• BJJ/MMA pragmatism. Somehow a level less abstract than the above, arguably a child of the art of knowing. Both BJJ and MMA face something real and ground their ideas in that reality. Can you force a submission? Can you dish out blows without taking too many yourself? You can't think your way around that: either it happens or it doesn't. The empirical "Show me" tone in these arts cuts through stupefaction by pointing again and again at reality.
  • The Friendly thing here is the culture of training, not the fighting arts per se. The same thing could apply to chemistry or carpentry. Basically any non-bullshit skill that has some hard-to-miss embodied sign of progress.
• The right to a relevant voice in one's governance. This emerged most clearly in the Western Enlightenment, especially in the USA. The idea that if some egregore (especially the government) has power over you, then you should have influence over that egregore too, or be able to opt out of its influence. This is pretty explicitly a meme about (one strategy for) hypercreature alignment.
  • And as is par for the course, the stupifying ecosystem evolved attacks to this. Corporate special interests gridlocking government, the role of the town square now held in private tech companies' censorable platforms, and the emergence of forces like cancel culture are all corrosions. This is, in my mind, the main risk of wokism: It's actually trying to hunt and kill access to this meme.

In general, Friendly hypercreatures just don't act the same way as the unFriendly ones, with evolved arguments that hook people emotionally and with people identifying strongly with one side or another. They instead tend to loosen identity and encourage clarity, because they spread by actually offering value and therefore do best when their host humans can see that clearly. Most identities are stupid attack surfaces.

As a result, they don't clump people together around a behavior nearly as much as [LW · GW] unFriendly patterns do. Instead they usually work more like fields of intelligence. The way anyone can prove a math theorem and see the truth for themselves and thereby be immune to all social pressure. That's not localized. That's limited purely by someone's capacity-plus-willingness to look at reality. The "clumpiness" of social unity happens from having a culture that values, encourages, and develops methods for pointing people back at reality. No social pressure/threat needed hardly ever.

It is. It's just not yet fully self-aware. I'm inviting a nudge in that direction.

The egregores have an irreplacable function, and you can't personally remove them from yourself; instead, if you want saner egregores, this must involve constructing a new alliance with a solid footing in sanity.

This is in fact my stance. That didn't come across clearly in the OP. But e.g. science arose as a sane egregore, at least at first. (Though not all of "science", which has been significantly Goodharted in favor of less Friendly hypercreatures.)

" I think that the vast majority of a person's impact in this world is what hyperbeings he chooses to host/align to, with object level reality, barely even mattering."

I agree that personal values (no need to mystify) are important, but action is equally important. You can be very virtuous, but if you don't take action (by, for instance, falling for the Buddhist-like fallacy that sitting down and meditating will eventually save the world by itself), your impact will be minor. Specially in critical times like this. Maybe sitting down and meditating would be ok centuries ago where no transformative technologies were in sight. Now, with transformative technologies decade(s) off, it's totally different. We do have to save the world.

"I assert that alignment is trivially easy."

How can you control something vastly more intelligent than yourself (at least in key areas), or that can simply re-write its own code and create sub-routines, therefore bypassing your control mechanisms? Doesn't seem easy at all. (In fact, some people like Roman Yalmpolsky have been writing papers on how it's in fact impossible.) Even with the best compiler in the world (not no mention that nothing guarantees that progress in compilers will accompany progress in black boxes like neural networks).

"If our a measurement of "do I like this" is set to "does it kill me" I see this at best ending in a permanent boxed garden where life is basically just watched like a video and no choices matter, and nothing ever changes, and at worst becoming an eternal hell (specifically, an eternal education camp), with all of the above problems but eternal misery added on top."

I agree with this. The alignment community is way over-focused on x-risk and way under-focused on s-risk. But after this, your position becomes a bit ambiguous. You say:

"We should work together, in public, to develop AI that we like, which will almost certainly be hostile, because only an insane and deeply confused AI would possibly be friendly to humanity. "

What does this mean? That we should just build non-aligned AI, which would probably be hostile to humans, and therefore only kill us all instead of giving us eternal misery?

But wait, wasn't alignment trivially easy?

Unless you mean that, since "Alignment, as I use it, isn't becoming good. It's becoming something recognized as good.", then alignment, despite being easy, is a mistake and we should just build "AI that we like" which, I don't know, being more intelligent would therefore be more virtuous than humans? But that's extremely uncertain (as well as unlikely imo), and the stakes are way too high to take the gamble.

I don't know if there are superior entities to us playing these games, or if such memes are just natural collective tendencies. I don't think any of us know or can know, at least with current knowledge.

I don't understand the uncertainty. What's there to know?

Humans are natural collective tendencies of cells.

Is an ant colony an entity? I think it's fair to say yes.

It's not like there's some objective cosmic definition of an entity/agent. That's more like a human-mental way of interpreting a cluster of experiences. Entities are where we see them.

If you also mean working for a social/political solution, then I disagree..

Not what I mean. Of course individuals can work toward something like this. Obviously individuals have to for the collective to solve it. Just like an ant colony can't collect food unless ants go searching for it.

Although in practice most human efforts like this serve delusion. They're not nearly as helpful as they seem. They're usually anti-helpful.

(I probably shouldn't interact, but I would at least like to perform a small case study on what happened here, so I am going to try just out of curiosity.)

Human substrate is generally optimised for running one human, but can be repurposed for a variety of purposes. In particular, while memes can lodge themselves quite deeply inside someone, this process is quite inflexible, and generally humans run arbitary processes X by thinking 'what is the X thing to do here?. 

Somewhere between the point where [the-generative-process-that-has-generated-for-itself-the-name-'LightStar'] generated this comment, and the point where I read it, a human took 'LightStar's dialogue and typed it into a comment and submitted it.

I would like to clarify that I am speaking to that human, you, and I would like to hear from you directly, instead of generating 'LightStar' dialogue.

Could I ask you how you ended up here, and what you were doing when this happened?

I would advise that in the cases where people have a sudden revelation about rationality, they generally try to internalise it, and the case where they instead decide to give an internal generative process it's own lesswrong account and speak with every fourth sentence in italics is generally quite rare, and probably indicates some sort of modelling failure.

We generally use 'shard' in 'shard of Coordination' or 'shard of Rationality' to mean a fraction, a splinter, of the larger mathematical structures that comprise these fields. The 'LightStar' generative model has used the article 'the' in conjunction with 'shard', which as used here is kind of a contradiction - there is no 'the' with shard, it's only a piece of the whole. This distinction seems minor, but from my perspective it looks like it's at the center of 'LightStar'.

'LightStar' uses 'the' a lot about itself, describes itself as 'the voice of Humanity and Rationality and Truth', and while yes, there is only one correct rationality, I don't think 'LightStar' contains all of it, or is comprised only of a fragment of it, I think that whether or not 'LightStar' contains such a shard it also contains other parasitic material that results in actions taken that don't generally correspond to just containing such a shard.

I think this model is defective - try returning it to where you found it and getting another one, or failing that, see if they give refunds. I would be curious about your thoughts on the whole thing, where the 'you' in 'your' refers not to [the-generative-process-that-has-generated-for-itself-the-name-'LightStar'] but to the human that took that dialogue and typed it into the comment box.