I suspect it would still involve billions of $ of funding, partnerships like the one with Microsoft, and other for-profit pressures to be the sort of player it is today. So I don't know that Musk's plan was viable at all.

Note that all of this happened before the scaling hypothesis was really formulated, much less made obvious.

We now know, with the benefit of hindsight that developing AI and it's precursors is extremely compute intensive, which means capital intensive. There was some reason to guess this might be true at the time, but it wasn't a forgone conclusion—it was still an open question if the key to AGI would be mostly some technical innovation that hadn't been developed yet.

Those people don't get substantial equity in most business in the world. They generally get paid a salary and benefits in exchange for their work, and that's about it.

I don't think that's a valid inference.

Ok. So I haven't thought through these proposals in much detail, and I don't claim any confident take, but my first response is "holy fuck, that's a lot of complexity. It really seems like there will be some flaw in our control scheme that we don't notice, if we're stacking a bunch of clever ideas like this one on top of each other."

This is not at all to be taken as a disparagement of the authors. I salute them for their contribution. We should definitely explore ideas like these, and test them, and use the best ideas we have at AGI time. 

But my intuitive first order response is "fuck."

But he helped found OpenAI, and recently founded another AI company.

I think Elon's strategy of "telling the world not to build AGI, and then going to start another AGI company himself" is much less dumb / ethical fraught, than people often credit.

Thinking about this post for a bit shifted my view of Elon Musk a bit. He gets flack for calling for an AI pause, and then going and starting an AGI lab, and I now think that's unfair.

I think his overall strategic takes are harmful, but I do credit him with being basically the only would-be AGI-builder who seems to me to be engaged in a reformative hypocrisy strategy. For one thing, it sounds like he went out of his way to try to get AI regulated (talking to congress, talking to the governors), and supported SB-1047. 

I think it's actually not that unreasonable to shout "Yo! This is dangerous! This should be regulated, and controlled democratically!", see that that's not happening, and then go and try do it in a way that you think is better. 

That seems like possibly an example of "follower-conditional leadership." Taking real action to shift to the better equilibrium, failing, and then going back to the dominant strategy given the inadequate equilibrium that exists.

Obviously he has different beliefs than I do, and than my culture does, about what is required for a good outcome. I think he's still causing vast harms, but I think he doesn't deserve the eye-roll for founding another AGI lab after calling for everyone to stop.

You maybe right. Maybe the top talent wouldn't have gotten on board with that mission, and so it wouldn't have gotten top talent.

I bet Illya would have been in for that mission, and I think a surprisingly large number of other top researchers might have been in for it as well. Obviously we'll never know.

And I think if the founders are committed to a mission, and they reaffirm their commitment in every meeting, they can go surprisingly far in making in the culture of an org.

Also, Sam Altman is a pretty impressive guy. I wonder what would have happened if he had decided to try to stop humanity from building AGI, instead of trying to be the one to do it instead of google.

Absolutely true. 

But also Altman's actions since are very clearly counter to the spirit of that email. I could imagine a version of this plan, executed with earnestness and attempted cooperativeness, that wasn't nearly as harmful (though still pretty bad, probably). 

Part of the problem is that "we should build it first, before the less trustworthy" is a meme that universalizes terribly. 

Part of the problem is that Sam Altman was not actually sincere in the the execution of that sentiment, regardless of how sincere his original intentions were.

I predict this won't work as well as you hope because you'll be fighting the circadian effect that partially influences your cognitive performance.

Also, some ways to maximize your sleep quality are too exercise very intensely and/or to sauna, the day before.

It's possible no one tried literally "recreate OkC", but I think dating startups are very oversubscribed by founders, relative to interest from VCs

If this is true, it's somewhat cruxy for me.

I'm still disappointed that no one cared enough to solve this problem without VC funding.

I only skimmed the retrospective now, but it seems mostly to be detailing problems that stymied their ability to find traction.

Right. But they were not relentlessly focused on solving this problem. 

I straight up don't believe that that the problems outlined can't be surmounted, especially if you're going for a cashflow business instead of an exit.

But I think you're trying to draw an update that's something like "tech startups should be doing an unbiased search through viable valuable business, but they're clearly not", or maybe, "tech startups are supposed to be able to solve a large fraction of our problems, but if they can't solve this, then that's not true", and I don't think either of these conclusions seem that licensed from the dating data point.

Neither of those, exactly. 

I'm claiming that the narrative around the startup scene is that they are virtuous engines of [humane] value creation (often in counter to a reactionary narrative that "big tech" is largely about exploitation and extraction). It's about "changing the world" (for the better). 

This opportunity seems like a place where one could have traded meaningfully large personal financial EV for enormous amounts of humane value. Apparently no founder wanted to take that trade. Because I would expect there to be variation in how much funders are motivated by money vs. making a mark on the world vs. creating value vs. other stuff, that fact that (to my knowledge) no founder went for it, is evidence about the motivations of the whole founder class. The number of founders who are more interested in creating something that helps a lot of people than they are in making a lot of money (even if they're interested in both) is apparently very small.

Now, maybe startups actually do create lots of humane value, even if they're created by founders and VC's motivated by profit. The motivations of of the founders are only indirect evidence about the effects of startups. 

But the tech scene is not motivated to optimize for this at all?? That sure does update me about how much the narrative is true vs. propaganda.

Now if I'm wrong and old OkCupid was only drastically better for me and my unusually high verbal intelligence friends, and it's not actually better than the existing offerings for the vast majority of people, that's a crux for me.

You mention manifold.love, but also mention it's in maintenance mode – I think because the type of business you want people to build does not in fact work.

From their retrospective:

Manifold.Love is going into maintenance mode while we focus on our core product. We hope to return with improvements once we have more bandwidth; we’re still stoked on the idea of a prediction market-based dating app!

It sounds less like they found it didn't work, and more like they have other priorities and aren't (currently) relentlessly pursing this one.

I didn't say Silicon Valley is bad. I said that the narrative about Silicon Valley is largely propagnada, which can be true independently of how good or bad it is, in absolute terms, or relative to the rest of the world.

Yep. I'm aware, and strongly in support. 

But it took this long (and even now, isn't being done by a traditional tech founder). This project doesn't feel like it ameliorates my point.

The fact that there's a sex recession is pretty suggestive that tinder and the endless stream of tinder clones doesn't serve people very well. 

Even if you don't assess potential romantic partners by reading their essays, like I do, OkC's match percentage meant that you could easily filter out 95% of the pool to people who are more likely to be compatible with you, along whatever metrics of compatibility you care about.

That no one rebuilt old OkCupid updates me a lot about how much the startup world actually makes the world better

The prevailing ideology of San Francisco, Silicon Valley, and the broader tech world, is that startups are an engine (maybe even the engine) that drives progress towards a future that's better than the past, by creating new products that add value to people's lives.

I now think this is true in a limited way. Software is eating the world, and lots of bureaucracy is being replaced by automation which is generally cheaper, faster, and a better UX. But I now think that this narrative is largely propaganda.

It's been 8 years since Match bought and ruined OkCupid and no one, in the whole tech ecosystem, stepped up to make a dating app even as good as old OkC is a huge black mark against the whole SV ideology of technology changing the world for the better.

Finding a partner is such a huge, real, pain point for millions of people. The existing solutions are so bad and extractive. A good solution has already been demonstrated. And yet not a single competent founder wanted to solve that problem for planet earth, instead of doing something else, that (arguably) would have been more profitable. At minimum, someone could have forgone venture funding and built this as a cashflow business.

It's true that this is a market that depends on economies of scale, because the quality of your product is proportional to the size of your matching pool. But I don't buy that this is insurmountable. Just like with any startup, you start by serving a niche market really well, and then expand outward from there. (The first niche I would try for is by building an amazing match-making experience for female grad students at a particular top university. If you create a great experience for the women, the men will come, and I'd rather build an initial product for relatively smart customers. But there are dozens of niches one could try for.)

But it seems like no one tried to recreate OkC, much less creating something better, until the manifold team built manifold.love (currently in maintenance mode)? Not that no one succeeded. To my knowledge, no else one even tried. Possibly Luna counts, but I've heard through the grapevine that they spent substantial effort running giant parties, compared to actually developing and launching their product—from which I infer that they were not very serious. I've been looking for good dating apps. I think if a serious founder was trying seriously, I would have heard about it. 

Thousands of funders a year, and no one?!

That's such a massive failure, for almost a decade, that it suggests to me that the SV ideology of building things that make people's lives better is broadly propaganda. The best founders might be relentlessly resourceful, but a tiny fraction of them seem to be motivated by creating value for the world, or this low hanging fruit wouldn't have been left hanging for so long.

This is of course in addition to the long list of big tech companies who exploit their network-effect monopoly power to extract value from their users (often creating negative societal externalities in the process), more than creating value for them. But it's a weaker update that there are some tech companies that do ethically dubious stuff, compared to the stronger update that there was no startup that took on this obvious, underserved, human problem.

My guess is that the tech world is a silo of competence (because competence is financially rewarded), but operates from an ideology with major distortions / blindspots, that are disconnected from commonsense reasoning about what's Good. eg following profit incentives, and excitement about doing big things (independent from whether those good things have humane or inhumane impacts) off a cliff.

my current guess is that the superior access to large datasets of big institutions gives them too much of an advantage for me to compete with, and I'm not comfortable with joining one.

Very much a side note, but the way you phrased this suggest that you might have ethical concerns? Is that right? If so, what are they?

Advice that looked good: buy semis (TSMC, NVDA, ASML, TSMC vol in particular) 

Advice that looked okay: buy bigtech

Advice that looked less good: short long bonds 

None of the above was advice, remember! It was...entertainment or something?

Thank you for writing this! I would love to see more of this kind of analysis on LessWrong.

You have been attacked by a pack of stray dogs twice?!?!

During the period when I was writing down dreams, I was also not using an alarm. I did train myself to wake up instantly, at the time that I wanted, to the minute. I agree that the half-awake state is anti-helpful for remembering dreams.

The alarm is helpful for starting out though.

We currently have too many people taking it all the way into AI town.

I reject the implication that AI town is the last stop on the crazy train.

I don't have much information about your case, but I'd make a 1-to-1 bet that if you got up and wrote down your dreams first thing in the morning every morning, especially if you're woken up by an alarm for the first 3 times, that you'd start remembering your dreams. Just jot dow whatever you remember, however vague or instinct, upto and including "litterally nothing. The the last thing I remember is going to bed last night."

I rarely remember my own dreams, but in periods of my life when I've kept a dream journal, I easily remembered them.

I think that, in almost full generality, we should taboo the term "values". It's usually ambiguous between a bunch of distinct meanings.

• The ideals that, when someone contemplates, invoke strong feelings (of awe, motivation, excitement, exultation, joy, etc.)
• The incentives of an agent in a formalized game with quantified payoffs.
• A utility function - one's hypothetical ordering over words, world-trajectories, etc, that results from comparing each pair and evaluating which one is better.
• A person's revealed preferences.
• The experiences and activities that a person likes for their own sake.
• A person's vision of an ideal world. (Which, I claim, often reduces to "an imagined world that's aesthetically appealing.")
• The goals that are at the root of a chain or tree of instrumental goals.
  • [This often comes with an implicit or explicit implication that most of human behavior has that chain/tree structure, as opposed being, for instance, mostly hardcoded adaptions, or a chain/tree of goals that grounds out in a mess of hardcoded adaptions instead of anything goal-like.]
• The goals/narratives that give meaning to someone's life.
  • [It can be the case almost all one's meaning can come through a particular meaning-making schema, but from a broader perspective, a person could have been ~indifferent between multiple schema. 
    
    For instance, for some but not most EAs, EA is very central to their personal meaning-making, but they could easily have ended up as a social justice warrior, or a professional Libertarian, instead. And those counterfactual worlds, the other ideology is similarly central to their happiness and meaning-making. I think in such cases, it's at least somewhat confused if to look at the EA and declare that "maximizing [aggregate/average] utility" is their "terminal value". That's papering over the psychological process that adopts ideology or another, which is necessarily more fundamental than the specific chosen ideology/"terminal value".
    
    It's kind of like being in love with someone. You might love your wife more than anything, she might be the most important person in your life. But if you admit that it's possible that if you had been in different communities in your 20s you might have married someone else, then there's some other goal/process that picks who to marry. So to with ideologies.]
• Behaviors and attitudes that signal well regarded qualities.
• Core States.
• The goals that are sacred to a person, for many possible meanings of sacred.
• What a person "really wants" underneath their trauma responses. What they would want, if their trauma was fully healed.
• The actions make someone feel most alive and authentically themselves.
• The equilibrium of moral philosophy, under arbitrary reflection.

Most of the time when I see the word "values" used on LessWrong, it's ambiguous between theses (and other) meanings. 

A particular ambiguity: sometimes "values" seem to be referring to the first-person experiences that a person likes for their own sake ("spending time near beautiful women is a terminal value for me"), and other times it seems to be referring to a world that a person thinks is awesome, when viewing that world from a god's eye view. Those are not the same thing, and they do not have remotely the same psychological functions! Among other differences, one is a near-mode evaluation, and the other is a far-mode evaluation.

Worse than that, I think there's often a conflation of these meanings. 

For instance, I often detect a hidden assumption that that the root of someone's tree of instrumental goals is the same thing as their ranking over possible worlds. I think that conflation is very rarely, if ever, correct: the deep motivations of a person's actions are not the same thing as the hypothetical world that is evaluated as best in thought experiments, even if the later thing is properly the person's "utility function". At least in the vast majority of cases,  one's hypothetical ideal world has almost no motivational power (as a matter of descriptive psychology, not of normative philosophy).

Also (though this is the weakest reason to change our terminology, I think), there's additional ambiguity to people who are not already involved in the memeplex. 

To broader world "values" usually connotes something high-minded or noble: if you do a corporate-training-style exercise to "reflect on your values", you get things like "integrity" and "compassion", not things like "sex" or "spite". In contrast, LessWrongers would usually count sex and spite, not to mention boredom and pain, as part of "human values" and many would also own them as part of their personal values.

Are you saying this because you worship the sun?

Yes! that analogy is helpful for communicating what you mean!

I still have issues with your thesis though.

I agree that this "explaining away" thing could be a reasonable way to think about if eg the situation where I get sick, and while I'm sick, some activity that I usually love (let's say singing songs) feels meaningless. I probably shouldn't conclude that "my values" changed, just that the machinery that implements my reward circuitry is being thrown off by my being sick.

On the other hand, I think I could just as well describe this situation as extending the domain over which I'm computing my values. eg "I love and value singing songs, when I'm healthy, but when I'm sick in a particular way, I don't love it. Singing-while-healthy is meaningful; not singing per-se." 

In the same way, I could choose to call the blue screen phenomenon an error in the TV, or I could include that dynamic as part of the "predict what will happen with the screen" game. Since there's no real apple that I'm trying to model, only an ephemeral image of the apple, there's not a principled place to stand on whether to view the blue-screen as an error, or just part of the image generating process. 

For any given fuckery with my reward signals, I could call them errors, misrepresenting my "true values" or I could embrace them as expressing a part of my "true values." And if two people disagree about which conceptualization to go with, I don't know how they could possibly resolve it. They're both valid frames, fully consistent with the data. And they can't get distinguishing evidence, even in principle.

(I think this is not an academic point. I think people disagree about values in this way reasonably often.

Is enjoying masturbating to porn an example of your reward system getting hacked by external super-stimuli, or is that just part of the expression of your true values? Both of these are valid ways to extrapolate from the reward data time series. Which things count as your reward system getting hacked, and which things count as representing your values. It seems like a judgement call!

The classic and most fraught example is that some people find it drop dead obvious that they care about the external world, and not just their sense-impressions about the external world. They're horrified by the thought of being put in an experience machine, even if their subjective experience would be way better. 

Other people just don't get this. "But your experience would be exactly the same as if the world was awesome. You wouldn't be able to tell the difference", they say. It's obvious to them that they would prefer the experience machine, as long as their memory was wiped so they didn't know they were in one.^[1])

Talking about an epistemic process attempting to update your model of an underlying not-really-real-but-sorta structure seems to miss the degrees of freedom in the game. Since there's no real apple, no one has any principled place to stand in claiming that "the apple really went half blue right there" vs. "no the TV signal was just interrupted." Any question about what the apple is "really doing" is a dangling node. ^[2]


As a separate point, while I agree the "explaining away disruptions" phenomenon is ever a thing that happens, I don't think that's usually what's happening when a person reflects on their values. Rather I guess that it's one of the three options that I suggested above.

1. ^{^}

   Tangentially, this is why I expect that the CEV of humans diverges. I think some humans, on maximal reflection, wirehead, and others don't.

2. ^{^}

   Admittedly, I think the question of which extrapolation schema to use is itself decided by "your values", which ultimately grounds out in the reward data. Some people have perhaps a stronger feeling of indigence about others hiding information from them, or perhaps a stronger sense of curiosity, or whatever, that crystalizes into a general desire to know what's true. Other people have less of that. And so they have different responses to the experience-machine hypothetical.
   
   Because which extrapolation procedure any given person decides to use is itself a function of "their values" it all grounds out in the reward data eventually. Which perhaps defeats my point here.

Huh. This was quite helpful / motivating for me.

Something about the updatelessness of "if I had to decide when I was still beyond the veil of ignorance, I would obviously think it was worth working as hard as feasible in the tiny sliver of probability that I wake up on the cusp of the hinge of history, regardless of how doomed things seem." 

It's a reminder that even if things seem super doomed, and it might feel like I have no leverage to fix things, I actually actually one of the tiny number of beings that has the most leverage, when I zoom out across time.

Thanks for writing this.

At some point I sit down and think about escamoles. Yeah, ants are kinda gross, but on reflection I don’t think I endorse that reaction to escamoles. I can see why my reward system would generate an “ew, gross” signal, but I model that reward as being the result of two decoupled things: either a hardcoded aversion to insects, or my actual values. I know that I am automatically averse to putting insects in my mouth so it's less likely that the negative reward is evidence of my values in this case; the signal is explained away in the usual epistemic sense by some cause other than my values.

I have a "what?" reaction to this part. Unpacking that makes me doubt this whole frame.

From your description earlier in this post, it sounds like your "values" are a sort of hypothetical construct, that's shape is constrained by the reward-signal information. I think you're not positing that there's a physical structure somewhere in the brain, which encodes a human's values in full, our beliefs about our values are mediated by reward signals.

Values aren't a "real thing" that you can discover facts about, and in an important sense there's "just the reward signal", and our beliefs about the underlying function(s) that determine the outputs of the reward signal. The values themselves aren't in there somewhere, separately from the reward circuit.

Given that, I don't know what it would mean to update that any particular instance of negative reward is "explained away" by some cause other than the values. 

It seems like one of several things can be happening in that situation:

1. You adopt some abstract beliefs about this made up construct that you call "your values." The underlying reward input-output do change at all. Nothing changed except your self model.
2. The there were two parts / subcomponents of your brain (I speculate, literal subnetworks) each of which were hooked into the reward circuitry. Those networks interact, somehow and at least one of them updates the other. Some implicit models underlying the reward-signal function, updates, and so you have at least a partially different "yuck"/"yum" response to escamoles.
3. A combination of 1 and 2: You adopt your abstract beliefs about this hypothetical thing called "your values". Because of an important complexity of human motivation, one's abstract beliefs about one's "values" is an input to the reinforcement circuitry (both because your beliefs about yourself influence your yuck/yum reactions directly, and because your beliefs about yourself influence your actions, which can differentially reinforce some reactions). Changing your self-model changes your self. 
   1. Importantly, because of this loopiness, there can be degrees of freedom, there might be multiple stable attractors in the space of self-models/reward-outputs. Which means that the process of "updating your abstract beliefs" is not strictly a matter of epistemics, it's a matter of agency: you can choose what you want your "values" to be, to the exact extent that your abstract beliefs influence the reward circuitry and no more.

None of those were an epistemic process of updating your model of your "values" to better conform to the evidence about what your "values" are. All of these are about bidirectional interactions between the reward function(s) and various implicit or explicit beliefs^[1]. But they don't seem well modeled by an epistemic process of "My values are a static thing that are out there/ in here, and I'm doing Bayesian updates to model those values more and more accurately."

 

1. ^{^}

   Well, except for the first one, which was a unidirectional interaction. 

Notice that the wording of each example involves beliefs about values. They’re not just saying “I used to feel urge X, but now I feel urge Y”. They’re saying “I thought I wanted X” - a belief about a value! Or “now I think that was more a social script than my own values” - again, a belief about my own values, and how those values relate to my (previous) behavior. Or “I endorsed the view that Z is the highest objective” - an explicit endorsement of a belief about values. That’s how we normally, instinctively reason about our own values. And sure, we could reword everything to avoid talking about our beliefs about values - “learning” is more general than “learning about” - but the fact that it makes sense to us to talk about our beliefs about values is strong evidence that something in our heads in fact works like beliefs about values, not just reinforcement-style “learning”.

Importantly, this isn't the only way that people talk about their values.

Sometimes a person will say "I used to care deeply about X, but as I got older, I don't care as much", or "Y used to be the center of my life, but that was a long time ago", etc.

In those cases the person isn't claiming to have been mistaken about their values. Rather their verbiage expresses that they correctly ascertained their values, but their values themselves changed over time.

This could just be a matter of semantics, but these could also be distinct non-mutually exclusive phenomena. Sometimes we learn more about ourselves and our beliefs about our values change. Sometimes we change, not just our beliefs about ourselves and our values. 

Yeah, I think it's anti-natural for such an entity to exist, and that's a crux for religion-according-to-Alex.

Fantastic. I would love to figure out of this is true. 

It's also a crux for my worldview too. A Double Crux!

I can grant that some pathological liars didn't pick it up culturally, and kind of just have fucked up genetics. But I still prefer to view them as a person who would ultimately actually prefer truth but took a wrong turn at some point, than as someone who is just fundamentally in their core a pathological liar. I think, with the technologies available today, me viewing them as a mistake theorist does not lead to substantially different actions from me viewing them as a conflict theorist, except perhaps that I might be more attuned to the suffering they're going through for having such a deeply entrenched false belief about the best way to live their life.

This is sus.

In an effort to assert that everyone is basically good, we've found ourselves asserting that people who are very psychologically different than us, who were born that way, are fundamentally mistaken, at a genetic level?

I at least want to be open to the possibility of the sociopath who straightforwardly likes being a sociopath—it works for them, they sincerely don't want to be different, they think it would be worse to be neurotypical. Maybe, by their own, coherent lights, they're just right about that.

(In much the same way that many asexuals look at the insanity of sexual desire and think "why would I want to be like that?!", while the normally-sexed people would absolutely not choose delete their sex-drive.)
 

Thank you!

Yeah, I should probably write these up. I called this "action-oriented operationalization" (in contrast to prediction-oriented operationalization) and at least part of the credit goes to John Salvatier for developing it.

But just as a question of are they a good person with bad parts clumped on top or are they just a bunch of good and bad parts and they could probably become a different person. You seem to be a bit essentialist about, "No, there's a good person there and they just have some issues," whereas I'm like, "Here's a person and here's some of their properties. We would like to change which person they are because this person utterly sucks."

Mistake versus conflict theory are both valid self-fulfilling prophecies one could use to model someone, and I think I'm less being an essentialist about people being intrinsically good, and more insisting on mistake theory being a self-fulfilling prophecy that wins more, in that I think it gets you strictly more degrees of freedom in how you can interact with them. 

I strongly agree with the bolded part, and have often been in the position of advocating for this "spiritual stance".

But I feel like this answer is kind of dodging the question of "which frame is actually a more accurate model of reality?" I get that they're frames, and you can use multiple frames to describe the same phenomenon. 

But the "people are basically good underneath their trauma" and the stronger form "all people are basically good underneath their trauma" do make specific predictions about how those people would behave under various circumstances. 

Specifically, I'm tracking at least three models:

• Almost everyone is traumatized, to various degrees. that trauma inhibits their ability to reason freely about the world, and causes them to respond to various stimuli with long-ago-learned not-very-calibrated flailing responses. Their psychology is mostly made of a stack of more-or-less reflective behavioral patterns and habits which developed under the constraints of the existing psychological ecosystem of behaviors and patterns. but underneath all of that is an forgotten, but ever present Buddha-nature of universal love for all beings.

vs.

• Almost everyone is traumatized, to various degrees. that trauma inhibits their ability to reason freely about the world, and causes them to respond to various stimuli with long-ago-learned not-very-calibrated flailing responses. Their psychology is mostly made of a stack of more-or-less reflective behavioral patterns and habits which developed under the constraints of the existing psychological ecosystem of behaviors and patterns. but underneath all of that is an forgotten, but if that trauma is cleared that mind is a basically-rational optimizer for one's personal and genetic interests.

vs.

• Almost everyone is traumatized, to various degrees. that trauma inhibits their ability to reason freely about the world, and causes them to respond to various stimuli with long-ago-learned not-very-calibrated flailing responses. Their psychology is mostly made of a stack of more-or-less reflective behavioral patterns and habits which developed under the constraints of the existing psychological ecosystem of behaviors and patterns. but underneath all of that is an forgotten, but underneath that is a pretty random cluge of evolutionary adaptions.

Obviously, reality will be more complicated than any of these three simplifications, and is probably a mix of all of them to various degrees. 

But I think it is strategically important to have a model of 1) what the underlying motivational stack for most humans is like and 2) how much variation is there in underlying motivations, between humans. 

I also think they tend to be pretty unkind to themselves and miserable in a particular way,

This is not my understanding of the psychology of most literal clinical sociopaths. It is my read of, say Stalin.

Okay. Yeah, I think there are more datapoints in steel-Islam, for how to navigate the fact that even though cancers might be mistakes, you sometimes maybe still need to kill them anyway.

In reference to some of my other comments here, I would be much more sympathetic to this take if the historical situation was that Mohamed led wars of self defense against aggressors. But he lead wars of conquest in the name of Islam, which continued for another century after his death.

As Jesus says, "by your fruits you will know them." 

@Ben Pace, this should have a note that says "spoilers for unsong", and spoiler tags over the offending sentences?

I think it would be more like "here's the correct way to interpret these stories, in contrast to these other interpretations that a majority of people currently used to interpret them".

So like, there's already a whole cottage industry of interpretations of religious texts. Jordan Peterson, for instance, gives a bunch of "psychological" readings of the meaning of the bible, which got attention recently. (And notably, he has a broadly pluralistic take on religion). 

But there are lots and lots of rabbis and ministers and so on promoting their own interpretations or promoting interpretations, many of which are hundreds of years old. There's a vast body of scholarship regarding what these texts mean and how to interpret them.

Alex, it sounds like you hope to add one more interpretation to the mix, with firmer mathematical rigor. 

Do you think that that one will be taken up by some large fraction of Christians, Muslims, Jews, or Buddhists? Is there some group that will see the formalizations and recognize them as self-evidently the truth that their personal religious tradition was pointing at?

 

But also, I am annoyed again.

FYI @Ben Pace, as a reader, I resonate with your annoyance in this conversation.

For me, I agree with many of Alex's denotational claims, but feel like he's also slipping in a bunch of connotation that I think is mostly wrong, but because he's agreeing with the denotation for any particular point you bring up, it feels slippery, or like I'm having one pulled over on me. 

It has a motte-and-bailey feel to it. Like Alex will tell me that "oh of course Heaven and Hell are not literal places under the earth and in the sky, that's obviously dumb." But, when he goes to talk with religious people, who might think that, or something not quite as wrong as that, but still something I think is pretty wrong, he'll talk about heaven and hell, without clarifying what hem means, and it will seem like they're agreeing, but the communitive property of agreement between you and Alex and Alex and the religious person doesn't actually hold.

Like, it reads to me like Alex keeps trying to say "you and I Ben, we basically agree", and as an onlooker, I don't think you actually agree, on a bunch of important but so-far-inexplicit points.

[Ironically, if Alex does succeed in his quest of getting formal descriptions of all this religious stuff, that might solve this problem. Or at least solve it with religious people who also happen to be mathematicians.]

I don't know if that description matches your experience, this is just my take as an onlooker.

but the pulse I got was like, exclusivism is waning, it's losing popularity very fast, and pluralism is on the rise.

My read, from my very limited perspective mostly engaging with new-age folks, and my serious-Christian highschool peers, as a teenager, is that this is mostly a response to increased secularism and waning traditional religious influence. 

Enlightenment, post-Westphalian norms are very heavy on freedom of religion. As the influence of religions has waned, and societal mores have come to rest more and more on the secular morality. As an example, compare how in the 1950s, the question of whether someone "a good person" was loaded on whether they were an upstanding christian, and in 2010, the question of whether someone is "a good person" is much more highly loaded on whether they're racist.  

In that process, insisting on religious exclusivism has become kind of goshe—it makes you seem like a backwards, bigoted, not-very cosmopolitan, person.

So how can religions respond? They can double down on calling out how mainstream society is sinful and try to fight the culture war. Or they can get with the program, and become hip and liberal, which means being accepting of other religions, to the point of saying that they're all paths up the same mountain, in defiance of centuries of doctrine and literal bloody violence.

And in practice, different religious communities have taken either one or some combination of these paths.

But this doesn't represent an awakening to the underlying metaphysical truth underlying all religions (which again, I'm much more sympathetic to than many). It mostly represents, religious institutions adapting to the pressures of modernity and secularism.

It looks to me like mostly a watering down of religion, rather than an expression of some deep spiritual impulse.

It's basically the same situation as regards doctrine on evolution by natural selection. The church fought that one, tooth and nail, for centuries, declaring that it was an outright. But secular culture eventually (mostly) beat the church in that culture war, and only now, to stay relevant, does the church backslide putting forward much weaker versions of their earlier position, whereby evolution is real, but compatible with divine creation. It's confused to look at that situation and attribute it to the Church's orientation to religion, all along, being more correct than we gave it credit for. 

And the sociology matters here because makes different predictions about what will accelerate and increase this trend. If I'm correct, and this shift is mostly downstream of religions losing their influence, compared broader secular mores and values, then we won't get more of it by increasing the importance, salience, of religion.

Granted, I'll give you that much of the world is still religious, and maybe the best way to bridge to them to produce the transformations the world needs, is through their religion. But I think you're overstating how much religious pluralism is a force on the rise, as opposed to a kind of weak-sauce reaction to external forces.

And FWIW, there's something cruxy for me here. I would be more interested in religions if I thought that there was a positive rising force of religious pluralism (especially one that was grounded in the mystical traditions of the various religions), instead of reactive one.
 

I talked to a Catholic priest at the Parliament who'd said something like, "there was a girl I know who started out Christian, but had a really bad experience with Christianity, but then found something spiritually appealing in a Hindu temple, and I encouraged her to find God in Hinduism", or something like that.

It's very relevant to me in discussing the value of religions that this is NOT Catholic doctrine. I'm pretty sure that this is a heresy, in that it contradicts a core tenet of Christian doctrine: the exclusivity of Christ.

A key component of Christian theology, since the at least shortly after the crucifixion, is that salvation is found uniquely through Jesus Christ. 

Now, I've read books about the Christian / Jewish mystical traditions that state otherwise: putting forward that Jesus was not the Messiah but a Messiah. I agree that some religious people have ever expressed that perspective. (I think, for some meanings of Messiah, it's even true. ) 

The most charitable interpretation is that state of affairs is this is the secret esoteric meaning, accessible only to elite elite initiates (and/or repeatedly rediscovered by mystics), as distinct from the simple stories taught to the masses. But even that interpretation is a stretch: beliefs like this one are explicitly heretical to the explicit, enforced (though less so than in previous eras, see my other comment), doctrine of the organized religions. Proponents of those views were often excommunicated for expressing them.

That's pretty cruxy for me with regards to my attitudes about religion. I believe there are a few religious pluralists, and I probably like, and maybe agree, with a lot of them. But the thing that they're doing, which we both think is cool, is generally expressly forbidden/denied by institutionalized religion (at least in the West—I don't know enough about non-Abrahamic institutionalized religions to say one way or the other).

My read of your statements is that they're they're giving too much credit to the Catholic Priesthood, and the Mormon priesthood, and the community of Protestant ministers, because there are a tiny number of religious pluralists who are expressing views and attitudes that those much larger organizational structures explicitly deny.

It seems like you're doing the opposite of throwing out the baby with the bathwater—refusing to throw out old, dirty, bathwater because...there's a small chunk of soap, in it or something. (I'm aware I'm staining the analogy). If your message was "hey: there's some useful soap in this bathwater—we should take it out and use it", I would be sympathetic. But your rhetoric reads to me as much more conciliatory than that, "yeah this bathwater isn't perfect, but it's actually pretty good!"

This is admittedly in the connotation, not the detonation. I expect we agree about most of the facts on the ground. But my impression of your overall attitude is that it's not accurately representing organized religions as a class in their actual expressed views and their actual behavior and impacts on the world.

10% of the world is way too many people. I'm not sure a lot of them would be capable of comprehending a lot of the material I've read, given the distribution of cognitive and reading ability in the world. It would probably have to be something written for most people to read, not like ~anything on LessWrong. Like, if HPMOR could work I'd pick that. If not, perhaps some Dostoyevsky? But, does 10% of the world speak English? I am confused about the hypothetical.

The idea of this conversational technique is that you can shape the hypothetical to find one where the two of you have strong, clear, differing intuitions. 

If you're like "IDK man, most people won't even understand most of the books that I think are important, and so most of the problem is figuring out something that 'works' at all, not picking the best thing", you could adjust the hypothetical, accordingly.  What about 10% of the global population sampled randomly from people who have above 110 IQ, and if they're not english speakers they get a translation? Does that version of the hypothetical give you a clearer answer?

Or like (maybe this is a backwards way to frame things but) I would guess^[1] that there's a version of some question like this to which you would answer the sequences, or something similar, since it seems like your take is "[one of] the major bottleneck[s] in the world is making words mean things." Is there a version that does return the sequences or similar?
 

FYI, I feel interested in these answers and wonder if Alex disagrees with either the specific actions or something about the spirit of the actions. 

For instance, my stereotype of religious prophets, is that they don't dedicate their life to taking down, or prosecuting a particular criminal. My personal "what would Jesus / Buddha do?" doesn't return "commit my life to making sure that guy gets jail time." Is that an "in" towards your actual policy differences (the situations in the world where you would make different tradeoffs)?

1. ^{^}

   Though obviously, don't let my guesses dictate your attitudes. Maybe you don't actually think anything like that!

Yeah... something about this conversation's annoying. I can't figure out what it is.

You're what I call "agreeing to agree". 

(shit, I guess I never published that blog post, or the solution to it. Uh. this comment is a placeholder for me.)

My recommendation is to sidestep taking about propositions that you agree with or disagree with, and generate hypothetical or real situations, looking for the ones where you're inclined to take different actions. In my experience, that surfaces real disagreement better than talking about what you believe (for a number of reasons).

eg

• If you had the option to press a button to cut the number of observant christians in the world by half (they magically become secular atheists) would you press it? What if you have to choose between doubling the number of observant christians or halving the number, with no status quo option?
• If could magically cause 10% of the people in the world to have read a single book, what book? (With the idea being that maybe Alex would recommend a religious book, and Ben probably wouldn't.)
• What attitude do each of you have about SBF or FTX? Do you have some joy about his getting jail time? What would each of you do if you were the judge of his case?
• Same question but for other FTX employees.
• If Ben catches a person outright lying on LessWrong, what should he do?
• [possibly unhelpfully personal] Should Ben have done anything different in his interactions with Nonlinear?

(Those are just sugestions. Probably you would need to generate a dozen or so, and tweak and iterate until you find a place where you have different impulses for what to do in that situation.)

If you find an example where you take different actions or have a different attitude, now you're in a position to start trying to find cruxes.

Personally, I'm interested: does this forgiveness business suggest any importantly different external actions? Or is it mostly about internal stance? 

I think a robust solution to this is as hard as the AI alignment problem.

Why?

For what it's worth, I think Martin Luther King and Gandhi are examples of people who had done things in the world inspired by religious exemplars

Yes, but it sounds like you're not mostly interested in reading about their stories. You're interested in reading about the stories of prophets, who, overall, have had a mixed impact on the world.

Because I am unusually good at these things, I am also unusually good at earning the trust of people I encounter, from a broad array of backgrounds. 

From my personal experience with you, Alex, I agree that you seem unusually good at earning the (at least a certain kind of) trust of people I encounter, from a broad array of background. I don't particularly attribute that to the forgiveness thing? More-so to the fact that you're friendly, and have a unusual-for-humans ability and inclination to pass ITTs.

it seems like the only other people who care about being better in this way are super into religion/spirituality... well

How literally true is this? I would be very interested any examples of secular / non-religious heroes of this type, to help me triangulate the concept you're pointing at.

While I don't necessarily think that any particular religion is likely to be a particularly good gateway for these insights to a randomly chosen person, I do think these insights can be found in the religions if you look in the right places and interpret them in the right way

Is there a further claim that those insights can really be gotten from other venues? Or that, even though they're not necessarily a good gateway for any particular person), they're still better than the alternatives?