Nice post! I like the ladder metaphor.

For events, one saving grace is that many people actively dislike events getting too large and having too many people, and start to long for the smaller cozier version at that point. So instead of the bigger event competing with the smaller one and drawing people away from it, it might actually work the other way around, with the smaller event being that one that "steals" people from the bigger one.

Previous LW discussion about the Inner Ring: [1, 2].

Good question! I would find it plausible that it would have changed, except maybe if the people you'd call would be in their fifties or older.

Based on the link, it seems you follow the Theravada tradition. 

For what it's worth, I don't really follow any one tradition, though Culadasa does indeed have a Theravada background.

Yeah, some Buddhist traditions do make those claims. The teachers and practitioners who I'm the most familiar with and trust the most tend to reject those models, sometimes quite strongly (e.g. Daniel Ingram here). Also near the end of his life, Culadasa came to think that even though it might at one point have seemed like he had predominantly positive emotions in the way that some schools suggested, in reality he had just been repressing them with harmful consequences.

Culadasa: As a result of my practice, I had reached a point where emotions would arise but they really had no power over me, but I could choose to allow those emotions to express themselves if they served a purpose. Well, it’s sort of a downweighting of emotions – negative emotions were strongly downweighted, and positive emotions were not downweighted at all. So this was the place I was coming from as a meditation teacher. I just never really experienced anger; when something would cause some anger to arise, I’d notice it and let go of it, and, you know, it wasn’t there. Negative emotions in general were just not part of my life anymore. So it was a process of getting in touch with a  lot of these emotions that, you know, I hadn’t been making space for because I saw them as unhealthy, unhelpful, so on and so forth.

Michael: So, in essence, you had bypassed them.

Culadasa: Yes, it’s a bypassing. I think it’s a very common bypassing, too, when somebody reaches this particular stage on the path. I mean, this is a big of a digression, but I think it maybe helps to put the whole thing into perspective, the rest of our conversation into perspective…

Michael: Please digress.

Culadasa: Okay. So this is a stage at which the sense of being a separate self completely disappears. I mean, prior to that, at stream entry, you know, there’s no more attachment to the ego, the ego becomes transparent, but you still have this feeling that I’m a separate self; it still produces craving; you have to work through that in the next path, and so on and so forth. But this is a stage where that very primitive, that very primal sense of being a separate self falls away. Now, what I know about this from a neuroscience point of view is that there’s a part of the brainstem which was the earliest concentration of neurons that was brain-like in the evolution of brains, and there are nuclei there that were responsible for maintaining homeostasis of the body, and they still do that today. One of their major purposes is to regulate homeostasis in the body, blood pressure, heart rate, oxygenation of the blood, you name it, just every aspect of internal bodily maintenance. With the subsequent development of the emotional brain, the structures that are referred to as the limbic system, evolution provided a way to guide animals’ behaviors on the basis of emotions and so these same nuclei then created ascending fibers into this limbic system, from the brainstem into these new neural structures that constituted the emotional brain.

Michael: So this very old structure that regulated the body linked up with the new emotional structures.

Culadasa: Right. It linked up with it, and the result was a sense of self. Okay? You can see the enormous value of this to an animal, to an organism. A sense of self. My goodness. So now these emotions can operate in a way that serves to improve the survival, reproduction, everything else of this self, right? Great evolutionary advance. So now we have organisms with a sense of self. Then the further evolution of cerebral cortex, all of these other higher structures, then that same sense of self became integrated into that as well. So there we have the typical human being with this very strong, very primal sense that “I am me. I am a separate self.” We can create all kinds of mental constructs around this, but even cats and dogs and deer and mice and lizards and things like that have this sense of self. We elaborate an ego on top of it. So there’s these two aspects to self in a human being. One is the ego self, the mental construct that’s been built around this more primal sense of self. So this is a stage at which that primal sense of self disappears and what usually seems to happen is, at the same time, there is a temporary disappearance of all emotions. I think that we’ll probably eventually find out that the neural mechanism by which we bring about this shift, that these two things are linked, because the sense of self is – its passageway to the higher brain centers, which constitute the field of conscious awareness that we live in and all of the unconscious drives that we’re responding to, the limbic system, the emotional brain, is the link.

Michael: Yes.

Culadasa: So something happens that interrupts that link. The emotions come back online, but they come back online in a different way from that point. So instead of being overcome by fear, anger, lust, joy, whatever, these things arise and they’re something that you can either let go of or not. [laughs] That’s the place where I was.

Michael: They seem very ephemeral…

Culadasa: Yes, right. They’re very ephemeral, and very easy to deal with, and there is a tendency for other people to see you as less emotional and truly you are because you’ve downregulated a lot of more negative emotions. But you’re by no means nonemotional; you’re still human, you still have the full gamut of human emotions available to you. But you do get out of the habit of giving much leeway to certain kinds of emotions. And the work that I was doing with Doug pushed me in the direction of, “Let’s go ahead and let’s experience some of those emotions. Let’s see what it feels like to experience the dukkha of wanting things to be different than the way they are.” So that’s what we did. And I started getting in touch with these emotions and their relationship to my current life situation where I wasn’t fulfilling my greatest aspirations because I was doing a lot of things that – stuff that had to be done, but that I had no interest in, but I had to do it and that’s what occupied my time.

I'm guessing that something similar is what's actually happening for a lot of the schools claiming complete elimination of all negative feelings. Insight practices can be used in ways that end up bypassing or suppressing a lot of one's emotions, but actually negative feelings are still having effects in the person, they just go unnoticed.

If you think about it, you can't be sad and not mind it. You can't be angry but not mind it. 

This disagrees with my experience, and with the experience of several other people I know.

The biggest question on my mind right now is, what does your friend think of this post now that you've written it? 

Agree. This connects to why I think that the standard argument for evolutionary misalignment is wrong: it's meaningless to say that evolution has failed to align humans with inclusive fitness, because fitness is not any one constant thing. Rather, what evolution can do is to align humans with drives that in specific circumstances promote fitness. And if we look at how well the drives we've actually been given generalize, we find that they have largely continued to generalize quite well, implying that while there's likely to still be a left turn, it may very well be much milder than is commonly implied.

Ending a relationship/marriage doesn't necessarily imply that you no longer love someone (I haven't been married but I do still love several of my ex-partners), it just implies that the arrangement didn't work out for one reason or another.

I would guess that getting space colonies to the kind of a state where they could support significant human inhabitation would be a multi-decade project, even with superintelligence? Especially taking into account that they won't have much nature without significant terraforming efforts, and quite a few people would find any colony without any forests etc. to be intrinsically dystopian.

hmm, I don't understand something, but we are closer to the crux :)

Yeah I think there's some mutual incomprehension going on :)

1. To the question, "Would you update if this experiment is conducted and is successful?" you answer, "Well, it's already my default assumption that something like this would happen". 
2. To the question, "Is it possible at all?" You answer 70%. 

So, you answer 99-ish% to the first question and 70% to the second question, this seems incoherent.

For me "the default assumption" is anything with more than 50% probability. In this case, my default assumption has around 70% probability.

It seems to me that you don't bite the bullet for the first question if you expect this to happen. Saying, "Looks like I was right," seems to me like you are dodging the question.

Sorry, I don't understand this. What question am I dodging? If you mean the question of "would I update", what update do you have in mind? (Of course, if I previously gave an event 70% probability and then it comes true, I'll update from 70% to ~100% probability of that event happening. But it seems pretty trivial to say that if an event happens then I will update to believing that the event has happened, so I assume you mean some more interesting update.)

Hum, it seems there is something I don't understand; I don't think this violates the law.

I may have misinterpreted you; I took you to be saying "if you expect to see this happening, then you might as well immediately update to what you'd believe after you saw it happen". Which would have directly contradicted "Equivalently, the mere expectation of encountering evidence—before you’ve actually seen it—should not shift your prior beliefs".

I agree I only gave the skim of the proof, it seems to me that if you can build the pyramid, brick by brick, then this solved the meta-problem.

for example, when I give the example of meta-cognition-brick, I say that there is a paper that already implements this in an LLM (and I don't find this mysterious because I know how I would approximately implement a database that would behave like this).

Okay. But that seems more like an intuition than even a sketch of a proof to me. After all, part of the standard argument for the hard problem is that even if you explained all of the observable functions of consciousness, the hard problem would remain. So just the fact that we can build individual bricks of the pyramid isn't significant by itself - a non-eliminativist might be perfectly willing to grant that yes, we can build the entire pyramid, while also holding that merely building the pyramid won't tell us anything about the hard problem nor the meta-problem. What would you say to them to convince them otherwise?

1. Let's say we implement this simulation in 10 years and everything works the way I'm telling you now. Would you update?

Well, it's already my default assumption that something like this would happen, so the update would mostly just be something like "looks like I was right".

2. What is the probability that this simulation is possible at all? 

You mean one where AIs that were trained with no previous discussion of the concept of consciousness end up reinventing the hard problem on their own? 70% maybe.

If you expect to update in the future, just update now.  

That sounds like it would violate conservation of expected evidence:

... for every expectation of evidence, there is an equal and opposite expectation of counterevidence.

If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction. If you’re very confident in your theory, and therefore anticipate seeing an outcome that matches your hypothesis, this can only provide a very small increment to your belief (it is already close to 1); but the unexpected failure of your prediction would (and must) deal your confidence a huge blow. On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence—before you’ve actually seen it—should not shift your prior beliefs.

To me, this thought experiment solves the meta-problem and so dissolves the hard problem.

I don't see how it does? It just suggests that a possible approach by which the meta-problem could be solved in the future.

Suppose you told me that you had figured out how to create cheap and scalable source of fusion power. I'd say oh wow great! What's your answer? And you said that, well, you have this idea for a research program that might, in ten years, produce an explanation of how to create cheap and scalable fusion power.

I would then be disappointed because I thought you had an explanation that would let me build fusion power right now. Instead, you're just proposing another research program that hopes to one day achieve fusion power. I would say that you don't actually have it figured it out yet, you just think you have a promising lead.

Likewise, if you tell me that you have a solution to the meta-problem, then I would expect an explanation that lets me understand the solution to the meta-problem today. Not one that lets me do it ten years in the future, when we investigate the logs of the AIs to see what exactly it was that made them think the hard problem was a thing.

I also feel like this scenario is presupposing the conclusion - you feel that the right solution is an eliminativist one, so you say that once we examine the logs of the AIs, we will find out what exactly made them believe in the hard problem in a way that solves the problem. But a non-eliminativist might just as well claim that once we examine the logs of the AIs, we will eventually be forced to conclude that we can't find an answer there, and that the hard problem still remains mysterious.

Now personally I do lean toward thinking that examining the logs will probably give us an answer, but that's just my/your intuition against the non-eliminativist's intuition. Just having a strong intuition that a particular experiment will prove us right isn't the same as actually having the solution.

I quite liked the way that this post presented your intellectual history on the topic, it was interesting to read to see where you're coming from.

That said, I didn't quite understand your conclusion. Starting from Chap. 7, you seem to be saying something like, "everyone has a different definition for what consciousness is; if we stop treating consciousness as being a single thing and look at each individual definition that people have, then we can look at different systems and figure out whether those systems have those properties or not".

This makes sense, but - as I think you yourself said earlier in the post - the hard problem isn't about explaining every single definition of consciousness that people might have? Rather it's about explaining one specific question, namely:

The explanatory gap in the philosophy of mind, represented by the cross above, is the difficulty that physicalist theories seem to have in explaining how physical properties can give rise to a feeling, such as the perception of color or pain.

You cite Critch's list of definitions people have for consciousness, but none of the three examples that you quoted seem to be talking about this property, so I don't see how they're related or why you're bringing them up.

With regard to this part:

If they do reinvent the hard problem, it would be a big sign that the AIs in the simulation are “conscious” (in the reconstructed sense).

I assert that this experiment would solve the hard problem, because we could look at the logs,[4] and the entire causal history of the AI that utters the words "Hard pro-ble-m of Con-scious-ness" would be understandable. Everything would just be plainly understandable mechanistically, and David Chalmer would need to surrender.

This part seems to be quite a bit weaker than what I read you to be saying earlier. I interpreted most of the post to be saying "I have figured out the solution to the problem and will explain it to you". But this bit seems to be weakening it to "in the future, we will be able to create AIs that seem phenomenally conscious and solve the hard problem by looking at how they became that". Saying that we'll figure out an answer in the future when we have better data isn't actually giving an answer now.

In the current era, the economics are such that war and violence tend to pay relatively badly, because countries get rich by having a well-developed infrastructure and war tends to destroy that, so conquest will get you something that won't be of much value. This is argued to be one of the reasons for why we have less war today, compared to the past where land was the scarce resource and military conquest made more sense. 

However, if we were to shift to a situation where matter could be converted into computronium... then there are two ways that things could go. One possibility is that it would be an extension of current trends, as computronium is a type of infrastructure and going to war would risk destroying it. 

But the other possibility is that if you are good enough at rebuilding something that has been destroyed, then this is going back to the old trend where land/raw matter was a valuable resource - taking over more territory allows you to convert it into computronium (or recycle and rebuild the ruins of the computronium you took over). Also, an important part of "infrastructure" is educated people who are willing and capable of running it - war isn't bad just because it destroys physical facilities, it's also bad because it kills some of the experts who could run those facilities for you. This cost is reduced if you can just take your best workers and copy as many of them as you want to. All of that could shift us back to a situation where the return on investment for violence and conquest becomes higher than for peaceful trade.

As Azar Gat notes in War in Human Civilization (2006), for most of human history, war ‘paid,’ at least for the elites who made decisions. In pre-industrial societies, returns to capital investment were very low. They could – and did – build roads and infrastructure, irrigation systems and the like, but the production multiplier for such investments was fairly low. For antiquity, the Roman Empire probably represents close to the best that could be achieved with such capital investments and one estimate, by Richard Saller, puts the total gains per capita at perhaps 25% over three centuries (a very rough estimate, but focus on the implied scale here; the real number could be 15% or 30%, but it absolutely isn’t 1000% or 100% or even probably 50%).

But returns to violent land acquisition were very, very high. In those same three centuries, the Romans probably increased the productive capacity of their empire by conquest 1,200% (note that’s a comma, not a dot!), going from an Italian empire of perhaps 5,000,000 to a Mediterranean empire in excess of 60,000,000 (and because productivity per capita was so relatively insensitive to infrastructure investments, we can on some level extrapolate production straight out of population here in a way that we couldn’t discussing the modern world). Consequently, the ‘returns to warfare’ – if you won – were much higher than returns to peace. The largest and most prosperous states tended to become the largest and most prosperous states through lots of warfare and they tended to stay that way through even more of it.

This naturally produced a lot of very powerful incentives towards militarism in societies. Indeed, Gat argues (and I agree) that the state itself appears to have emerged as a stage in this competitive-militarism contest where the societies which were best at militarizing itself and coordinating those resources survived and aggregated new resources to themselves in conflict; everyone else could imitate or die (technically ‘or suffer state-extinction’ with most of the actual people being subjugated to the new states and later empires). [...]

And this makes a lot of sense if you think about the really basic energy economy of these societies: nearly all of the energy they are using comes from the land, either in the form of crops grown to feed either humans or animals who then do work with that energy. Of course small amounts of wind and water power were used, but only small amounts.

As Gat notes, the industrial revolution changed this, breaking the agricultural energy economy. Suddenly it was possible, with steam power and machines, to use other kinds of energy (initially, burning coal) to do work (more than just heating things) – for the first time, societies could radically increase the amount of energy they could dispose of without expanding. Consequently – as we’ve seen – returns to infrastructure and other capital development suddenly became much higher. At the same time, these new industrial technologies made warfare much more destructive precisely because the societies doing the warfare now had at their disposal far larger amounts of energy. Industrial processes not only made explosives possible, they also enabled such explosives to be produced in tremendous quantities, creating massive, hyper-destructive armies. Those armies were so destructive, they tended to destroy the sort of now-very-valuable mechanical infrastructure of these new industrial economies; they made the land they acquired less valuable by acquiring it. So even as what we might term ‘returns to capital’ were going wildly up, the costs of war were also increasing, which mean that ‘returns to warfare’ were going down for the first time in history.

It’s not clear exactly where the two lines cross, but it seems abundantly clear that for the most developed economies, this happened sometime before 1914 because it is almost impossible to argue that anything that could have possibly been won in the First World War could have ever – even on the cynical terms of the competitive militarism of the pre-industrial world – been worth the expenditure in blood and treasure.

And as far as we can tell, there don't appear to be any sharp discontinuities here, such that above a certain skill level it's beneficial to take things by force rather than through negotiation and trade. It's plausible that very smart power-seeking AIs would just become extremely rich, rather than trying to kill everyone.

I think this would depend quite a bit on the agent's utility function. Humans tend more toward satisficing than optimizing, especially as they grow older - someone who has established a nice business empire and feels like they're getting all their wealth-related needs met likely doesn't want to rock the boat and risk losing everything for what they perceive as limited gain. 

As a result, even if discontinuities do exist (and it seems pretty clear to me that being able to permanently rid yourself of all your competitors should be a discontinuity), the kinds of humans who could potentially make use of them are unlikely to.

In contrast, an agent that was an optimizer and had an unbounded utility function might be ready to gamble all of its gains for just a 0.1% chance of success if the reward was big enough. 

Cool that you published this! Could you post some example dialogues with the bot that you think went particularly well?

Hmm, I think people have occasionally asked me "how's your week going" on dating apps and I've liked it overall - I'm pretty sure I'd prefer it over your suggested alternative! No doubt to a large extent because I suck at cooking and wouldn't know what to say. Whereas a more open-ended question feels better: I can just ramble a bunch of things that happen to be on my mind and then go "how about yourself?" and then it's enough for either of our rambles to contain just one thing that the other party might find interesting.

It feels like your proposed question is a high-variance startegy: if you happen to find a question that the other person finds easy and interesting to answer, then the conversation can go really well. But if they don't like the direction you're offering, then it'd have been better to say something that would have given them more control over the direction.

That sounds fun, feel free to message me with an invite. :)

stream-entry might be relatively easy and helpful

Worth noting that stream entry isn't necessarily a net positive either:

However, if you’ve ever seen me answer the question “What is stream entry like,” you know that my answer is always “Stream entry is like the American invasion of Iraq.” It’s taking a dictatorship that is pretty clearly bad and overthrowing it (where the “ego,” a word necessarily left undefined, serves as dictator). While in theory this would cause, over time, a better government to form, it will assuredly leave a period without any government, when the day-to-day functions of government are simply not carried out. The path is supposed to be about, as the Buddha says, “Suffering and the end of suffering,” but as far as I’ve seen, the correlation between stream entry and suffering is about 0; suffering is as likely to get better as it is to get worse. Whether it’s better to have a pre-awakening dictatorship or a post-awakening anarchy is basically a toss-up. Upali and I like to describe stream entry as “a big flaming turd of false advertisement,” as we both experienced quite extreme suffering subsequent to stream entry.

However, I do not think that the purpose of lesswrong is served by engaging with religious ideology here,

I didn't say we should engage with it! I was still speaking within the context of barbs at religion at the Solstice. I agree we should continue to reject (epistemically unsound versions of) religion, just not also be needlessly hostile to it in contexts where it could be avoided with some small tweaks and without compromising on any principles.

Why does this suggest to you that the community needs to be less hostile to religion, instead of more or roughly the same amount?

Usually if a group signals hostility to X and some X-people are thick-skinned enough to participate anyway, there'll be a much greater number of X-people who are less thick-skinned and decide to stay out. Even if the X-people could make good contributions, as they empirically can.

And if their contributions are bad, they'll just be downvoted on their own (lack of) merits, the same as any other bad post.

I have enjoyed reading your stuff over the years, from all the spirituality-positive people I find your approach especially lucid and reasonable, up there with David Chapman.

Thank you! That's high praise. :)

But, since I seem to be able to make sense of them without having to meditate myself, it always left me bemused as to whether meditation really is the "royal road" to these kinds of insight, and if whatever extra it might offer is worth the effort. 

Heh, I remember that at one point, a major point of criticism about people talking about meditation on LW was that they were saying something like "you can't understand the benefits of meditation without actually meditating so I'm not going to try, it's too ineffable". Now that I've tried explained things, people wonder what the point of meditating might be if they can understand the explanation without meditating themselves. :) (I'm not annoyed or anything, just amused. And I realize that you're not one of the people who was making this criticism before.)

Anyway, I'd say it's one thing to understand an explanation of the general mechanism of how insights are gotten, and another to actually experience the insights from the inside in a way that shifts your unconscious predictions.

That being said, is it worth the effort for you? I don't know, we kinda concluded in our dialogue that it might not be for everyone. And there are risks too. Maybe give some of it a try if you haven't already, see if you feel motivated to continue doing it for the immediate benefits, and then just stick to reading about it out of curiosity if not?

So, I guess, my real question for the therapy and spirituality-positive people is why they think that their evidence for believing what they believe is stronger than that of other people in that field who have different models/practices/approaches but about the same amount of evidence for its effectiveness. 

Good question. One thing that I'm particularly confused about is why me and Scott Alexander seem to have such differing views on the effectiveness of the "weird therapies". My current position is just something like... "people seem to inhabit genuinely different worlds for reasons that are somewhat mysterious, so they will just have different experiences and priors leading to different beliefs, and often you just have to go with your own beliefs even if other smart people disagree because just substituting the beliefs of others for your own doesn't seem like a good either". And then hopefully if we continue discussing our reasons for our beliefs for long enough, at some point someone will figure out something.

My current epistemic position is also like... it would be interesting to understand the reason, and I don't have a great model of where the disagreement comes from. And I'm happy to discuss it with people who might disagree. But it also doesn't feel like a huge priority, given that I do feel convinced enough that these things work on a level that's sufficient for me.

I have no reason to doubt that plenty of therapists/coaches etc. have good evidence that something that they do works, but believing that they have a good and complete explanation of what exactly works or why is orders of magnitude harder, and I don't think that anybody in the world could reasonably claim to have the complete picture, or anything close to it.

Yes I definitely agree with this.

Usually, a valuable community should only welcome members insofar as it can still maintain its identity and reason for existing. Some communities, such as elite universities, should and do have strict barriers for entry (though the specifics are not always ideal). The culture of lesswrong would probably be erased (that is, retreat to other venues) if lesswrong were mainstreamed and successfully invaded by the rest of the internet. 

Yes, but none of this require overt hostility to religion (as opposed to just rejection). I think that as long as religious people accept the conversational norms and culture on LW, them bringing in some new perspectives (that are still compatible with overall LW norms) ought to be welcome. 

Many traits tend to be correlated for reasons of personality rather than strict logic. So if you select for people on atheism then you may also select for certain ways of thinking, and there can be ways of thinking that are just as rational, but underrepresented among atheists. Selecting against those ways of thinking can make the intellectual community more impoverished.

Take the author of this post. He has openly said that he's religious; he has also written four posts with 70+ karma, including one that as of this writing has 259 karma and a Curated status, so LW seems to consider him a positive influence. (Not all of his posts have gotten a lot of karma, but then so neither have all of mine.) I don't think it would have been a good thing if LW's hostility to religion had driven him away so that he would never have participated.

The first thing to note is that that very page says that the state of evidence on the verdict is mixed, with different studies pointing in different directions, results depending on how you conduct your meta-analyses, and generally significant disagreement about whether this is really a thing.

I also think that this comment from @DaystarEld , our resident rationalist therapist, makes a lot of sense:

For one thing, the Dodo bird verdict is (maybe not surprisingly, given point 3) not as well supported as people widely think. It originated decades ago, and may have set in motion the very effects that led to its own eventual lack of relevance. The study I linked to in the OP, if correct, points to just such an invalidation by presenting findings that a particular modality works better for a certain type of treatment than alternatives.

But if we take it at face value, the answer could just come down to "the human element." Maybe good therapists are what matter and the modality, as long as it's not utterly bankrupt, is just a vehicle. Personally I don't believe that's the full story, but a good relationship with the therapist does seem more important than anything else, and that factor being mostly independent from what modality the therapist uses may account for a large part of it.

Ultimately though, I think part of what my post is tries to do is point out that these different philosophies don't necessarily contradict each other, but rather are different lenses through which to view the problems the client has. When I get a client that responds super well to CBT, and then another client who doesn't but grabs IFS and runs with it, I don't think "well I guess these modalities are equally effective" or think that some kind of paradox is occurring, I just think that different maps are better for different people at navigating the territory, even if they're dealing with the same "problem."

I know it feels a bit like a cop-out, but honestly given how complex people are, and how different each problem can be even if it shares the same diagnosis, I would be pretty shocked if a single modality just blew all the others out of the water for every kind of problem that someone might face. Which isn't to say that they're all the same, either, just that guidelines for good therapy have to include more than just singling out specific modalities, but also identifying which ones might work best with each client.

This touches upon a related issue, which is that there are some serious challenges with trying to apply an RCT-type methodology on something like this. For an RCT, you'll want to try to standardize things as much as possible, so that - for example - if you are measuring the effectiveness of Cognitive Behavioral Therapy, then every therapist you've classified as "doing CBT" actually does do CBT and nothing else. But a good therapist won't just blindly apply one method, they'll consider what they think might work best for this particular client and then use that.

Suppose you have a study where therapists A and B are both familiar with both CBT and Internal Family Systems. Therapist A is assigned to the CBT condition and therapist B is assigned to the IFS condition. As a result, A spends some time doing CBT on clients CBT is a poor match for, and B spends some time doing IFS on clients IFS is a poor match for. The study finds that CBT and IFS have roughly similar, moderate efficacy. What the study fails to pick up on is that if both A and B had been allowed to pick the method that works best on each client, doing IFS for some and CBT for some, then the effect of the method might have been significantly greater.

But you can't really compare the efficiency of methods by doing an RCT where everyone is allowed to just do whatever method they like, or worse, some hybrid method that pulls in from many different therapy techniques. Or maybe you could do it and just ask the therapists to write down what method they used with each client afterward... but that would probably require some really complicated statistical method to try to analyze, exactly the kind of thing where everyone will just end up fighting over the right way to interpret the results afterward.

Another thing is that changes in subjective well-being are often just really hard to measure well. For one, various measures you'd naively expect to correlate with each other, don't really:

The study asked people in 156 countries to “value their lives today on a 0 to 10 scale, with the worst possible life as a 0 and the best possible life as a 10.” [...] societal factors such as gross domestic product per capita; extensiveness of social services; freedom from oppression; and trust in government and fellow citizens can explain a significant proportion of people’s average life satisfaction [...] the Nordic countries—Finland, Sweden, Norway, Denmark, Iceland—tend to score highest in the world.

But when you look at how much positive emotion people experience, the top of the world looks very different. Suddenly, Latin American countries such as Paraguay, Guatemala and Costa Rica are the happiest countries on earth. [...]

Things get even more complicated when we look at the prevalence of depression in different countries. In one comparison made by the World Health Organization, the per capita prevalence of unipolar depressive disorders is highest in the world in the United States. Among Western countries, Finland is number two. Paradoxically then, the same country can be high on both life satisfaction and depression. [...]

Finally, some people might argue that neither life satisfaction, positive emotions nor absence of depression are enough for happiness. Instead, something more is required: One has to experience one’s life as meaningful. [...] [On a measure of meaningfulness] African countries including Togo and Senegal were at the top of the ranking, while the U.S. and Finland were far behind. Here, religiosity might play a role: The wealthier countries tend to be less religious on average, and this might be the reason why people in these countries report less meaningfulness.

What I’m trying to say is that, as regards happiness, it’s complicated. Different people define happiness very differently. And the same person or country can be high on one dimension of happiness while being low on another dimension of happiness. Maybe there is no such thing as happiness as such. Instead we should look at these dimensions separately and examine how well various nations are able to support each of them.

I think this finding actually makes a lot of sense. At one point in my life, if you had asked me to measure my well-being on a scale from 0 to 10, I might have said something like, "well I'm feeling pretty depressed and don't find my work very satisfying and my romantic relationships aren't working out, but then on the other hand I do have a secure job, it pays reasonably well, I live in a safe country, and overall I feel like there are a lot of things that are objectively just pretty good, so let's say a 7".

Setting aside the fact that someone might simultaneously report being depressed while also giving a relatively high life-satisfaction number, here are some ways in which I feel like I've benefited that feel like they're at least somewhat linked to these practices:

• It used to be that I would randomly remember various faux pases or mistakes I'd made that felt embarrassing or shameful, or generally experience painful feelings of regret for past life decisions. These could feel pretty intensely unpleasant. While something like this might still occasionally happen for a bit if I make a new mistake, or I might get some very mild twinge of unpleasantness related to some past thing, to a first approximation it's correct to say that this has stopped happening. What's in the past is in the past, and I don't really experience much regret or "damn I should(n't) have said X"  anymore.
• When I'm on my phone, it's been getting more common recently that I maintain awareness of my peripheral vision and the rest of room, as well as still continuing to feel my body, rather than just getting completely sucked into the phone and forgetting about the world around me. This makes it somewhat easier to stop being on my phone and done something else.
• I feel like pain and displeasure are less strongly linked to suffering than they used to be. It's easier to experience something as painful and unpleasant while simultaneously being okay with having that experience. I was at men's retreat recently where the facilitator asked if we'd want to try some amateur boxing (with the appropriate safety gear) after he'd drill us on some basic technique. I've had a pretty strong fear of physical pain as well as feeling generally bad at physical things, so this offer felt pretty frightening to me, but I figured that objectively thinking it'd probably be fine and that the fear is just a sensation, so I can be with that sensation. Then I kept being in a state of mild panic for the entirety of the drills and then in the actual match I was hyperventilating, but also that felt okay and a positive experience overall, rather than traumatizing and like the end of the world as it would have felt some years earlier.

Now if you asked me to report on my life satisfaction now... I do feel like those things represent some significant improvements to my life, but a scale of 0 to 10 is pretty coarse, and I still haven't gotten romantic relationships working out. So maybe I'd still say 7? Possibly 8, but 7 doesn't feel unreasonable either.

It's also interesting that out of the various ways of measuring well-being (life satisfaction, positive emotion experienced, amount of depression, sense of meaning), "pain being linked with less suffering" wouldn't necessarily show up on any of those. Someone might still have feel essentially the same emotionally, but just e.g. feel more okay with being depressed and sad often.

There's also other pieces of weirdness about life satisfaction measures in general - for example, people might give a slightly negative score on the overall life satisfaction, while giving positive scores on all subareas of life satisfaction that are asked about.

The Wikipedia article on the verdict was saying that it's controversial since there are debates about exactly what kinds of measures to include in the meta-analyses, with people on the "pro-verdict side" accusing people on the "anti-verdict side" of sometimes cherry-picking measures and results that do show a positive change. But it does seem to me possible to drastically improve a person's well-being on one measure without it showing up on any others, in ways that you might not be able to predict before running the study. And if someone then appeals to that one measure having improved, they might be cherry-picking or they might just be legitimately drawing attention to the one thing that does manage to measure a real change.

There's also the fact that when I offer emotion coaching for people, the results will vary and some people probably get basically no benefit, others get a moderate benefit, and then some others benefit massively. Here's a testimonial from one client I had:

I attended a few IFS sessions with Kaj towards the end of last year.

I don't say this lightly, but the sessions with Kaj had a transformative impact on my life. Before these sessions, I was grappling with significant work and personal-related challenges. It's hard to capture how bad it had become - my life felt like it had ground to a halt. My ability to function was very limited and this had lasted for over two years. Despite trying various methods, and seeing various professionals, I hadn't seen much improvement in this time.

However, after just a few sessions (<5) with Kaj, I overcame substantial internal barriers. This not only enabled me to be more productive again on the work I cared about but also to be kinder to myself. My subjective experience was not one of constant cycling in mental pain. I could finally apply many of the lessons I had previously learned from therapists but had been unable to implement.

I remember being surprised at how real the transformation felt. I can say now, almost a year later, that it was also not transient, but has lasted this whole time. 

As a result, I successfully completed some major professional milestones. On the personal front, my life has also seen positive changes that bring me immense joy.

I owe this success to the support from Kaj and IFS. I had been sceptical of 'discrete step' changes after so many years of pain with little progress, but I can now say I am convinced it is possible to have significant and enduring large shifts in how you approach yourself, your life and your pursuits.

Now, this person is definitely a bit of an outlier, but only somewhat. I get a client who gets utterly transformed in less than five sessions maybe... 1-3 times a year? Whereas some smaller but still significant change in a session or two, like a client saying that the first session with me feels more useful than all of the therapy they previously had combined, feels basically just normal and expected at this point. And anecdotally this - going from very little previous experience to hearing "wow you're better than all the 'real' therapists I saw previously" - seems like a common experience for people who start doing offering kind of coaching or therapy that effectively focuses on memory reconsolidation.

Maybe it's just personal fit, and my personality happens to match super-well with some people and then they get enormous benefits, and the problem was that none of the ordinary therapists they tried were an equally good match? And this is also the case with everyone else I keep hearing stories from? I guess that could be the case, but...

I realize that this is still very anecdotal. But my perspective is something like: my experience is that either I have personally benefited massively from these things or I am utterly delusional to the point of not being able to trust anything about my own experience, many of my friends who have done something similar give me plausible accounts of how they have benefited massively from these things, many of my clients give me plausible accounts of how they have benefited massively from these things... and main point against is that some controversial studies have difficulty establishing that the methods are effective, at least depending on who you ask and how you analyze them, and this seems like the kind of thing that is just really hard to analyze with RCTs in general.

And also, things being hard to analyze with RCTs is a common thing! Famously, nobody did an RCT measuring the effectiveness of parachutes against a control group without them, before parachutes became broadly used. If someone tells me that their name is Mark, I tend to believe that this is their name, even though it'd be really hard for me to run an RCT establishing this. Of all the things that I know and act based on, only a very small portion of it is something that can be established with RCTs.

Now, of course, just the fact that I'm convinced doesn't mean that you would have a reason to be convinced. It's totally valid to go "Okay but I still think you're just going by anecdotal evidence and possibly deluding yourself". And maybe I am! But again, that doesn't seem like the most likely hypothesis to me.

---

Talking about Buddhism and meditation more specifically, there are various findings showing results, e.g. this summary from the book Altered Traits:

... at the start of contemplative practice, little or nothing seems to change in us. After continued practice, we notice some changes in our way of being, but they come and go. Finally, as practice stabilizes, the changes are constant and enduring, with no fluctuation. They are altered traits.

Taken as a whole, the data on meditation track a rough vector of progressive transformations, from beginners through the long-term meditators and on to the yogis. This arc of improvement seems to reflect both lifetime hours of practice as well as time on retreat with expert guidance.

The studies of beginners typically look at the impacts from under 100 total hours of practice—and as few as 7. The long-term group, mainly vipassana meditators, had a mean of 9,000 lifetime hours (the range ran from 1,000 to 10,000 hours and more).

And the yogis studied in Richie’s lab, had all done at least one Tibetan-style three-year retreat, with lifetime hours up to Mingyur’s 62,000. Yogis, on average had three times more lifetime hours than did long-term meditators—9,000 hours versus 27,000.

A few long-term vipassana meditators had accumulated more than 20,000 lifetime hours and one or two up to 30,000, though none had done a three-year retreat, which became a de facto distinguishing feature of the yogi group. Despite the rare overlaps in lifetime hours, the vast majority of the three groups fall into these rough categories.

There are no hard-and-fast lifetime hour cutoffs for the three levels, but research on them has clustered in particular ranges. We’ve organized meditation’s benefits into three dose-response levels, roughly mapping on the novice to amateur to professional rankings found in expertise of all kinds, from ballerinas to chess champions. [...]

Sticking with meditation over the years offers more benefits as meditators reach the long-term range of lifetime hours, around 1,000 to 10,000 hours. This might mean a daily meditation session, and perhaps annual retreats with further instruction lasting a week or so—all sustained over many years. The earlier effects deepen, while others emerge.

For example, in this range we see the emergence of neural and hormonal indicators of lessened stress reactivity. In addition, functional connectivity in the brain in a circuit important for emotion regulation is strengthened, and cortisol, a key hormone secreted by the adrenal gland in response to stress, lessens.

Loving-kindness and compassion practice over the long term enhance neural resonance with another person’s suffering, along with concern and a greater likelihood of actually helping. Attention, too, strengthens in many aspects with long-term practice: selective attention sharpens, the attentional blink diminishes, sustained attention becomes easier, and an alert readiness to respond increases. And long-term practitioners show enhanced ability to down-regulate the mind-wandering and self-obsessed thoughts of the default mode, as well as weakening connectivity within those circuits—signifying less self-preoccupation. These improvements often show up during meditative states, and generally tend to become traits.

Shifts in very basic biological processes, such as a slower breath rate, occur only after several thousand hours of practice. Some of these impacts seem more strongly enhanced by intensive practice on retreat than by daily practice.

While evidence remains inconclusive, neuroplasticity from long-term practice seems to create both structural and functional brain changes, such as greater working connection between the amygdala and the regulatory circuits in the prefrontal areas. And the neural circuits of the nucleus accumbens associated with “wanting” or attachment appear to shrink in size with longer-term practice.

While in general we see a gradient of shifts with more lifetime meditation hours, we suspect there are different rates of change in disparate neural systems. For instance, the benefits of compassion come sooner than does stress mastery. We expect studies in the future will fill in the details of a dose-response dynamic for various brain circuits. Intriguing signs suggest that long-term meditators to some degree undergo state-by-trait effects that enhance the potency of their practice. Some elements of the meditative state, like gamma waves, may continue during sleep.

Now as I recall, the book does have a number of caveats about how many of the studies on meditation are low-quality, relatively small, et cetera. But again, there are again serious challenges to running an RCT - sure you can do a study where you give someone a limited meditation intervention, like asking them to do some meditation practice for a month. But if some of the benefits are going to start showing up around 1,000 hours of practice and some at 10,000, you can't just find a random assortment of people who have never meditated before and tell them to do thousands of hours of it or to go on month- or year-long retreats.

Furthermore, when we measure that competence, it usually falls somewhere within the human range of performance. 

I think that for this to be meaningfully true, the LLM should be able to actually replace humans at a given task. There are some very specific domains in which this is doable (e.g. creative writing assistant), but it seems to me that they are still mostly too unreliable for this. 

I've worked with getting GPT-4 to act as a coach for business customers. This is one of the domains that it excels at - tasks can be done entirely inside a chat, the focus is on asking users questions and paraphrasing them so hallucinations are usually not a major issue. And yet it's stupid in some very frustrating ways that a human wouldn't be. 

For example, our users would talk with the bot at specific times, which they would schedule using a separate system. Sometimes they would ask the bot to change their scheduled time. The bot wasn't interfaced to the actual scheduling system, but it had been told to act like a helpful coach, so by default it would say something like "of course, I have moved your session time to X". This was bad, since the user would think the session had been moved, but it hadn't.

Well, easy to fix, right? Just add "if the user asks you to reschedule the session or do anything else that requires doing something outside the actual conversation, politely tell them that you are unable to do that" to the prompt. 

This did fix the problem... but it created a new one. Now the bot would start telling the user "oh and please remember that I cannot reschedule your session" as a random aside, when the user had never said anything about rescheduling the session. 

Okay, so what about adding something like "(but only tell this if the user brings it up, don't say it spontaneously)" to our prompt? That reduced the frequency of the spontaneous asides a little... but not enough to eliminate it. Eventually we just removed the whole thing from the prompt and decided that the occasional user getting a misleading response from the bot is better than it randomly bringing this up all the time.

Another basic instruction that you would think would be easy to follow would be "only ask one question at a time". We had a bit in a prompt that went like "Ask exactly one question. Do not ask more than question. Stop writing your answer once it contains a question mark." The end result? GPT-4 happily sending multi-question messages like "What is bothering you today? What kinds of feelings does that bring up?".

There are ways to fix these issues, like having another LLM instance check the first instance's messages and rewrite any that are bad. But at that point, it's back to fragile hand-engineering to get the kinds of results one wants, because the underlying thing is firmly below a human level of competence. I don't think LLMs are (or at least GPT-4 is not) yet at the kind of level of high reliability involved in human-level performance.

I'm guessing Gordon is referring to a class of things in the category of "attempted telekinesis", where people have the implicit expectation that it is possible to change something by just willing it - at a sufficiently subtle or implicit level that it persists unnoticed even if the person would never endorse it explicitly. The curse of the counterfactual is another description of this kind of thing. And the kind of "faith" he's describing is (I'm guessing, from having some familiarity with a similar thing) a kind of mental move that cuts through this type of mistake, by remembering implicitly that which is also believed explicitly.

Big fan of both of your writings, this dialogue was a real treat for me.

Thanks! Glad you liked it.

But... I don't want to end up net negative on agency. In fact my primary objective is to end up strongly net positive.

I think that the likely impact on agency is complicated. One question is the extent to which your current agency is driven by something like pain avoidance. 

@Matt Goldenberg has a nice concept of a mode of motivation he calls "the self-loathing monster", where one effectively motivates themselves by stacking on more fear/pain of failure to overcome the fear/pain of doing something. A classic example would be procrastinating until just before the deadline, and then at the last moment getting an urgency to complete the thing and doing it at the last moment while find everything very uncomfortable. 

The more strongly one's motivation is built like this, the more likely it is that there will be a loss of agency after the sources of pain are removed, as one hasn't developed positive forms of motivation that could pick up the slack when the negative forms of motivation are removed. That's not to say that such a person would be doomed to a lifetime of non-agency! It's possible to learn positive motivation, but it's going to take time. Possibly several years.

On the other hand, Tucker Peck has a nice talk ("Meditation and Social Justice" on this page) about the way that many important things are really hard, and that if you need to see success right away, you may have little chance than to burn out. In that kind of a situation, a more enlightened-y mindset may be exactly what you need:

If you look at how many, I guess millions of people, risked their lives to create social change and they did it and then in a lot of countries, it just disappeared, you know, very quickly. It was back [to] the way it used to be. I read two of Gandhi's autobiographical books in this past winter. And you know, in South Africa, [...] he's there for maybe 18 years and he finally is able to Improve the standing of the Indians in South Africa and then he leaves and everything goes right back to the way that it used to be. 

It's not the change isn't possible, it's that change is awfully slow and things that look like victories can turn out to be nothing. [...] So if you have an attachment to a sense of self, feeling like a failure is going to be a lot harsher than if you don't. But if you have an attachment to the outcome, I don't see anything you could do besides burn out. [...]

When you are able to - not completely lose the sense of self necessarily, [but] at least diminish it, or have periods when it subsides - you can go to this place where all of your actions can be motivated by justice, by compassion, seeing yourself equally valuable to everybody else. And there's no sense of burnout because there's nothing else to do. Like the idea of giving up doesn't make sense. There's nothing to give up. 

When you read people like Martin Luther King, Nelson Mandela and Mahatma Gandhi [...], the people who really lived the, religion of justice - they seem indefatigable. And they don't seem to mind if they die from this. They sometimes don't even seem to mind if they lose.

But suffering is the same. We evolved the ability to suffer to help our survivability. It's not inherent to life in any sense, it's not even required. As far as I know, positive reinforcement achieves the same as negative reinforcement. Suffering is a motivator.

It sounds to me like you're talking about pain rather than suffering. In my experience, pain acts as a motivating factor even if it's not associated with suffering. Indeed, suffering indicates that pain is being resisted, so the full signal is not being properly heard.

You can also just accept that you think it's a problem because you choose to do so, or because your nature demands it of you. If you think something is wrong with reality, then you create a world which you cannot love.

I actually agree with this: it's as you say, that it's good to have signals for hunger, physical pain, etc.. Or even if it wasn't good, those signals are still describing an aspect of reality whether one thinks that's good or not.

Suffering, in my experience, is created when a part of the mind says that this is a sign of something being wrong with reality. It says: "it should not be so that I am hungry right now". Or it says: "I reject the world in which I am in pain". The conflict between a part of the mind that tries to reject the presence of the signal, and the part of the mind that is accurately perceiving reality and creating the signal, is what creates suffering. (I discussed a version of this in more detail here.)

Once you accept that the presence of the signal, once you accept that reality is the way it is, then suffering ceases and you are better able to do something about the signal. Part of the reason why I think suffering is bad, is that it involves a rejection of reality.

Meaning" exists as a concept in the first place because it's a part of us, to say that the concept is false because we can't find it outside of us is to forget that we created it in the first place. It would be silly to say that numbers don't exist because we can't find them outside of mathematics, right? Nothing exists inherently, that is, "universally", outside of itself. Why would it?

I would phrase this as: meaning and numbers do not exist ultimately, but they do exist conventionally.

Do you know of "spiritual" things outside of Buddhism by the way?

See the list in my first message in the dialogue. Of the items on that list, only meditation was strongly associated with Buddhism in particular (and there seems to be a lot of convergent evolution around meditation, e.g. some Christian meditation seems to be basically doing the same thing, just conceptualized differently).

I've never really managed to locate myself on the MCTB maps. Sometimes I'll have a brief period when it feels like some particular stage might fit my experience, but then that stops being the case.

Didn't have the chance to read your post yet, but this paper might be of interest if you haven't seen it.

I don't quite understand your comment. It sounds like you're saying that spirituality is too scientific and not subjective enough? That's not the kind of criticism I would have expected!

We think lust and greed are bad for the same reason. We think the ego is bad for the same reason.

I don't! Nor do I think that power is bad. I do think that suffering is bad, though. (Or at least it's bad in a conventional, if not ultimate sense.)

I'll warn you about deconstruction, it will result in nihilism. Spirituality should be construction

I think you need both. If you can deconstruct yourself to some extent, then you can also reconstruct yourself more flexibly. Like a structure of Lego that you first break down back into individual Legos, and then put together in a new way.

To what extent do you think ~unenlightenment in an individual is caused by the need to fit in socially?

I'd guess to a very significant extent, though I think there are also actual developmental stages that are distinct from social constraints. E.g. one axis of development in "enlightenment" involves taking more and more things as object, coming to experience them as mental constructs rather than as intrinsic aspects of reality. I think this involves the development of something like additional neuronal circuitry that provides increasingly meta levels of awareness into your mind, separate from any social considerations. (Social considerations might very well act as blockers for developing some of that awareness, however.)

Q2: Do you think people benefit from being ~unenlightened or spiritually unskilled? Precisely how so?

Fitting in socially is quite important! We wouldn't have evolved to do that if it wasn't useful, and as I mentioned in the dialogue, some Buddhist lineages that didn't were likely wiped out because they started making too much trouble.

Also depends on how you define "benefit", but if spiritual development makes you e.g. care less about money and status, then you'll probably end up having less money and status. Even if that makes you happier, it might make you worse off in terms of external conditions, and more likely to be hurt by people who do have that money and status. 

Someone like Stalin seems to have been quite spiritually unskillful, but his paranoia and desire for power got him to the position of being a dictator and killing off quite a few other people. On some measure of "better off", it might have been better for some of those others to also be equally unenlightened and more power-hungry, so one of them would have become the ruler instead and survived.

meditation can mess you up

Oh, definitely. I'm... actually not sure how come we didn't mention this, given that I've included links to the risks of meditation in my posts before. Here's a caveat and some references from an earlier post of mine:

Getting deep in meditation requires a huge investment of time and effort - though smaller investments are also likely to produce benefits - and is associated with its own risks [1 2 3 4].

I'd note that while this has a nice list of technologies that were successfully stopped, it's missing examples for which this failed. Some examples:

• The prohibition of alcohol failed
• Various illegal drugs continue to be sold and used in enormous amounts despite massive efforts to stop this
• Online piracy of music, movies, games, etc. seems to have declined from its peak due to better online marketplaces, but illegal file-sharing is still a thing
• China failed to keep silk production just to itself

I think a common factor in these is that they are much easier to produce/do undetected and with limited resources than e.g. nuclear power.

Time Wrote the Rocks

This is supposed to be anti-Biblical, in particular, anti-Young Earth Creationist. But Old Earth Creationism / Divinely Guided Evolution is also very well established in Christian tradition, and is fairly popular. [...]

I don't think it would be too hard to remove/de-emphasize the (partially imagined) conflict with religion and turn this into a song I really like.

I'm curious what you think of the original version of the song, which I interpret as something like anti-Young Earth but pro-Divinely Guided Evolution.

This bit

The community has a carefully cultivated culture that makes it a kind of sanctuary from the rest of the world where rationalists can exchange ideas without reestablishing foundational concepts and repeating familiar arguments (along with many other advantages).

and this bit

The examples you point to do not demonstrate hostility towards religious people, they demonstrate hostility towards religion. This is as appropriate here as hostility towards factory farming is at a vegan group. [...] Lesswrong hates religion in the way that lipids hate water.

are I think conflating rejection of religion and hostility to religion. There are plenty of contexts where it's expected for people to follow one set of norms rather than another: if I want to publish in a physics journal, it's expected that I follow the established research conventions and concepts of physics rather than, say, philosophy. And vice versa. Each field rejects the norms of the other - but this can be done without papers published in either field having random barbs against the other field.

One can reject religion without hostility to it, just as physics rejects the norms of philosophy without being hostile to it. 

I'm also skeptical of the claim that the examples demonstrate hostility toward religion rather than religious people - tribal instincts are tribal instincts, and do not make such fine-grained distinctions. The "Brighter Than Today" bit is a pretty clear example: the people who want to silence the Prometheus are religious. As the OP points out, there's no strong reason to expect prehistoric religions to act that way. (E.g. what we know of ancient religions is that they were mostly practical with doctrine derived from what worked, instead of practice being derived from doctrine.) So what's that line doing in the song? Well, there's a pretty obvious line of reasoning that would have created it: a thought along the lines of "well religious people are nasty and conservative, so probably they would also have opposed the invention of fire...". 

Even if that wasn't the original generator, it's still implicitly spreading that kind of message. I think that's generally bad, because tribal instincts worsen people's reasoning, so anything that feeds into them (including subtle barbs at outgroups) would generally be better off avoided.

I also somewhat disagree with this bit:

Lesswrong hates religion in the way that lipids hate water.

I'd say this is roughly accurate for religions that require belief in things without evidence. There's a range of religions that don't require that, I think including some mysticism/practice-based (rather than belief-based) versions of Christianity that make no strong claims about what God is, and allow for God to be interpreted in an abstract/metaphorical way while being based on evidence of the form "if you do these practices, you are likely to experience these kinds of effects".

Someone pointed out that this only seems to work if the screenshots include the "ChatGPT" speaker tag; if you only screenshot the text of ChatGPT's most recent response without the label indicating it is from ChatGPT, it seems to fail. Oddly, in one of my tests, it seemed to recognize its own text on the first time I sent it a screenshot, but then didn't manage to figure out what to do next (nor did it mention this insight in the later replies).

So maybe this is more about it recognizing its own name than itself in a mirror?

Come to think of it, how is it that humans pass the mirror test? There's probably a lot of existing theorizing on this, but a quick guess without having read any of it: babies first spend a long time learning to control their body, and then learn an implicit rule like "if I can control it by an act of will, it is me", getting a lot of training data that reinforces that rule. Then they see themselves in a mirror and notice that they can control their reflection through an act of will...

This is an incomplete answer since it doesn't explain how they learn to understand that the entity in the mirror is not a part of their actual body, but it does somewhat suggest that maybe humans just interpolate their self-awareness from a bunch of training data too.

I replicated the result; ChatGPT got the right answer after the fourth screenshot.

I'm unlikely to reply to further object-level explanation of this, sorry. 

No worries! I'll reply anyway for anyone else reading this, but it's fine if you don't respond further.

Giving up on transhumanism as a useful idea of what-to-aim-for or identify as, separate from how much you personally can contribute to it.

It sounds like we have different ideas of what it means to identify as something. For me, one of the important functions of identity is as a model of what I am, and as what distinguishes me from other people. For instance, I identify as Finnish because of reasons like having a Finnish citizenship, having lived in Finland for my whole life, Finnish being my native language etc.; these are facts about what I am, and they're also important for predicting my future behavior.

For me, it would feel more like rationalization if I stopped contributing to something like transhumanism but nevertheless continued identifying as a transhumanist. My identity is something that should track what I am and do, and if I don't do anything that would meaningfully set me apart from people who don't identify as transhumanists... then that would feel like the label was incorrect and imply wrong kinds of predictions. Rather, I should just update on the evidence and drop the label.

As for transhumanism as a useful idea of what to aim for, I'm not sure of what exactly you mean by that, but I haven't started thinking "transhumanism bad" or anything like that. I still think that a lot of the transhumanist ideals are good and worthy ones and that it's great if people pursue them. (But there are a lot of ideals I think are good and worthy ones without identifying with them. For example, I like that museums exist and that there are people running them. But I don't do anything about this other than occasionally visit one, so I don't identify as a museum-ologist despite approving of them.)

More directly: avoiding "pinning your hopes on AI" (which, depending on how I'm supposed to interpret this, could mean "avoiding solutions that ever lead to aligned AI occurring" or "avoiding near-term AI, period" or "believing that something other than AI is likely to be the most important near-future thing"

Hmm, none of these. I'm not sure of what the first one means but I'd gladly have a solution that led to aligned AI, I use LLMs quite a bit, and AI clearly does seem like the most important near-future thing.

"Pinning my hopes on AI" meant something like "(subconsciously) hoping to get AI here sooner so that it would fix the things that were making me anxious", and avoiding that just means "noticing that therapy and conventional things like that work better for fixing my anxieties than waiting for AI to come and fix them". This too feels to me like actually updating on the evidence (noticing that there's something better that I can do already and I don't need to wait for AI to feel better) rather than like rationalizing something.

Okay! It wasn't intended as prescriptive but I can see it as being implicitly that.

What do you think I'm rationalizing? 

That's a pseudonym Duncan used at one point, see e.g. the first line of this comment.

That makes sense to me, though I feel unclear about whether you think this post is an example of that pattern / whether your comment has some intent aimed at me?

There's something about this framing that feels off to me and makes me worry that it could be counterproductive. I think my main concerns are something like:

1) People often figure out what they want by pursuing things they think they want and then updating on the outcomes. So making them less certain about their wants might prevent them from pursuing the things that would give them the information for actually figuring it out.

2) I think that people's wants are often underdetermined and they could end up wanting many different things based on their choices. E.g. most people could probably be happy in many different kinds of careers that were almost entirely unlike each other, if they just picked one that offered decent working conditions and committed to it. I think this is true for a lot of things that people might potentially want, but to me the framing of "figure out what you want" implies that people's wants are a lot more static than this.

I think this 80K article expresses these kinds of ideas pretty well in the context of career choice:

The third problem [with the advice of "follow your passion"] is that it makes it sound like you can work out the right career for you in a flash of insight. Just think deeply about what truly most motivates you, and you’ll realise your “true calling”. However, research shows we’re bad at predicting what will make us happiest ahead of time, and where we’ll perform best. When it comes to career decisions, our gut is often unreliable. Rather than reflecting on your passions, if you want to find a great career, you need to go and try lots of things.

The fourth problem is that it can make people needlessly limit their options. If you’re interested in literature, it’s easy to think you must become a writer to have a satisfying career, and ignore other options.

But in fact, you can start a career in a new area. If your work helps others, you practice to get good at it, you work on engaging tasks, and you work with people you like, then you’ll become passionate about it. The ingredients of a dream job we’ve found are most supported by the evidence, are all about the context of the work, not the content. Ten years ago, we would have never imagined being passionate about giving career advice, but here we are, writing this article.

Many successful people are passionate, but often their passion developed alongside their success, rather than coming first. Steve Jobs started out passionate about zen buddhism. He got into technology as a way to make some quick cash. But as he became successful, his passion grew, until he became the most famous advocate of “doing what you love”.

Comment retracted because right after writing it, I realized that the "leastwrong" is a section on LW, not its own site. I thought there was a separate leastwrong.com or something. In this case, I have much less of a feeling that it makes a global claim.

Edit: An initial attempt is "The LeastWrong" feels a bit like a global claim of "these are the least wrong things on the internet". 

This is how it feels to me. 

Whether you can find a logic in which that interpretation is not coherent doesn't seem relevant to me. You can always construct a story according to which a particular association is actually wrong, but that doesn't stop people from having that association. (And I think there are reasonable grounds for people to be suspicious about such stories, in that they enable a kind of motte-and-bailey: using a phrasing that sends the message X, while saying that of course we don't mean to send that message and here's an alternative interpretation that's compatible with that phrasing. So I think that a lot of the people who'd find the title objectionable would be unpersuaded by your alternative interpretation, even assuming that they bothered to listen to it, and they would not be unreasonable to reject it.)

Software/internet gives us much better ability to find.

And yet...

The past few decades have recorded a steep decline in people’s circle of friends and a growing number of people who don’t have any friends whatsoever. The number of Americans who claim to have “no close friends at all” across all age groups now stands at around 12% as per the Survey Center on American Life.

The percentage of people who say they don’t have a single close friend has quadrupled in the past 30 years, according to the Survey Center on American Life.¹

It’s been known that friendlessness is more common for men, but it is nonetheless affecting everyone. The general change since 1990 is illustrated below.

Taken from "Adrift: America in 100 Charts" (2022), pg. 223. As a detail, note the drastic drop of people with 10+ friends, now a small minority.

The State of American Friendship: Change, Challenges, and Loss (2021), pg. 7

Although these studies are more general estimates of the entire population, it looks worse when we focus exclusively on generations that are more digitally native. When polling exclusively American millennials, a pre-pandemic 2019 YouGov poll found 22%have “zero friends” and 30% had “no best friends.” For those born between 1997 to 2012 (Generation Z), there has been no widespread, credible study done yet on this question — but if you’re adjacent to internet spaces, you already intuitively grasp that these same online catalysts are deepening for the next generation.

Still, the fact that individual companies, for instance, develop layers of bureaucracy is not an argument against having a large economy.

This is true in principle, but population growth has led to the creation of larger companies in practice. ChatGPT when I asked it what proportion of the economy is controlled by the biggest 100 companies: 

For a rough estimate, consider the market capitalization of the 100 largest public companies relative to GDP. As of early 2023, the market capitalization of the S&P 100, which includes the 100 largest U.S. companies by market cap, was several trillion USD, while the U.S. GDP was about 23 trillion USD. This suggests a significant but not dominant share, with the caveat that market cap doesn't directly translate to economic contribution.

And if the population in every country would grow, then we'd end up with larger governments even if we kept the current system and never established a world government. To avoid governments getting bigger, you'd need to actively break up countries into smaller ones as their population increased. That doesn't seem like a thing that's going to happen.

A possible countertrend would be something like diseconomies of scale in governance. I don't know the right keywords to find the actual studies on this. Still, it generally seems to me like smaller nations and companies are better run than bigger ones, as the larger ones develop more middle management and organizational layers mainly incentivized to manipulate themselves rather than to do the thing they're supposedly doing. This does not just waste the resources of the government itself, it also damages everyone else as the legislation they enact starts getting worse and worse. And the larger the system becomes, the harder any attempts to reform it become.

Better matching to other people. A bigger world gives you a greater chance to find the perfect partner for you: the best co-founder for your business, the best lyricist for your songs, the best partner in marriage.

I'm skeptical of this; "better matching" implies "better ability to find". But just increasing the size of the population does not imply a better chance to find the best matches, given that it also increases the number of non-matches proportionally. And I think it's already the case that the ability to find the people is a much bigger bottleneck than just their existence.

It's also worth noting that as the population grows, so does the number of competitors. Maybe a 100x bigger population would have 100x the lyricists, but it may also have 100x the people wanting to hire those lyricists for themselves.

(Similar points also apply to the other "better matching" items.)

Which religion claims nothing supernatural at all happened?

Secular versions of Buddhism, versions of neo-paganism that interpret themselves to ultimately be manipulating psychological processes, religions whose conception of the divine is derived from scientific ideas, etc. More generally, many religions that define themselves primarily through practice rather than belief can be compatible with a lack of the supernatural (though of course aren't necessarily).

Agree. The advice I've heard for avoiding this is, instead of saying "try X", ask "what have you already tried" and then ideally ask some follow-up questions to further probe why exactly the things they've tried haven't worked yet. You might then be able to offer advice that's a better fit, and even if it turns out that they actually haven't tried the thing, it'll likely still be better received because you made an effort to actually understand their problem first. (I've sometimes used the heuristic, "don't propose any solutions until you could explain to a third party why this person hasn't been able to solve their problem yet".)

At that moment, he was enlightened.

I somehow felt fuzzy and nice reading this; it's so distinctly your writing style and it's nice to have you around, being you and writing in your familiar slightly quirky style. (It also communicated the point well.)