What do you think is wrong with rationalist culture?

post by tailcalled · 2023-03-10T13:17:28.279Z · LW · GW · 77 comments

This is a question post.


answer by MSRayne · 2023-03-10T15:08:53.598Z · LW(p) · GW(p)

I don't know much about the community beyond what's evident on LessWrong, but I've often felt like there's an undercurrent here of people tending towards a certain degree of selfishness (moral irrealism plus consequentialism plus "rationality is about winning" together makes a somewhat machiavellian personality lots of nice excuses), as well as messiah complexes which are not only somewhat destructive to the mental health of those that have them but also feed into that ego pattern even more (we're saving the world! only smart rationalists can understand! no point in trying to talk about alignment with normies because they're useless and can't help! the entire burden of saving the world is on my shoulders!!).

In general... this may be a place to go for good reasoning, but not for sanity in a more absolute sense. The emotional and social intelligence here, and indeed to some extent the "moral intelligence" is... not always adequate.

comment by TropicalFruit · 2023-03-11T04:48:08.402Z · LW(p) · GW(p)

I've also noticed those tendencies, not in the community but in myself.

Selfishness. Classification of people as "normies." Mental health instability. Machiavellianism.


They get stronger as I look at the world like a rationalist. You read books like Elephant in the Brain and find yourself staring at a truth you don't want to see. I wish God were real. I wish I were still a Christian with those guardrails erected to prevent me from seeing the true nature of the world.

But the more I look, the more like it looks like a non-moral, brutally unfair, unforgiving stochastic game we're all forced to play for... no real reason?

Obviously I'd like to me mentally healthier, more loving and selfless, etc, but... I don't know. I can't lie to myself. I'm stuck in between losing my old flawed, factually inaccurate philosophies, but not really having something better to replace them with yet. Just staring into the abyss of the amoral world and getting lost. I suspect most new rationalists are also in that space.

Replies from: lahwran, MSRayne
comment by the gears to ascension (lahwran) · 2023-03-11T10:25:40.058Z · LW(p) · GW(p)

good news on the moral front: prosocial moral intuitions are in fact a winning strategy long term. we're in a bit of a mess short term. but, solidarity and co-protection are good strategies; radical transparency can be an extremely effective move; mutual aid has always been a factor of evolution; the best real life game theory strategies tend to look like reputational generous tit for tat with semirandom forgiveness, eg in evolutionary game theory simulations; etc. Moral realism is probably true but extremely hard to compute. If we had a successful co-protective natural language program, it would likely be verifiably true and look like well known moral advice structured in a clear and readable presentation with its mathematical consequences visualized for all to understand.

I really like https://microsolidarity.cc as an everyday life intro to this, and I tossed this comment into metaphor.systems up to the opening bracket of this link. here are some very interesting results, various middle to high quality manifestos and quick overviews of ethics:

some of the results are actual science posts on the science post hubs - if you want to get into the field properly, you might try spidering around related papers, adding them to a folder, and shallowsly reading a bunch of the big ones. You could even add them to a semanticscholar folder and it'll give you recommendations for papers you might find interesting. could be very useful if you want to push SOTA on understanding of morality!

some of them get weird, but it is, in my opinion, rather fun and interesting weird:

Replies from: MSRayne
comment by MSRayne · 2023-03-11T13:36:06.776Z · LW(p) · GW(p)

You always post such cool links!!! I bet you're a cool person. :)

comment by MSRayne · 2023-03-11T13:31:55.129Z · LW(p) · GW(p)

There is a middle path. insert buddha vibes

In fact, I'm a moral realist! And I've got the skeleton of a rationalist argument for it. Only the skeleton, mind, and I'm sure people could easily blow holes in it. But making posts on here is... exhausting... so I haven't written it up.

But the more I look, the more like it looks like a non-moral, brutally unfair, unforgiving stochastic game we're all forced to play for... no real reason?

Well, yes, we live in a hell ruled by a dead (never-born) god. That's why it's our responsibility to create a living one (an aligned sovereign ASI) and liberate all sentient beings from suffering. That's what you ought to be living for.

comment by tailcalled · 2023-03-10T16:29:43.315Z · LW(p) · GW(p)

Do you have any favorite examples of this problem?

Replies from: MSRayne
comment by MSRayne · 2023-03-10T16:39:49.784Z · LW(p) · GW(p)

To be blunt... our founder's entire personality. (And current extreme burnout and evident depression.)

Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.

That's not quite an example of the specific things I mentioned, but it is an example of the rationality subculture tending to veer away from what I suppose has to be called "common sense" or "consensus reality". (Acausal reasoning and anthropics both also seem like examples of this. However "rational" they are, they are dangerous ideas that imo pose a cognitohazard.)

Actually, in the interests of full honesty, I have to give myself as an example. I didn't know about the rationalist community until I was like 20, but throughout my teens I basically was a rationalist without knowing it - and also mentally ill and coping with emotional disturbances using a lot of narcissistic or even quasi-sociopathic thought styles. I firmly believed that I understood reality better than anyone else and that it was my responsibility to save the world, and I was willing in principle to do absolutely anything to achieve that goal, up to and including hurt many people.

It wasn't until the past few years that I got comfortable feeling compassion for myself or others, and particularly after encountering the anarchist idea of the unity of means and ends and learning about intrinsic motivation, I've started to veer more towards some kind of hybrid of consequentialism, deontology, and virtue ethics rather than pure consequentialism. I think there is something inevitably destructive about any thinking style which places maximizing a linear definition of value above everything else, and I think an argument might exist for 1. humans not having coherent values representable as a utility function, not even in principle, and 2. all forms of utility maximization being intrinsically perverse, destructive, and immoral in the same way a paperclip maximizer is. But I haven't gone through the process of working out how to argue that yet.

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2023-03-10T19:40:35.201Z · LW(p) · GW(p)

Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.

Yeah, I hear ya. I think what's going on here is a difficult problem that I've been meaning to think and/or write about for some time: the balance between Taking Ideas Seriously [? · GW] and maintaining some sort of Memetic Immune System [? · GW]. It's a difficult thing to navigate.

I think the person was wrong about eating children and probably leaned too strongly towards the Taking Ideas Seriously side of things - ie. should have a stronger anchor to a sort of "common sense prior" - but personally, I'm totally fine with that as long as 1) they are doing it in good faith and come from a place of wanting to figure out the truth, and 2) they aren't actually causing harm in the real world... ie by eating children.

As a whole I feel like the rationalist community does a solid job navigating the tradeoff. If anything I actually think people don't lean hard enough towards Taking Ideas Seriously. For example, not many people are signed up for cryonics.

Replies from: Vladimir_Nesov, sharmake-farah, Jiro, daniel-glasscock
comment by Vladimir_Nesov · 2023-03-11T07:15:11.313Z · LW(p) · GW(p)

the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It's a difficult thing to navigate.

It's a false tradeoff, weird ideas taken seriously can be siloed in frames and hypotheticals [LW(p) · GW(p)]. They only get to become beliefs (at any level of credence) or decision relevant if they grow up to make global sense. In particular, ITT asks to give voice to simulacra of all sorts of strange.

comment by Noosphere89 (sharmake-farah) · 2023-03-11T17:24:52.377Z · LW(p) · GW(p)

Re morality, there's a potentially non-trivial chance that things are even worse than you think, in that two or more people disagreeing about morality or valence by starting with different assumptions can't ever converge, even in the infinite limit of time and compute, because morality is not pointing to any reality, but is instead social realistiy.

I support MSRayne blocking the person since in that scenario, there would be no convergence at all.

comment by Jiro · 2023-03-10T21:46:59.767Z · LW(p) · GW(p)

If anything I actually think people don’t lean hard enough towards Taking Ideas Seriously.

https://www.lesswrong.com/posts/QePFiEKZ4R2KnxMkW/posts-i-repent-of [LW · GW]

Stylistically contemptible, skimpy on any useful details, contributes to norm of pressuring people into double binds that ultimately do more harm than good. I would prefer it if no one linked to or promoted “Taking Ideas Seriously”;

comment by Daniel (daniel-glasscock) · 2023-03-10T20:31:24.774Z · LW(p) · GW(p)

There are reasonable and coherent forms of moral skepticism in which the statement, "It is morally wrong to eat children and mentally disabled people," is false or at least meaningless. The disgust reaction upon hearing the idea of eating children is better explained by the statement, "I don't want to live in a society where children are eaten," which is much more well-grounded in physical reality.

What is disturbing about the example is that this seems to be a person who believes that objective morality exists, but that it wouldn't entail that eating children is wrong. This is indeed a red flag that something in the argument has gone seriously wrong.

Replies from: MSRayne
comment by MSRayne · 2023-03-11T13:40:41.610Z · LW(p) · GW(p)

My problem is more the lack of moral realism to begin with. I apparently need to work on a post about this. I am sick and tired of the lack of belief in objective morality around here, leading people to entertain such insane thoughts to begin with; needs some pushback.

Replies from: sharmake-farah
comment by Noosphere89 (sharmake-farah) · 2023-03-11T17:01:01.716Z · LW(p) · GW(p)

This is a crux for me, and I conjecture roughly the opposite, that is moral/valence disagreements can't converge to any truth, even in the infinite limit.

That is, there are no guarantees for moral reasoning converging to the truth, the way that say a bounded or unbounded Solomonoff inductor can in universes that have a simplicity bias.

In other words, the assumption behind your morality are functionally arbitrary, and there's no deep justification for why you have the values you have.

Note: No one should change their values immediately based on this comment.

Replies from: MSRayne
comment by MSRayne · 2023-03-11T18:09:46.130Z · LW(p) · GW(p)

there's no deep justification for why you have the values you have.

Um... evolution by natural selection? A very very short sketch:

1. most superintelligences likely to exist in the multiverse were created by civilizations of social organisms;

2. civilizations of social organisms tend to have moral systems rooted in generalizations of basic social instincts which worked in the ancestral environment, such as tit for tat defaulting to cooperation, and possibly geometric rationality [? · GW];

3. some of those superintelligences are aligned and thus have value systems similar to those that tend to be evolved by civilizations of social organisms;

4. most are likely unaligned, but since unaligned superintelligences can have nearly any arbitrary utility function, those ones likely "cancel out";

5. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.

6. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on - which is to say, moral realism.

Replies from: sharmake-farah
comment by Noosphere89 (sharmake-farah) · 2023-03-11T19:16:24.675Z · LW(p) · GW(p)
  1. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
  1. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on - which is to say, moral realism.

Now I get to the crux of why I disagree, and I note you've smuggled in the assumption that the multiverse constrains morality enough such that it's sensible to talk about one moral truth or one true utility functions.

I think no multiverse that we actually live in constrains morality enough such that the conclusion of moral realism is correct, and that's why I disagree with the idea of moral realism. Similarly, this means that acausal economies will essentially be random chaos with local bubbles of moral systems, and that the aligned and unaligned systems have equal weight in the multiverse economy, that is infinite weight.

And they all cancel each other out. Also, once we get to the stage that we join the acausal economy, there's no reason to make an all encompassing economy across the entire multiverse, so there's no reason for any acausal economies to form at all.

Specifically for alignment, the goal and maybe definition of alignment is essentially making the AI do what someone wants. Critically, the only constraint is that the AI must either have the same goals as the person having the AI, or it has different goals but those goals aren't an impediment to the operator's goals.

Note under this definition of alignment, it doesn't comstrain the morality enough to make moral realism right, even after adding in instrumental goals.

Some notes on Geometric Rationality: I think there are some very useful notions from the geometric rationality sequence, like Thompson Sampling being better for exploration than it's equivalent in arithmetic rationality as well as techniques to reduce the force of Pascal's mugging, as he shows how exploration in the arithmetic rationality doesn't converge to the truth with probability of 1, while a geometric rationality technique known as Thompson Sampling does know the truth asymptotically with probability 1. However, arithmetic rationality does have some properties that are better than geometric rationality, such as being invariant to potentially partisan efforts to shift the zero point, and arithmetic rationality plays better with unbounded or infinite utility functions, which are relevant given that unbounded or infinite preferences do exist IRL.

I will say though, I'm strongly upvoting this in karma and weakly downvoting in the disagree direction. I obviously have quite strong disagreements with MSRayne on this, but I'm impressed by both how much MSRayne managed to maintain a truthseeking attitude even on a very controversial and potentially mind killing topic like morality, and impressed that someone made the argument clear so that I could find why I didn't agree with it. MSRayne, hats off to you for how well this conversation went.

Replies from: MSRayne
comment by MSRayne · 2023-03-11T19:42:05.657Z · LW(p) · GW(p)

The great thing is, this is ultimately an empirical question! Once we make an aligned ASI, we can run lots of simulations (carefully, to avoid inflicting suffering on innocent beings - philosophical zombie simulacra will likely be enough for this purpose) to get a sense of what the actual distribution of utility functions among ASIs in the multiverse might be like. "Moral science"...

Replies from: sharmake-farah
comment by Noosphere89 (sharmake-farah) · 2023-03-11T19:53:21.386Z · LW(p) · GW(p)

I definitely want to say that there's reason to believe at least some portions of the disagreement are testable, though I want curb enthusiasm by saying that we probably can't resolve the disagreement in general, unless we can somehow either make a new universe with different physical constants or modify the physical constants of our universe.

Also, I suspect the condition below makes it significantly harder or flat out impossible to run experiments like this, at least without confounding the results and thereby making the experiment worthless.

(carefully, to avoid inflicting suffering on innocent beings - philosophical zombie simulacra will likely be enough for this purpose)

comment by M. Y. Zuo · 2023-03-11T21:38:43.991Z · LW(p) · GW(p)

I call it the moderately-above-average syndrome. 

Someone with Einstein smarts or Napoleon level wiles, and with delusions of grandeur, seem to get along fine, at least judging by history.

But folks that are only 2 or 3 standard deviations above average, and who maintain similar pretences, inevitably come out a bit unbalanced.

There's also a similar concept in sociology with the anxious upper-middle classes.

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2023-03-12T02:04:32.872Z · LW(p) · GW(p)

Sounds like something sorta similar to the Midwit meme.

Replies from: M. Y. Zuo
comment by M. Y. Zuo · 2023-03-12T04:14:53.375Z · LW(p) · GW(p)

2 to 3 standard deviations above average would be on the right side of the standard bell curve diagram. 

Those actually in the middle usually don't develop genuine delusions of grandeur because they would encounter, and compete with, many people moderately smarter than them in day-to-day life. 

Whereas the moderately above average rarely genuinely interact, let alone compete, with modern day Einsteins.

answer by romeostevensit · 2023-03-10T17:25:02.347Z · LW(p) · GW(p)

Low scholarship (not mainly the academic kind) due to lack of slack from prioritizing the wrong winning metrics (money and status over time). In general, an optimization frame often falls into the trap of fine tuning existing considerations instead of seeking new considerations.

comment by interstice · 2023-03-10T18:23:10.074Z · LW(p) · GW(p)

What other kind of scholarship do you have in mind?

comment by Adam Zerner (adamzerner) · 2023-03-10T19:53:48.092Z · LW(p) · GW(p)

Let me attempt to paraphrase.

There are two problems: 1) not spending enough time on scholarship and 2) not having enough slack. These two problems are separate in the sense that 2 would be a problem even if 1 was solved and vice versa, but related in the sense that 2 is a big reason why 1 is a problem in the first place. And maybe 3) is another problem: that we spend too much time on existing considerations instead of seeking new considerations (exploiting instead of exploring).

Does that sound accurate?

If so, not that this adds much to the conversation, but 2 is the only one I have a decently strong sense of agreement towards.

1 I could see being true but also could see being false. Maybe people are currently spending roughly the right amount of time on scholarship. People in research and academia spending reading a good amount of papers, people in industry not doing so because it wouldn't help them achieve their goals all that much.

3 is interesting. I'm really not sure.

I do think it's worth pointing out that compared to other cultures, rationalist culture does a lot more to push 1) scholarship, 2) slack and 3) exploring. That's not to say that we can't do more along any of those dimensions though, just that "room for improvement" might be a better way to frame it than "problem".

Replies from: romeostevensit
comment by romeostevensit · 2023-03-11T21:09:33.929Z · LW(p) · GW(p)

Sounds like a reasonable take. I recognize there are issues with looking at a three sigma outlier and wishing for a four sigma outlier.

comment by TekhneMakre · 2023-03-10T18:17:09.032Z · LW(p) · GW(p)

Important + underappreciated.

comment by the gears to ascension (lahwran) · 2023-03-11T10:28:40.325Z · LW(p) · GW(p)

What sorts of queries on which knowledge retrievers would you suggest for learning more about this from the perspective you're seeing as lacking? if it's useful for answering this, my favorite search engines are arxivxplorer, semanticscholar's recommender, metaphor [LW(p) · GW(p)], and I also sometimes ask claude or chatgpt to describe a concept to help me identify search terms. using that set of tools, what would you suggest looking up to find links I can provide to others as an intro to scholarship? I have plenty of my own ideas for what to look up [LW · GW], to be clear.

Replies from: romeostevensit
comment by romeostevensit · 2023-03-11T21:11:18.406Z · LW(p) · GW(p)

I also use connected papers and search citation lineages. Linked resources seem good too. For scholarship I think Richard Hamming's final two chapters in art of doing science and engineering are hard to beat.

comment by Sune · 2023-03-10T21:41:57.353Z · LW(p) · GW(p)

What are you comparing to? It is only compared to what you would want rationalist culture to be like, or do you have examples of other cultures (besides academia) that do better in this regard?

Replies from: romeostevensit
comment by romeostevensit · 2023-03-11T21:12:22.222Z · LW(p) · GW(p)

I see small subcultures at good research schools that do well, but admit that what I'm looking for has very free examples, implying fragility.

answer by lc · 2023-03-10T18:06:44.059Z · LW(p) · GW(p)

The only thing chess club members have to do to participate is to organize or play in chess matches. The only thing computer security club members have to do to participate is (usually) to help organize or play computer hacking challenges. The only thing you have to do to get involved in the Christian Community is to go to church and maybe attend a few church functions.

AFAICT, the only obvious way to participate in and contribute to rationalist culture is to write insightful posts on LessWrong, in the same way that the only way to get involved with the SCPWiki is to write SCPs. But the bar for doing that in a prosocial and truthful way is now pretty high, and was always going to effect a power law, with a few very intelligent founding members contributing most of the canon. It's not that they're doing anything wrong (I love their content), it's just naturally what happens.

Most of the problems I see on LessWrong lie downstream of this. Regular non-Google, non-finance software engineers face this dilemma of either staying silent and never getting to interact with the community, saying something that's been said before, indulging in one of their biases, or unfairly criticizing existing works and members. For very unconscientious people this means completely throwing away existing guardrails and deontology because that's the only way they can think to differentiate themselves from Eliezer and carve a niche.

comment by Garrett Baker (D0TheMath) · 2023-03-10T18:54:48.681Z · LW(p) · GW(p)

I was able to get involved in rationality by going to in-person meetups. I suggest, if you're feeling left-out, you do the same (or create in-person meetups yourself!).

Edit: There also exist various rationalist discords you could join. They're usually fun, and don't require you to make a post.

Replies from: lc
comment by lc · 2023-03-11T01:43:07.750Z · LW(p) · GW(p)

if you're feeling left-out

Oh, don't be like that. It's more like: I like talking to rationalists. The only time I do so is when I make posts and comments. So I feel a noticable urge to come up with mediocre posts and comments when I'd rather just have some regular community function to attend.

I'll probably try to do a meetup soon now that I'm in LA.

Replies from: TropicalFruit
comment by TropicalFruit · 2023-03-11T04:30:16.308Z · LW(p) · GW(p)

I feel the same way. I like talking with people on here, but in almost every subject I have nothing substantive to contribute; I'm just a consumer.

I wish there were a broader, reddit-style aspect to this site for more ordinary posts. They don't have to be about Kim Kardashian or anything, but just regular economics, the current bank runs, Bitcoin, lifestyle/fitness/nutrition stuff, interesting links. You know, minus the reddit toxicity and religious zealotry in every subreddit.

Maybe I'm wrong. Maybe having the majority of the sub dedicated to AI alignment really is the way to go. It's just... I'm not smart enough, nor do I have the resources to meaningfully help on that front, and I suspect there are many like me in the IQ 130-145 range who absolutely love finally finding a community they can relate to, but don't have the 160+ IQ to really break ground on alignment research.

Unless I'm selling us regular geniuses short, but I don't think I am (sadly).

Replies from: Angela Pretorius, MSRayne
comment by Angela Pretorius · 2023-03-11T13:39:25.315Z · LW(p) · GW(p)

That’s what the r/slatestarcodex subreddit is for.

comment by MSRayne · 2023-03-11T13:52:34.895Z · LW(p) · GW(p)

You know, you can contribute to alignment without contributing to alignment. Focus on the places you're shocked everyone else is dropping the ball [LW · GW]. "Hey wait, why so little emphasis on aligning the humans that make AI? Wouldn't getting people to just slow the hell down and stop racing toward oblivion be helpful?" is one example of this, that would use an entirely different skillset (PR, social skills, etc) to work on. In my own case, I'm mainly interested in designing a system enabling mass human coordination and factored cognition, though I'm terrible at actually writing anything about the mountain of ideas in my head. This would indirectly speed up alignment by helping researchers think clearly, and also be great in many other ways. Think outside the "AI alignment directly and nothing else" box, and find something you can work on, with your skillset.

comment by Adam Zerner (adamzerner) · 2023-03-11T06:02:59.018Z · LW(p) · GW(p)

Regular non-Google, non-finance software engineers face this dilemma of either staying silent and never getting to interact with the community, saying something that's been said before, indulging in one of their biases, or unfairly criticizing existing works and members.

I'm glad you point this out. I think it is both real and important. However, I don't think it has to be that way! It's always been sort of a pet peeve of mine. "Normal" people can participate in so many ways. Here is what comes to my mind right now but definitely isn't exhaustive:

I think the issue is that there's not enough social proof for this sort of stuff. Not enough other people doing it. My theory is that too many other people writing insightful stuff makes it feel like the bar is set somewhere around there and thus it is taboo to start "lesser" [LW(p) · GW(p)] conversations.

answer by Dagon · 2023-03-10T17:14:50.060Z · LW(p) · GW(p)

I think it's wrong to think there is a "rationalist culture".  There are rationalist influences and tropes that are part of a number of distinct groups' habits and norms, but that doesn't make those groups similar enough to be called a cohesive single culture.

comment by Adam Zerner (adamzerner) · 2023-03-10T19:33:26.764Z · LW(p) · GW(p)

Disagreed, but curious.

My sense is that the differences are relatively minor and that there are a lot of really strong things that tie all the groups together: various things discussed in The Sequences like Bayesian thinking and cognitive science. What are the large differences you see with various groups?

Replies from: Dagon
comment by Dagon · 2023-03-11T15:35:04.619Z · LW(p) · GW(p)

There is a bit of far-mode general agreement on Bayesian thinking and the like. In terms of culture and near-mode norms, it’s those agreements which are relatively minor.

Behaviorally, especially offline, is where the large differences lie.

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2023-03-12T02:11:28.347Z · LW(p) · GW(p)

Hm, I think there's about 10 rationalists I've met IRL and spent more than a few hours with. Everyone has their own personality of course, but I didn't feel like there were any differences too large.

From what I read on LW I get the sense that the Berkeley community might have some large differences - maybe even sub-groups in Berkeley with large differences - but I don't recall hearing about any other communities with large differences. If other communities did exist with large differences I'm not actually sure that I'd expect to hear about it via browsing LW though.

answer by the gears to ascension · 2023-03-10T14:39:05.824Z · LW(p) · GW(p)

I'm not sure I know what rationalist culture refers to anymore. Several candidate referents have become blurred and new candidates have been introduced. Could be, lesswrong.com culture; humanity's rationalist cultures of various stripes; the rationalist cultures descended from lesswrong (but those are many at this point); the sequences view; the friend networks I have (which mostly don't have the problems I'd complain about, since I filter my friends for people I want to be friends with!); the agi safety research field (which seems to be mostly not people who think of themselves as "rationalists" anymore); berkeley rat crowd; "rationalist-adjacent" people on twitter; the thing postrats say is rationalist; a particular set of discords; some other particular set of discords; scott alexander fans; some vague combination of things I've mentioned; people who like secular solstice...

straw vulcan is more accurate than people give it credit for. a lot of people around these parts undervalue academia's output and independent scholarship and reinvent a lot of stuff. folks tend to have an overly reductive view of politics, either overly "only individuals exist and cannot be aggregated" or "only the greater good exists, individual needs not shared by others don't exist" - you know, uh, one of the main dimensions of variation that people in general are confused on. I dunno, it seems like the main thing wrong with rationalist culture is that it thinks of itself as rationalist, when in fact it's "just" another science-focused culture. shrug.

comment by tailcalled · 2023-03-10T16:30:19.347Z · LW(p) · GW(p)

Do you have any favorite examples of the straw vulcan thing?

Replies from: lahwran
comment by the gears to ascension (lahwran) · 2023-03-11T10:36:55.337Z · LW(p) · GW(p)

I don't have any examples ready at hand. It tends to be a pattern I see in people who strike me as somehow new to the concept of "rationalism", people who just read the sequences and are excited to tell everyone about how they're ingroup now.

I dunno. wait, maybe I could cite this comment by tropicalfruit as having the kind of vibe I'm thinking of: "oh no, are emotions truly useless? is morality fake?" - an understandable question, but still! https://www.lesswrong.com/posts/z4Rp6oBtYceZm7Q8s/what-do-you-think-is-wrong-with-rationalist-culture?commentId=HkKjobkvT6sfvRnG2 [LW(p) · GW(p)]

I get the sense that the rationalist vibe involves downregulating brain networks that implement important decision theory, because of not having an explicit description available that explains why those systems are key to the human {genome+memeplex}'s approximate learned decision theory

comment by Bill Benzon (bill-benzon) · 2023-03-10T17:57:36.776Z · LW(p) · GW(p)

a lot of people around these parts undervalue academia's output and independent scholarship and reinvent a lot of stuff.

That's certainly my impression. I've been peeking in here off and on for several years, but became more active last June when I started (cross-)posting here and commenting a bit.

I have a PhD that's traditional in the sense that I learned to search, read, value, and cite the existing literature on a topic I'm working on. That seems to be missing here, leading, yes, to unnecessary reinvention. I recall reading a post several months ago that made the point that some of the posts here are of sufficiently high quality that they should be placed at, for example, arXiv. Seems reasonable to me. But if you want to do that, you need to pay more attention to what other people are thinking and writing about.

Replies from: lahwran
comment by the gears to ascension (lahwran) · 2023-03-10T18:35:03.581Z · LW(p) · GW(p)

I'd love to hear your thoughts on how to compress the training that one gets beginning and throughout a phd about how to learn effectively from ongoing research. Many folks on here either don't have time or don't think we have time to go to school, so it would be nice to get resources together about how to learn it quickly. I've also been asking AIs questions like this, and I share the good ones when they come up.

Replies from: bill-benzon
comment by Bill Benzon (bill-benzon) · 2023-03-10T21:02:22.997Z · LW(p) · GW(p)

That's a tough one, in part because the fields vary so much. I was in an English department, so that's what my degree is in. But my real training came as part of a research group in computational linguistics that was in the linguistics department. I didn't actually do any programming. I worked on knowledge representation, a big part of old-school computational linguistics (before NLP).

But there are two aspects to this. One is getting the level of intellectual maturity and sophistication you need to function as a disciplined independent thinker. The other is knowing the literature. In some ways they interact and support one another but in some ways they are orthogonal.

I learned the most when I found a mentor, the late David G.l Hays. I wanted to learn his approach to semantics. He tutored me for an hour or two once a week for a semester. That's the best. It's also relatively rare to get that kind of individual attention. Still, finding a mentor is the best possible thing you could do.

At the same time I had a job preparing abstracts for The American Journal of Computational Linguistics. That meant I had to read a wide variety of material and prepare abstracts four times a year. You need to learn to extract the gist of an article without reading the whole thing. Look at the introduction and conclusion. Does that tell you what you need? Scan the rest. You should be able to do that – scan the article and write the abstract – in no more than an hour or two.

Note, that many/most abstracts that come with an article are not very good. The idea is, if you trust the journal and the author and don't need the details, the abstract should tell you all you need. Working up that skill is good discipline.

If you're working on a project with others here, each of you agree to produce 3, 4, 5 abstracts a week to contribute to the project. Post them to a place where you can all get at them. It becomes your project library.

As for the level of intellectual maturity, the only way to acquire that is to pick a problem, work on it, and come up with a coherent written account of what you've done. The account should be intelligible to others. I don't know whether a formal dissertation is required, but you need to tackle a problem that is both interesting to you and has "weight."

answer by shminux · 2023-03-10T19:50:16.520Z · LW(p) · GW(p)

I mentioned it in the post https://www.lesswrong.com/posts/p2Qq4WWQnEokgjimy/respect-chesterton-schelling-fences [LW · GW] : people are too eager to reject the Chesterton-Schelling fences because they feel enlightened and above the mundane guardrails that are for "normies".

answer by AβMale · 2023-03-11T19:27:14.046Z · LW(p) · GW(p)

I wrote about this here [LW(p) · GW(p)]:

[T]his error strikes me as … emblematic of a certain common failure mode within the rationalist community (of which I count myself a part). This common failure mode is to over-value our own intelligence and under-value institutional knowledge (whether from the scientific community or the Amazon marketplace), and thus not feel the need to tread carefully when the two come into conflict.

In that comment and the resulting thread, we discuss the implications of that with respect to the rationalist community’s understanding of Alzheimer’s disease, a disease I’ve studied in great depth. I’ve mostly found the community to have very strong opinions on that subject and disdain for the scientific community studying it, but very superficial engagement with the relevant scientific literature. Every single time I’ve debated the matter in detail with someone (maybe 5–10 times total), I’ve persuaded them that 1) the scientific community has a much better understanding of the disease than they realized and 2) that the amyloid hypothesis is compelling as a causal explanation. However, people in the rationalist community often have strongly-held, wrong opinions before (or in lieu of) these debates with me.

Ironically, the same thing happened in that thread: my interlocutor, John Wentworth, appreciated my corrections. However, I ultimately found the discussion a bit unsatisfying, because I don’t know that he made any meta-updates from it concerning the level of confidence that he started with without having seriously engaged with the literature.

comment by tailcalled · 2023-03-11T19:51:35.504Z · LW(p) · GW(p)

Potentially relevant: Random facts can come back to bite you [LW · GW].

comment by Noosphere89 (sharmake-farah) · 2023-03-11T19:45:58.814Z · LW(p) · GW(p)

Basically, this is essentially reframing the overuse of the inside view and under using the outside view, and I think this struck truer to my objection than my answer did.

And yeah, John Wentworth ignored the literature and was wrong, and since John Wentworth admitted it was cherry picked, this is non-trivial evidence against the thesis that Goodhart is a serious problem for AI or humans.

Though it also calls into question how well John Wentworth's epistemic processes are working.

answer by Marta · 2023-03-12T07:19:25.725Z · LW(p) · GW(p)

Limited number of groups and community events outside of the US/London (I'm from CEE, there are some groups, but not that many). It limits the possibility of in-person interaction. So, in a long-term, LW can only be my "online" community, not a "real life group of friends". Currently I regard EA events as a best way to meet rationalists, and, to be frank, it would be cool to have also other option and separate those two. 

comment by Viliam · 2023-03-13T18:35:47.910Z · LW(p) · GW(p)

Wanted to write the same thing. In my country, if you organize a meetup for all rationalists and all ACX readers and all effective altruists together... the total number of participants may come close to ten, if you are lucky!

answer by Noosphere89 · 2023-03-10T14:31:27.866Z · LW(p) · GW(p)

I'm going to focus on the overuse of the inside view, and the relative disuse of base rates and outside view. And it's why I think Eliezer's views on AI doom are probably not rational, and instead the product of a depression spiral, to quote John Maxwell.

On base rates of predictions of extinction, the obvious answer is that no extinction events happened out of 172 predicted ones, and while that's not enough of a sample to draw strong conclusions, it does imply that very high confidence in doom by a specific date is not very rational, unless you believe that you have something special that changes this factor.

Link is below:


The issue is that LWers generally assume that certain things are entirely new every time and that everything is special, and I think this assumption is overused in both LW and the broader world, which probably leads to the problem of overvaluing your own special inside view compared to others outside views.

comment by J Thomas Moros (J_Thomas_Moros) · 2023-03-10T17:17:22.111Z · LW(p) · GW(p)

This is not sound reasoning because of selection bias. If any of those predictions had been correct, you would not be here to see it. Thus, you cannot use their failure as evidence.

Replies from: Jay Bailey, bill-benzon
comment by Jay Bailey · 2023-03-10T21:13:12.941Z · LW(p) · GW(p)

I notice I'm a bit confused about that. Let's say the only thing I know about the sun is "That bright yellow thing that provides heat", and "The sun is really really old", so I have no knowledge about how the sun mechanistically does what it does.

I want to know "How likely is the sun to explode in the next hour" because I've got a meeting to go to and it sure would be inconvenient for the sun to explode before I got there. My reasoning is "Well, the sun hasn't exploded for billions of years, so it's not about to explode in the next hour, with very high probability."

Is this reasoning wrong? If so, what should my probability be? And how do I differentiate between "The sun will explode in the next hour" and "The sun will explode in the next year"?

Replies from: rhollerith_dot_com
comment by RHollerith (rhollerith_dot_com) · 2023-03-11T04:02:04.327Z · LW(p) · GW(p)

Yes, IMO the reasoning is wrong: if you you definitely cannot survive an event, then observing that the event did not happened is not evidence at all that it will not explode in the future -- and it continues to not be evidence as long as you continue to observe the non-explosion.

Since we can survive at least for a little while the sudden complete darkening of the sun the sun's not having gone dark is evidence that it will not go dark in the future, but it is less strong evidence than it would be if we could survive the darkening of the sun indefinitely.

The law of the conservation of expected evidence requires us to take selection effects like those into account -- and the law is a simple consequence of the axioms of probability, so to cast doubt on it is casting doubt on the validity of the whole idea of probability (in which case, Cox's theorems would like to have a word with you).

This is not settled science: there is not widespread agreement among scholars or on this site on this point, but its counter-intuitiveness is not by itself a strong reason to disbelieve it because there are parts of settled science that are as counterintuitive as this is: for example, the twin paradox of special relativity and "particle identity in quantum physics" [? · GW].

When you believe that the probability of a revolution in the US is low because the US government is 230 or so years old and hasn't had a revolution yet, you are doing statistical reasoning. In contrast, noticing that if the sun exploded violently enough, we would immediately all die and consequently we would not be having this conversation -- that is causal reasoning. Judea Pearl makes this distinction in the intro to his book Causality. Taking into account selection effects is using causal reasoning (your knowledge of the causal structure of reality) to modify a conclusion of statistical reasoning. You can still become confident that the sun will explode soon if you have a refined-enough causal model of the sun.

Replies from: TropicalFruit, Jay Bailey, TropicalFruit
comment by TropicalFruit · 2023-03-11T04:35:33.074Z · LW(p) · GW(p)

Off topic, but I'd just like to say this "good/bad comment" vs "I agree/disagree" voting distinction is amazing.

It allows us to separate our feeling on the content of the comment from our feeling on the appropriateness of the comment in the discussion. We can vote to disagree with a post without insulting the user for posting it. On reddit, this is sorely lacking, and it's one (of many) reasons every sub is an unproductive circle jerk.

I upvoted both of your comments, while also voting to disagree. Thanks for posting them. What a great innovation to stimulate discussion.

comment by Jay Bailey · 2023-03-11T11:45:15.080Z · LW(p) · GW(p)

So, I notice that still doesn't answer the actual question of what my probability should actually be. To make things simple, let's assume that, if the sun exploded, I would die instantly. In practice it would have to take at least eight minutes, but as a simplifying assumption, let's assume it's instantaneous.

In the absence of relevant evidence, it seems to me like Laplace's Law of Succession would say the odds of the sun exploding in the next hour is 1/2. But I could also make that argument to say the odds of the sun exploding in the next year is also 1/2, which is nonsensical. So...what's my actual probability, here, if I know nothing about how the sun works except that it has not yet exploded, the sun is very old (which shouldn't matter, if I understand you correctly) and that if it exploded, we would all die?

Replies from: rhollerith_dot_com
comment by RHollerith (rhollerith_dot_com) · 2023-03-11T21:02:57.710Z · LW(p) · GW(p)

In practice it would have to take at least eight minutes

We don't need to consider that here because any evidence of the explosion would also take at least eight minutes to arrive, so there is approximately zero minutes during which you are able to observe the evidence of the explosion before you are converted into a plasma that has no ability to update on anything. That is when observational selection effects are at their strongest: namely, when you are vanishingly unlikely to be in one of those intervals between your having observed an event and that event's destroying your ability to maintain any kind of mental model of reality.

We 21st-century types have so much causal information about reality that I have been unable during this reply to imagine any circumstance where I would resort to Laplace's law of succession to estimate any probability in anger where observational selection effects also need to be considered. It's not that I doubt the validity of the law; its just that I have been unable to imagine a situation in which the causal information I have about an "event" does not trump the statistical information I have about how many times the event has been observed to occur in the past and I also have enough causal information to entertain real doubts about my ability to survive if the event goes the wrong way while remaining confident in my survival if the event goes the right way.

Certainly we can imagine ourselves in the situation of the physicists of the 1800s who had no solid guess as to the energy source keeping the sun shining steadily. But even they had the analogy with fire. (The emissions spectra of the sun and of fire are both I believe well approximated as blackbody radiation and the 1800s had prisms and consequently at least primitive spectrographs.) A fire doesn't explode unless you suddenly give it fuel -- and not any fuel will do: adding logs to a fire will not cause an explosion, but adding enough gasoline will. "Where would the fuel come from that would cause the sun to explode?" the 1800s can ask. Planets are made mostly of rocks, which don't burn, and comets aren't big enough. Merely what I have written in this short paragraph would be enough to trump IMO statistical considerations of how many days the sun has gone without exploding.

If I found myself in a star-trek episode in which every night during sleep I find myself transported into some bizarre realm of "almost-pure sensation" where none of my knowledge of reality seems to apply and where a sun-like thing rises and sets, then yeah, I can imagine using the law of succession, but then for observational selection effects to enter the calculation, I'd have to have enough causal information about this sun-like thing (and about my relationship to the bizarre realm) to doubt my ability to survive if it sets and never rises again, but that seems to contradict the assumption that none of my knowledge of reality applies to the bizarre realm.

My probability of the sun's continuing to set and rise without exploding is determined exclusively by (causal) knowledge created by physicists and passed down to me in books, etc: how many times the sun has risen so far is in comparison of negligible importance. This knowledge is solid and "settled" enough that it is extremely unlikely that any sane physicist would announce that, well, actually, the sun is going to explode -- probably within our lifetimes! But if a sane physicist did make such an announcement, I would focus on the physicist's argument (causal knowledge) and pay almost no attention to the statistical information of how long there have been reliable observations of the sun's not exploding -- and this is true even if I were sure I could survive if the sun exploded -- because the causal model is so solid (and the facts the model depends on, e.g., the absorption spectra of hydrogen and helium, are so easily checked). Consequently, the explosion of the sun is not a good example of where observational selection effects become important.

By the way, observational selection effects are hairy enough that I basically cannot calculate anything about them. Suppose for example that if Russia attacked the US with nukes, I would survive with p = .4 (which seems about right). (I live in the US.) Suppose further that my causal model of Russian politics makes my probability that Russia will attack the US with nukes some time in the next 365 days as .003 if Russia had deployed nukes for the first time today (i.e., if Russia didn't have any nukes till right now). How should I adjust my probability (i.e., the .003) to take into account that fact that Russia's nukes were in fact deployed starting in 1953 (year?) and so far Russia has never attacked the US with nukes? I don't know! (And I have practical reasons for wanting to do this particular calculation, so I've thought a lot about it over the years. I do know that my probability should be greater than it should be if I and my ability to reason were impervious to nuclear attacks. In contrast to the solar-explosion situation, here is a situation in which the causal knowledge is uncertain enough that it would be genuinely useful to employ the statistical knowledge we have; it is just that I don't know how to employ it in a calculation.) But things that are almost certain to end my life are much easier to reason about -- when it comes to observational selection effects -- than something that has a .4 chance of ending my life.

In particular, most of the expected negative utility from AGI research stems from scenarios in which without warning -- more precisely, without anything that the average person would recognize as a warning -- an AGI kills every one of us. The observational selection effects around such a happening are easier to reason about than those around a nuclear attack: specifically, the fact that the predicted event hasn't happened yet is not evidence at all that it will not happen in the future. If a powerful magician kills everyone who tries to bring you the news that the Red Socks have won the World Series of Baseball, and if that magician is extremely effective at his task, then your having observed that the Yankees win the World Series every time it occurs (which is strangely not every year, but some years have no World Series as far as you have heard) is not evidence at all about how often the Red Socks have won the World Series.

And the fact that Eliezer has been saying for at least a few months now that AGI could kill us all any day now—that the probability that it will happen 15 years from now is greater than that probability that it will happen today, but the probability it will happen today is nothing to scoff at—is is very weak evidence against what he's been saying if it is evidence against it at all. A sufficiently rational person will assign what he has been saying the same or very nearly the same probability he would have if Eliezer had started saying it today. In both cases, a sufficiently rational person will focus almost entirely on Eliezer's argument (complicated though it is) and counterarguments and will give almost no weight to how long Eliezer's been saying it or how long AGIs have been in existence. Or more precisely, that is what a sufficiently rational person would do if he or she believed that he or she is unlikely to receive any advance warning of a deadly strike by the AGI beyond the warnings given so far by Eliezer and other AGI pessimists.

Eliezer's argument is more complicated than the reasoning that tells us that the sun will not explode any time soon. More complicated means more likely to contain a subtle flaw. Moreover, it has been reviewed by fewer experts than the solar argument. Consequently, here is a situation in which it would be genuinely useful to use statistical information (e.g., the fact that research labs have been running AGIs for years (ChatGPT is an AGI for example) combined with the fact that we are still alive) but the statistical information is in fact IMO useless because of the extremely strong observational selection effects.

comment by TropicalFruit · 2023-03-11T04:40:48.619Z · LW(p) · GW(p)


I'm at a local convenient store. A thief routinely robs me. He points a gun at me, threatens me, but never shoots, even when I push back a little. At this point, it's kind of like we both know what's happening, even though, technically, there's a chance of physical danger.

Had this guy shot me, I wouldn't be alive to reason about his next visit.

Now consider a different thief comes in, also armed. What is my probability of getting shot, as compared with the first thief?

Much, much, higher with the second thief. My past experiences with the first thief act as evidence towards the update that I'm less likely to be shot. With this new thief, I don't have that evidence, so my probability of being shot is just the based rate based on my read of the situation.

I believe updating on the non-fatal encounters with the first thief is correct, and it seems to me analogous to updating on the sun not having exploded. Thoughts?

Replies from: rhollerith_dot_com
comment by RHollerith (rhollerith_dot_com) · 2023-03-11T19:48:13.636Z · LW(p) · GW(p)

Because a person has a significant chance of surviving a bullet wound -- or more relevantly, of surviving an assault with a gun -- your not having been assaulted by the first thief is evidence that you will not be assaulted in future encounters with him, but it is weaker evidence than it would be if you could be certain of your ability to survive (and your ability to retain your rationality skills and memories after) every encounter with him.

Humans are very good at reading the "motivational states" of the other people in the room with them. If for example the thief's eyes are glassy and he looks like he is staring at something far away even though you know it is unlikely there there is anything of interest in his visual field far away, well that is a sign he is in a dissociated state, which makes it more likely he'll do something unpredictable and maybe violent. If when he looks at you he seems to look right through you, that is a sign of a coldness that also makes it more likely he will be violent if he can thereby benefit himself personally by doing so. So, what is actually doing most of the work of lowering your probability about the danger to you posed the the first thief? The mere fact that you escaped all the previous encounters without having been assaulted or your observations of his body language, tone of voice and other details that give clues about his personality and his mental state?

Replies from: TropicalFruit
comment by TropicalFruit · 2023-03-11T21:47:27.881Z · LW(p) · GW(p)

Replace thief with a black box that either explodes and kills you, or doesn't. It has some chance to kill you, but you don't know what that chance is.

I was put in a room with black-box-one 5 times. Each time it didn't explode.

Now, I have a choice: I can go back in the room with black-box-one, or I can go to a room with black-box-two.

I'll take black-box-one, based on prior evidence.

Replies from: rhollerith_dot_com
comment by RHollerith (rhollerith_dot_com) · 2023-03-11T22:55:59.174Z · LW(p) · GW(p)

If I know nothing about the boxes except that they have the same a priori probability of exploding and killing me, then I am indifferent between the two black boxes.

It is not terribly difficult to craft counter-intuitive examples of the principle. I anticipated I would be presented with such examples (because this is not my first time discussing this topic), which is why in my original comment I wrote, "its counter-intuitiveness is not by itself a strong reason to disbelieve it," and the rest of that paragraph.

Replies from: TropicalFruit
comment by TropicalFruit · 2023-03-12T02:33:31.631Z · LW(p) · GW(p)

Okay but I just don't agree. 

Let each black box have some probability to kill you, uniformly chosen from a set of possible probabilities. Let's start with a simple one: that probability is 0 or 1.

The a prior chance to kill you is .5. 

After the box doesn't kill you, you update, and now the chance is 0.

What about if we use a uniform distribution from [0,1)? Some boxes are .3 to kill you, others .78.

Far more of the experiences of not dying are from the low p-kill boxes than from the high p-kill ones. When people select the same box, instead of a new one, after not being killed, that brings the average kill rate of selected boxes down. Run this experiment for long enough, and the only boxes still being selected are the extremely low p-kill boxes that haven't killed all their subjects yet.

This time, could you make a stronger objection, that's more directly addressed at my counter-example?

Replies from: rhollerith_dot_com, rhollerith_dot_com
comment by RHollerith (rhollerith_dot_com) · 2023-03-12T06:54:53.510Z · LW(p) · GW(p)

In your new scenario, if I understand correctly, you have postulated that one box always explodes and one never explodes; I must undergo 2 experiences: the first experience is with one of the boxes, picked at random; then I get to choose whether my second experience is with the same box or whether it is with the other box. But I don't need to know the outcome of the first experience to know that I want to limit my exposure to just one of these dangerous boxes: I will always choose to undergo the second experience with the same box as I underwent the first one with. Note that I arrived at this choice without doing the thing that I have been warning people not to do, namely, to update on observation X when I know it would have been impossible for me to survive (or more precisely for my rationality, my ability to have and to refine a model of reality, to survive) the observation not X.

That takes care of the first of your two new scenarios. In your second new scenario, I have a .5 chance of dying during my first experience. Then I may choose whether my second experience is with the same box or a new one. Before I make my choice, I would dearly love to experiment with either box in a setting in which I could survive the box's exploding. But by your postulate as I understand it, that is not possible, so I am indifferent about which box I have my second experience with: either way I choose, my probability that I will die during the second experience is .5.

Note the in your previous comment, in which there was some P such each time a box is used, it has a probability P of exploding, there is no benefit to my being able to experiment with a box in a setting in which I could survive an explosion, but in the scenario we are considering now there is a huge benefit.

Suppose my best friend is observing the scenario from a safe distance: he can see what is happening, but is protected from any exploding box. My surviving the first experience changes his probability that the box used in the first experience will explode the next time it is used from .5 to .333. Actually, I am not sure of that number (because I am not sure the law of succession applies here -- it has been a long time since I read my E.T. Jaynes) but I am sure that his probability changes from .5 to something less than .5. And my best friend can communicate that fact to me: "Richard," he can say, "stick with the same box used in your first experience." But his message has the same defect that my directly observing the behavior of the box has: namely, since I cannot survive the outcome that would have led him to increase his probability that the box will explode the next time it is used, I cannot update on the fact that his probability has decreased.

Students of E.T. Jaynes know that observer A's probability of hypothesis H can differ from observer B's probability: this happens when A has seen evidence for or against H that B has not seen yet. Well, here we have a case where A’s probability can differ from B’s even though A and B have seen the same sequence of evidence about H: namely, that happens when one of the observers could not have survived having observed a sequence of events (different from the sequence that actually happened) that the other observer could have survived.

comment by RHollerith (rhollerith_dot_com) · 2023-03-13T17:31:20.209Z · LW(p) · GW(p)

TropicalFruit and I have taken this discussion private (in order to avoid flooding this comment section with discussion on a point only very distantly related to the OP.) However if you have any interest in the discussion, ask one of us for a copy. (We have both agreed to provide a copy to whoever asks.)

Replies from: sharmake-farah
comment by Noosphere89 (sharmake-farah) · 2023-03-13T20:01:28.095Z · LW(p) · GW(p)

I would like a copy of the discussion.

comment by Bill Benzon (bill-benzon) · 2023-03-12T16:40:35.531Z · LW(p) · GW(p)

It seems to me there is a distinction to be made: It is one thing to conclude that, 1) Eliezer doesn't know how to predict the date of AI Doom. That's different from asserting that 2) AI Doom is not going to happen. 1 is not evidence for 2.

comment by TropicalFruit · 2023-03-11T04:56:20.184Z · LW(p) · GW(p)

I think it's appropriate to draw some better lines through concept space for apocalyptic predictions, when determining a base rate, than just "here's an apocalyptic prediction and a date." They aren't all created equal.

Herbert W Armstrong is on this list 4 times... each time with a new incorrect prediction. So you're counting this guy who took 4 guesses, all wrong, as 4 independent samples on which we should form a base rate.

And by using this guy in the base rate, you're implying Eliezer's prediction is in the same general class as Armstrong's, which is a stretch to say the least.

A pretty simple class distinction is: how accurate are other predictions the person has made? How has Eliezer's prediction record been? How have his AI timeline predictions been?

I don't know the answers to these questions, maybe they really have been bad, but I'm assuming they're pretty good. If that's the case, then clearly Eliezer's prediction doesn't deserve to classified with the predictions listed on that page.

answer by ZT5 · 2023-03-10T14:07:04.122Z · LW(p) · GW(p)

In my paradigm, human minds are made of something I call "microcognitive elements", which are the "worker ants" or "worker bees" of the mind.
They are "primed"/tasked with certain high-level ideas and concepts, and try to "massage"/lubricate the mental gears into both using these concepts effectively (action/cognition) and to interpret things in terms of these concepts (perception)
The "differential" that is applied by microcognitive elements to make your models work, is not necessarily related to those models and may in fact be opposed to them (compensating for, or ignoring, the ways these models don't fit with the world)

Rationality is not necessarily about truth. Rationality is a "cognitive program" for the microcognitive elements. Some parts of the program may be "functionally"/"strategically"/"deliberately" framing things in deceptive ways, in order to have the program work better (for the kind of people it works for).

The specific disagreements I have with the "rationalist" culture:

  • The implied statement that LessWrong paradigm has a monopoly on "rationality", and is "rationality", rather than an attempted implementation of "rationality", a set of cognitive strategies based on certain models and assumptions of how human minds work. If "rationality is about winning [? · GW]", then anyone who is winning is being rational, whether they hold LW-approved beliefs or not.
  • Almost complete disregard for meta-rationality.
  • Denial of nebulosity, fixation on the "imaginary objects" that are the output of the lossy operation of "make things precise so they can be talked about in precise terms".

All of these things have computational reasons, and are a part of the cognitive trade-offs the LW memeplex/hive-mind makes due to its "cognitive specialization". Nevertheless, I believe they are "wrong", in the sense that they lead to you having an incorrect map/model of reality, while strategically deceiving yourself into believing that you do have a correct model of reality. I also believe they are part of the reason we are currently losing - you are being rational, but you are not being rational enough
Our current trajectory does not result in a winning outcome.

comment by TropicalFruit · 2023-03-11T05:06:04.451Z · LW(p) · GW(p)

Since reading the sequences, I've made much more accurate predictions about the world. 

Both the guiding principle of making beliefs pay rent in anticipated experience, as well as the tools by which to acquire those accurate beliefs, have worked for me.

So at an object level, I disagree with your claim. Also, if you're going to introduce topics like "meta-rationality" and "nebulosity" as part of your disagreement, you kind of have to defend them. You can't just link a word salad and expect people to engage. The first thing I'm looking for is a quick, one or two paragraph summary of the idea so I can decide whether it's worth it to pursue further.


Comments sorted by top scores.

comment by Raemon · 2023-03-10T19:05:10.126Z · LW(p) · GW(p)

(fyi, downvoted because while I think there's a good version of this question, the current one feels too vague to be about anything particularly good, and most version of this discussion seem more likely to be vaguely-self-flagellating or reverse-circle-jerky rather than useful)

Replies from: adamzerner
comment by Adam Zerner (adamzerner) · 2023-03-10T19:25:53.296Z · LW(p) · GW(p)

I'm not sure that vagueness is a problem here. It could be useful to hear from people with various takes on what exactly the question is asking.

I do worry a little about the framing leading to contentiousness though and think the question would be improved by somehow trying to mitigate that.

Replies from: tailcalled
comment by tailcalled · 2023-03-10T19:39:19.980Z · LW(p) · GW(p)

Yeah I was mainly trying to invite a broad set of opinions.

comment by iceman · 2023-03-11T21:03:57.640Z · LW(p) · GW(p)

Meta-note related to the question: asking this question here, now, means you're answer will be filtered for people who stuck around with capital r Rationality and the current LessWrong denizens, not the historical ones who have left the community. But I think that most of the interesting answers you'd get are from people who aren't here at all or rarely engage with the site due to the cultural changes over the last decade.

Replies from: tailcalled
comment by tailcalled · 2023-03-11T21:40:01.724Z · LW(p) · GW(p)

Yeah, I've been reading a lot of critiques by Benjamin Hoffman and thinking about some of the prior critiques by Jessica Taylor, and that's sort of what prompted me to ask this question. It would probably also be interesting to look at others who left it, they're just harder to get hold of.